
OSS
ã€ãã³ã
ãã¬ãžã³
æè¡ããã°
ããã«ã¡ã¯ããœãªã¥ãŒã·ã§ã³ã¢ãŒããã¯ãã®å®äœçŸã§ãã 2026幎7æ15æ¥(æ°Ž) ã«éå¬ããããNeuron Community â 2026 Vol.1ãã®æ§åãã¬ããŒãããŸãããã®ã€ãã³ãã¯ã2025幎3æã«ç«ã¡äžãããããNeuron Communityãã®ååã®ããšéå¬ããŸããã ä»åã¯ã AWS Summit Japan 2026 éå¬åŸãšããããšããããAWS Summit Japan ã®æ¯ãè¿ãããAWS Neuron ã®ã¢ããããŒãæ
å ±ãå€ãã®å
容ãšãªã£ãŠããŸãã Neuron Community ãšã¯ AWS ã§ã¯ãæ©æ¢°åŠç¿ã®ãã¬ãŒãã³ã°ãšæšè«ã®ããã®é«æ§èœã§è²»çšå¯Ÿå¹æã®é«ãæ©æ¢°åŠç¿ã¢ã¯ã»ã©ã¬ãŒã¿ïŒ AWS Trainium ã AWS Inferentia ïŒãããã³æ·±å±€åŠç¿ãšçæ AI ã¯ãŒã¯ããŒããå®è¡ããããã«äœ¿çšããã SDK ã® AWS Neuron ãæäŸããŠããŸãããNeuron Communityãã¯ããŠãŒã¶ãŒéã§ AWS Trainium / AWS Inferentia / AWS Neuron ã®ç¥èŠå
±æãä¿é²ããå ŽãšããŠçºè¶³ããŸããã ãNeuron Communityãã¯ã䞻㫠Discord ã䜿çšããŠéå¶ãããŠããŸããèå³ãæã£ãŠããã ããæ¹ã¯ãäžèšã® URL ããåå ããŠã¿ãŠãã ããã AWS Neuron Community (Discord) : https://discord.gg/DUx4g3Z3pq ãªãŒããã³ã°ïŒNeuron Community ã®æãç«ã¡ãšã«ã©ã¯ãªç€Ÿã§ã® Trainium åãçµã¿çŽ¹ä» äžå±± æºæ æ°ïŒã«ã©ã¯ãªæ ªåŒäŒç€Ÿ åç· åœ¹ CPO) è³æïŒåŸæ¥å
¬é ãªãŒããã³ã°ã»ãã·ã§ã³ã§ã¯ãã«ã©ã¯ãªæ ªåŒäŒç€Ÿã®äžå±±æ°ããçºè¡šããŠããã ããŸãããã«ã©ã¯ãªæ ªåŒäŒç€Ÿã¯ã2023幎ããäžè²«ã㊠AWS Trainium ãå©çšãç¶ããŠãããNeuron Community ã®ç«ã¡äžãã«ã倧ããªè²¢ç®ãããŠããã ããŠããŸãããã®çºè¡šã§ã¯ãNeuron Community ã®å§ãŸãã«ã€ããŠç޹ä»ããŠããã ããŸããããŸããã«ã©ã¯ãªæ ªåŒäŒç€Ÿã® AWS Trainium ã«é¢ãã2ã€ã®åãçµã¿ã«ã€ããŠã玹ä»ããŠããã ããŸããã1ã€ç®ã®åãçµã¿ã¯ãAmazon EKS äžã«æ§ç¯ããã ãNeuron 忣åŠç¿ãã©ãããã©ãŒã ã ã§ãããã®ãã©ãããã©ãŒã ãæ§ç¯ããããšã§ãã€ã³ãã©é¢é£ã®ç¥èãååã§ã¯ãªãã¡ã³ããŒã§ã忣åŠç¿ãå®è¡ã§ããç°å¢ã®æŽåãé²ããŠããããã§ãã2ã€ç®ã®åãçµã¿ã¯ãAWS Trainium ã® NKI ã«ãŒãã«éçºãä¿é²ããããã® ãã«ãŒãã«éçºãšãŒãžã§ã³ãã ã§ãããã® AI ãšãŒãžã§ã³ãã«ãããNKI ã«ãŒãã«éçºããšãŒãžã§ã³ããèªåŸçã«é²ããããããã«ãªããšããããšã§ããæåŸã«ãä»åŸã® Neuron Community ã®æŽ»åã«ã€ããŠããããªãŒãã³ãªå Žã«ããŠãããããšããçºä¿¡ãããŠããã ããŸããã AWS Summit Japan æ¯ãè¿ãâ ã»ãã·ã§ã³ãã€ãžã§ã¹ããâŒ€èŠæš¡åŠç¿ãã AI ãšãŒãžã§ã³ãã®æšè«ãŸã§ ~ ã³ã¹ãå¹çãšæ§èœã䞡✎ãã AWS Trainium ã®å
šè² ~ã æŸ€ 亮倪 (Amazon Web Services Japan G.K.) è³æïŒ âAI ãšãŒãžã§ã³ãã®æšè«ããâŒ€èŠæš¡åŠç¿ãŸã§â ã³ã¹ãå¹çãšæ§èœã䞡✎ãã AI ã€ã³ã㩠̶ AWS Trainium ã®å
šè² Amazon Web Services Japan G.K. ã®æŸ€ããã¯ã AWS Summit Japan 2026 ã®æ¯ãè¿ããšããŠããâAI ãšãŒãžã§ã³ãã®æšè«ããå€§èŠæš¡åŠç¿ãŸã§â ã³ã¹ãå¹çãšæ§èœãäž¡ç«ãã AI ã€ã³ã㩠㌠AWS Trainium ã®å
šè²ãã®ã»ãã·ã§ã³ãã15 åã®ãã€ãžã§ã¹ãçã§ç޹ä»ããŸããããã®ãã€ãžã§ã¹ãã§ã¯ãAWS Trainium ã®å
žåçãªäœ¿ãæ¹ãšããŠã âA. ã³ãŒãã¯ãã®ãŸãŸã§åŠç¿ã³ã¹ããäžãããâ ã âB. æ§èœãæé©åãããâ ã âC. æšè«ã³ã¹ããåºå®åãããâ ã®3ç¹ã«æ³šç®ããŠèª¬æããŸããã A. ã§ã¯ã Native PyTorch supportïŒããŒã¿çïŒ ã䜿ããGPUåãPyTorchã³ãŒãã®ããã€ã¹æå®ã cuda ãã neuron ã«å€æŽããŠãAWS Trainium äžã§åŠç¿ãå®è¡ããæ¹æ³ã玹ä»ããŸããããã¢ã§ã¯ãGPT-2 ã®åŠç¿ã¹ã¯ãªãããå®è¡ããŸãããB. ã§ã¯ã NKI (Neuron Kernel Interface) ã«ãã AWS Trainium ã®ããŒããŠã§ã¢åœä»€ã»ããã«çŽæ¥ã¢ã¯ã»ã¹ã㊠AI ã«ãŒãã«ã®æé©åãå¯èœã§ããããšã玹ä»ããŸããããŸããæ§èœæé©åãšãããã°ã®ã¯ãŒã¯ãããŒãæ¯æŽãã Neuron ExplorerãNKI ã®éçºã AI ãšãŒãžã§ã³ãã§å éããããã®ãªãŒãã³ãœãŒã¹ããŒã«ãããã® âNeuron Agentic Developmentâ ã«ã€ããŠã玹ä»ããNeuron Agentic Development ã®ãã¢ãèŠãŠããã ããŸãããC. ã§ã¯ vLLM on Trainium ã䜿ãããšã§ããªãŒãã³ãŠã§ã€ãã¢ãã«ãAWS Trainium äžã§ãµãŒãã³ã°ã§ããŸããããã§ã¯ãopenai/gpt-oss-20b ã¢ãã«ã AWS Trainium äžã§ãµãŒãã³ã°ãããã¢ãèŠãŠããã ããŸããã AWS Summit Japan æ¯ãè¿ãâ¡ ããŒã¹å±ç€ºç޹ä»ã✣æ AI ãâœããã€ã³ãã©æè¡ã 赀柀 Toshinobu (Amazon Web Services Japan G.K.) Amazon Web Services Japan G.K. ã®èµ€æŸ€ããã¯ã AWS Summit Japan 2026 ã®æ¯ãè¿ããšããŠãããŒã¹å±ç€ºãçæAIãæ¯ããã€ã³ãã©æè¡ãã«ã€ããŠç޹ä»ããŸããããã®å±ç€ºã¯ãè€æ°ã®ãã«ãã¢ãŒãã«ã¢ãã«ã Amazon EC2 trn2.48xlarge ã§ãµãŒãã³ã°ããæ§åãèŠãŠããã ããã®ã§ããã«ãã¢ãŒãã«ã¢ãã«ã§ç»åã®ç·šéãè¡ããŸããé³å£°ã§ç»åç·šéã®æç€ºããããš Whisper Large v3 ã§é³å£°èªèãè¡ããQwen3-VL-8B-Instruct ã§å
ã«ãªãç»åãç·šéããããã®æç€ºãçæããŸããæç€ºã¯ Qwen-Image-Edit-2511 ã«æž¡ãããç»åãç·šéãããŸããç·šéãããç»åã¯ãQwen3-VL-8B-Instruct ã䜿ã£ãŠæç€ºéãã«ç·šéã§ããŠããããè¬è©ããXTTSv2 ã§é³å£°åºåããŸãããã®çºè¡šã§ã¯ãïŒå¹ã®åç«ã®ã€ã©ã¹ããã4å¹ã«å¢ãããšããç»åç·šéã®æ§åãèŠãŠããã ããŸããã ãŸãããã®ãã¢ãå®çŸããŠããã¢ãŒããã¯ãã£ã«ã€ããŠã®èª¬æãè¡ããŸããã4ã€ã®ã¢ãã«ã®trn2.48xlarge ã® 64 è«çã³ã¢ãžã®ã¢ãã±ãŒã·ã§ã³ããã¢ãã«ã®ãããã€ãããŒãªã©ã説æããŠããŸãã AWS Trainium / Inferentia / Neuron SDK ææ°ã¢ããããŒã åžžäž å€§å² (Amazon Web Services Japan G.K.) è³æïŒ Neuron Communit 2026 Vol.1 AWS Trainium / Neuron ææ°ã¢ããããŒã Amazon Web Services Japan G.K. ã®åžžäžããã¯ãã¡ããã©ã€ãã³ã忥ã«ãã¬ãæ±äº¬ã® WBSïŒã¯ãŒã«ãããžãã¹ãµãã©ã€ãïŒã§ AI åãã¢ããŸã³ç¬èªã®åå°äœéçºãç¹é ãããããšã«è§Šããèªèº«ãæå±ããã¢ããŸã³å
ã®ãããéçºéšéãã¢ã³ããã«ãã©ããã«ã€ããŠç޹ä»ããŸãããAnthropic ãšã®å
±åãããžã§ã¯ã Project Rainier ã§ã¯ããããŸã§ã« 140 äžåè¶
ã® Trainium 2 ããã³ Trainium 3 ãããã皌åäžã§ããããšïŒWBS å
ã®ç¹éã«ãŠç޹ä»ïŒã OpenAI ã 2GW èŠæš¡ã§ã® Trainium æ¡çšãçºè¡š ããããšããŸãåŸæ¥ã®ãã£ãããããå AI ãããšãŒãžã§ã³ãå AI ãžãšã·ããããäžã§ãAI ãããã«å ã AWS Graviton ããã»ããµã®éèŠæ§ãå¢ããŠããç¹ã玹ä»ããŸããã Meta ãæ°åäžã® Graviton ã³ã¢ã§ Agentic AI ãã¹ã±ãŒã«ããŠããäºäŸ ã«ãè§ŠããŸããã æ¬¡ã«ã柀ã®ã»ãã·ã§ã³ã§ã玹ä»ããã AWS Trainium åãã® SDKãAWS Neuronãã®ã¢ããããŒããšããŠãã©ã€ãã©ãªã®ãã€ãã£ãåïŒNative PyTorchãNative vLLMïŒã®ææ°ç¶æ³ã玹ä»ããŸããã ãŸãã7 æ 7 æ¥ã«ãªãªãŒã¹ãããææ°ã® Neuron 2.31 ã§ã¯ãæ§èœæé©åã®èŠã§ãã NKIïŒNeuron Kernel InterfaceïŒãš NKI Library ã«å€§ããªã¢ããããŒãããã£ãç¹ãããã« NKI ã«ãŒãã«éçºçšã®ãšãŒãžã§ã³ãã³ãŒãã£ã³ã°æ©èœ Neuron Agentic Development ã«ããã«ãŒãã«èªåæé©åã«ãŒããžã®æ³šåã玹ä»ããã»ãã·ã§ã³ãç· ãããããŸããã â» ã€ãã³ãéå¬åŸã® 2026幎7æ20æ¥(æ) ã« vLLM Neuron Beta ããããªãã¯ãªãªãŒã¹ããŸããïŒ ãããã« éç®3åç®ã® Neuron Community ã¯ãã«ã©ã¯ãªæ ªåŒäŒç€Ÿã§ã® AWS Trainium ãžã®åãçµã¿ã®çºè¡šããAWS Summit Japan 2026 ã®æ¯ãè¿ããAWS Neuron é¢é£ã®ææ°ã¢ããããŒãæ
å ±ã®ç޹ä»ãšãå
å®ããå
容ãšãªããŸãããAWS Summit Japan 2026 ã®ã»ãã·ã§ã³åç»ã¯ã AWS Summit Japan ã® Web ããŒãž ã«ç»é²ããã ãããšã§ãªã³ããã³ãèŠèŽãå¯èœã§ãããèå³ã®ããæ¹ã¯ããã²ç»é²ããŠã¿ãŠãã ããã çºè¡šåŸã«ã¯ä»åŸã® Neuron Community ã«ã€ããŠã®ãã£ã¹ã«ãã·ã§ã³ãè¡ãããçŽ 1 幎ã¶ãã®éå¬ãšãªã£ãããšãèžãŸããããé«ãé »åºŠã§éå¬ããŠããããšãã声ãæãããŸãããAWS ãšããŠãç©æ¥µçã«æ¯æŽããŠãããŸãã ä»åŸã® Neuron Community ããDiscord ãäžå¿ã«åéãåç¥ãè¡ã£ãŠãããŸããèå³ãæã£ãŠããã ããæ¹ã¯ããã²ãäžèšã® URL ããåå ããŠã¿ãŠãã ããã AWS Neuron Community (Discord) : https://discord.gg/DUx4g3Z3pq èè
ã«ã€ã㊠å®äœçŸ é
çŽ (Usami Masanori) è£œé æ¥ã®ã客æ§ãæ
åœãããœãªã¥ãŒã·ã§ã³ã¢ãŒããã¯ãã§ãã è£œé æ¥ã®ã客æ§ã®ã¯ã©ãŠã掻çšãæ¯æŽããŠããŸãã åžžäž å€§å² (Tokoyo Hiroshi) AWS Annapurna Labs ã®ãœãªã¥ãŒã·ã§ã³ã¢ãŒããã¯ãã§ãã Annapurna Labs ãæäŸãã AWS TrainiumãInferentia ã®æè¡æ¯æŽã«æ³šåããŠããŸãã
1. ã¯ããã« ããã«ã¡ã¯ããã©ãããã©ãŒã ãšã³ãžãã¢ãªã³ã°ããŒã ã«æå±ããå°æ³ïŒ @naotoko_ ïŒã§ãã æ¬èšäºã¯ãåããŒã ã®åŸ³å¯ïŒ @yannKazu1 ïŒãå·çããã æ¶ããã©ã³ããŒã®èŠ³æž¬åºç€ãã©ãéžãã ã â Datadogã»ãããŒãžãã»OSS ãæéäœç³»ã§æ¯ã¹ãŠ Loki + Prometheus ã«æ±ºãã話 ãã®æ¬çªç°å¢ãžã®å°å
¥ç·šã§ããEKS Auto Mode ã§ãã¹ãããŠãã Self-hosted Runner ã®ç£èŠåºç€ã« Grafanaã»Prometheusã»Lokiã»Alloy ãå°å
¥ããéã«ããã£ããã€ã³ããšè§£æ±ºçããåã³ã³ããŒãã³ãããšã«ãäŒãããŸãã Grafanaã»Prometheusã»Lokiã»Alloy ãã©ã®ãããªæ§æã§å®è£
ãããã¯ãäžèšããã°ã®ã å®è£
ïŒã©ãçµãã ã ããã芧ãã ããã 2. Alloy - ã¡ããªã¯ã¹ãéè€ããã»åããªã 2-1. DaemonSet ã®éè€ scrape åé¡ Alloy 㯠DaemonSet ã§ãããã€ããŠãããå
šããŒãã«1ã€ã〠Pod ãç«ã¡äžããæ§æãšãªã£ãŠããŸãã ããã©ã«ãã®ãŸãŸäœ¿çšãããšãAlloy ã¯ã¡ããªã¯ã¹åéæã«ã¯ã©ã¹ã¿ãŒå
šäœãã scrape 察象ã®äžèЧãååŸãããã®ãªã¹ãã«å¯ŸããŠå®æçã« HTTP ãªã¯ãšã¹ããéããŸããDaemonSet ã®å Pod ãããããç¬ç«ããŠãããå®è¡ãããããå
š Pod ãåãã¿ãŒã²ããäžèЧãååŸããŠããŸããŸãã ããšãã° kube-state-metrics ã ARC controller ã®ããã«ã¯ã©ã¹ã¿ãŒã«1ã€ãããªããšã³ããã€ã³ãã¯ãå
šãŠã® Alloy Pod ãåããšã³ããã€ã³ãã scrape ãã«è¡ãããããŒãæ°åéè€ããŸããããã«ãnode-exporter ã kubelet ã®ããã«ããŒãããšã«ååšãããšã³ããã€ã³ãã§ããå Alloy Pod ãã¯ã©ã¹ã¿ãŒå
šããŒãåã®ãšã³ããã€ã³ããçºèŠããŠå
šãŠ scrape ãããããåæ§ã«ããŒãæ°åéè€ããŸãããã®çµæãå
šãŠã® scrape 察象ã§åäžã®ã¡ããªã¯ã¹ãéè€ã㊠Prometheus ã«éãããŠããŸããŸãã å Alloy Pod ãããããç¬ç«ã«ã¯ã©ã¹ã¿ãŒå
šäœã®ã¿ãŒã²ããäžèЧãååŸãããããã¯ã©ã¹ã¿ãŒã«1ã€ãããªã kube-state-metrics ããããŒãããšã«ååšãã kubelet ããå
š Pod ãã scrape ãããã ãªããã°åéã§ã¯åãåé¡ãèµ·ããªãã ãã°åéã¯å Alloy Pod ãããŒãã®ããŒã«ã«ãã¡ã€ã«ïŒ /var/log/pods/ ïŒãèªãæ¹åŒã§ããããŒãAã® Alloy ã¯ããŒãAã®ãã°ã ããããŒãBã® Alloy ã¯ããŒãBã®ãã°ã ããèªããããPod éã§ããŒã¿ãéè€ããŸãããèªã察象ãããŒãåäœã§èªç¶ã«åå²ãããŠãããããã¡ããªã¯ã¹ã®ãããªéè€ã®åé¡ãèµ·ããŸããã§ããã 解決çïŒAlloy ã®ã¯ã©ã¹ã¿ãªã³ã°ãæå¹åãã Alloy ã«ã¯ã¯ã©ã¹ã¿ãªã³ã°æ©èœããããPod å士ãã¯ã©ã¹ã¿ãŒã圢æã㊠scrape 察象ãèªåçã«åæ
ããŸããïŒè©³ããä»çµã¿ã¯å²æïŒ Alloy Pod å士ãã¯ã©ã¹ã¿ãŒãçµã¿ãåã¿ãŒã²ããã®æ
åœããããã1ã€ã® Pod ã«æ±ºãããæ
åœã¯ããã·ã¥ã§æ±ºãŸããããå³ã®ããã«èªåãšåãããŒãäžã®ã¿ãŒã²ãããæ
åœãããšã¯éããªãã èšå®ã¯2段éå¿
èŠã§ãããŸã Helm chart ã® values ã§ã¯ã©ã¹ã¿ãªã³ã°ãæå¹åããDaemonSet ã®å
š Pod ãã¯ã©ã¹ã¿ãŒã圢æããããã«ããŸãã alloy : clustering : enabled : true ãã®ããã§ãåæ
ããããå prometheus.scrape ãããã¯ã« clustering { enabled = true } ãä»ããŸãã prometheus.scrape "kube_state_metrics" { targets = [...] clustering { enabled = true } forward_to = [prometheus.remote_write.default.receiver] } 泚æç¹ãšããŠããã®2ã€ã¯ã»ããã§åããŠæ©èœããŸããHelm values åŽãæå¹åããã« prometheus.scrape åŽã ãæžããŠã no-opïŒäœãããªãïŒã«ãªããéã« Helm values åŽã ãæå¹åããŠã clustering ãããã¯ãä»ããŠããªã scrape ã³ã³ããŒãã³ãã¯åŸæ¥ã©ããå
š Pod ãç¬ç«ã« scrape ãç¶ããŸãã 2-2. kubelet scrape ã® InternalIP å¯Ÿå¿ Grafana ã§ã¡ããªã¯ã¹ã確èªãããš Pod ã® CPUã»ã¡ã¢ãªã No data ã«ãªã£ãŠããŸããã調ã¹ããš kubelet ã® scrape ãå
šããŒãã§å€±æããŠããŸããã åå 㯠Alloy ã __meta_kubernetes_node_name ïŒããŒãåïŒã scrape å
ã®ã¢ãã¬ã¹ãšããŠäœ¿ã£ãŠããããã§ããEKS Auto Mode ã§ã¯ããŒãåãã€ã³ã¹ã¿ã³ã¹ ID ã«ãªãããåå解決ã§ãããå
šããŒãã§ã¿ã€ã ã¢ãŠãããŠããŸã£ãŠããŸããã éåžžã® EKS ã§ã¯ããŒãå㯠ip-xxx-xxx-xxx-xxx.ap-northeast-1.compute.internal ã®ãããªãã©ã€ããŒã DNS åã«ãªãããã __meta_kubernetes_node_name ã§ãåå解決ã§ããŸããäžæ¹ EKS Auto Mode ã§ã¯ i-xxxxxxxxxxxxxxxxx ã®ãã㪠EC2 ã€ã³ã¹ã¿ã³ã¹ ID ãããŒãåã«ãªããããåå解決ã§ããªããšããç¶æ³ã§ããã 解決çïŒInternalIP ã䜿ã __meta_kubernetes_node_address_InternalIP ã«åãæ¿ããããšã§ãããŒãåã§ã¯ãªã IP ã¢ãã¬ã¹ã§çŽæ¥ã¢ã¯ã»ã¹ããããã«ããŸããã rule { source_labels = ["__meta_kubernetes_node_address_InternalIP"] regex = "(.+)" replacement = "${1}:10250" target_label = "__address__" } 3. Prometheus - OOM ãšæ¬ æãšã®æŠã 3-1. OOM åé¡ å°å
¥åŸãPrometheus ãµãŒããŒã OOMKilled â åèµ·åãç¹°ãè¿ãããã«ãªããŸãããåå ã¯1ã€ã§ã¯ãªããè€æ°ãéãªã£ãŠããŸããã åå 1ïŒAlloy ãš Prometheus ã®äºéåã蟌㿠Prometheus ã® Helm chart ã¯ããã©ã«ãã§ããã€ãã® scrape jobïŒ kubernetes-nodes-cadvisor ã» kubernetes-pods çïŒãæå¹ã«ãªã£ãŠããŸããä»åã®æ§æã§ã¯ Alloy ãäžå
çã« scrape ã㊠Prometheus ã« remote_write ããèšèšã®ãããPrometheus èªèº«ã® scrape job ãš Alloy ã® remote_write ã§åãã¡ããªã¯ã¹ã2çµè·¯ã§å
¥ã£ãŠããŸããã ãã㯠Alloy ã scrape ããã¿ãŒã²ãããšéè€ããŠãã chart ããã©ã«ãã® scrape job ãç¡å¹åããããšã§è§£æ±ºããããšãã§ããŸãã scrapeConfigs : kubernetes-nodes : false kubernetes-nodes-cadvisor : false kubernetes-pods : false # ... åå 2ïŒcAdvisor ã®é« cardinality ã©ãã« cAdvisor 㯠kubelet ã«çµã¿èŸŒãŸããŠãããã³ã³ããã®ãªãœãŒã¹äœ¿çšéãåéããã³ã³ããŒãã³ãã§ããAlloy 㯠kubelet ã® /metrics/cadvisor ãšã³ããã€ã³ãã scrape ããããšã§ãã®ã¡ããªã¯ã¹ãååŸããŠããŸããcAdvisor ã®ã¡ããªã¯ã¹ã«ã¯ image ã» name ãªã©ã®ã©ãã«ãä»ããŠãããåæçšŒåããã©ã³ããŒæ°ãå€ãã»ã© cardinalityïŒç³»åæ°ïŒãççºçã«å¢ããŠããŸããŸãã ããã§ãPrometheus ã«éãåã®æ®µéã§ Alloy åŽã§ã¡ããªã¯ã¹ããã°èª¿æ»ã«äžèŠãªã©ãã«ã drop ããããšã§ãPrometheus ã«éãããç³»åæ°ãæ¥µåæžããããã«ããŸããã ã¡ãªã¿ã«ã image ã¯ã³ã³ããã€ã¡ãŒãžåã name ã¯ã³ã³ããã©ã³ã¿ã€ã äžã®ã³ã³ãã IDïŒcontainerd ã§ã¯ 64 æ¡ã® 16 鲿ååïŒãå
¥ããŸãã name ã¯ã³ã³ãããèµ·åãããã³ã«ç°ãªãå€ã«ãªãããã»ãŒãŠããŒã¯ã§ãããéèšãçµã蟌ã¿ã«ã¯äœ¿ããªããšå€æãããã®2ã€ã®ã©ãã«ã drop ããŸãããäžæ¹ã§ãã¡ããªã¯ã¹ã®ã¯ãšãªã§å®éã«äœ¿ã namespace ã» pod ã» container ãšãã£ãã©ãã«ã¯æ®ããŠããŸããç°å¢ã«å¿ããŠã調æ»ã«äœ¿ããªãã©ãã«ã¯å¯èœãªç¯å²ã§ drop ããããã«ããã®ãç¡é£ã§ãã rule { regex = "image|name" action = "labeldrop" } ãã ãã id ã©ãã«ã ãã¯æ®ãå¿
èŠããããŸããcAdvisor ã¯åãã¡ããªã¯ã¹åã§è€æ°ã®ã³ã³ããã®æ
å ±ãåéããŠããããããããåºå¥ããããã« id ã©ãã«ã䜿ãããŠããŸãã id ã drop ãããšç°ãªãã³ã³ããã®ã¡ããªã¯ã¹ãã©ãã«ã»ããäžã§åäžç³»åãšããŠæ±ãããŠãããŸããŸãããã®çµæãåãã¿ã€ã ã¹ã¿ã³ãã«è€æ°ã®ãµã³ãã«ãå±ãã Prometheus ã duplicate sample for timestamp ãšã©ãŒãè¿ããŠããŸãããæ³šæãå¿
èŠã§ãã åå 3ïŒHead Block ã®ã¡ã¢ãªåžžé§ Prometheus ã¯ã¡ããªã¯ã¹ãåãåããšãæžã蟌ã¿å¹çã®ããã«ãŸãã¡ã¢ãªäžã«äžæçã«è²¯ããŸããããã Head Block ã§ãã äžå®æéãçµã€ãšã¡ã¢ãªãã EBS äžã®ãã¡ã€ã«ã«æžãåºãããŸããããã©ã«ãã§ã¯ Head Block ã3æéåïŒmin-block-duration ã®1.5åïŒã«éããæç¹ã§å€ã2æéåããããã¯ãšããŠæžãåºãããŸããã€ãŸãåžžã«1ã3æéåã®ããŒã¿ãã¡ã¢ãªã«æ®ãç¶ãããããã¡ããªã¯ã¹ã®éãå€ãã»ã©ã¡ã¢ãªäœ¿çšéãå¢ããŸãã storage.tsdb.min-block-duration ã 2h ãã 30m ã«ççž®ããããšã§ãã®ãäžæçã«è²¯ããæéããçãããããé »ç¹ã« EBS ã«æžãåºãããšã§ã¡ã¢ãªåžžé§éãåæžããŸããããã®ä»£ããã以åã¯ã¡ã¢ãªããååŸã§ããŠããããŒã¿ã EBS ããååŸãããããã¯ãšãªãé
ããªããã¬ãŒããªãããããŸããSelf-hosted Runner ã®ç£èŠãšããæ§è³ªäžã確èªãããã®ã¯åºæ¬çã«çŽè¿ã®ç¶æ³ã§ããããã30åããå€ãããŒã¿ã®ååŸãå€å°é
ããªã£ãŠãåé¡ãªããšå€æããŠããŸãã ãªãã storage.tsdb.min-block-duration 㯠--help ã«ã衚瀺ãããªã hidden ãã©ã°ã§ãå
¬åŒã«ã¯ããã¹ãçšéããšãããŠããŸããæåãçè§£ããããã§å©çšããŠãã ããã extraArgs : storage.tsdb.min-block-duration : 30m 3-2. out-of-order ãµã³ãã«åé¡ Alloy ã¯ã©ã¹ã¿ãªã³ã°ãæå¹åããŠãããPrometheus ã« out of order sample ãšã©ãŒã倧éã«åºãããã«ãªããŸããã Alloy ã®ã¯ã©ã¹ã¿ãªã³ã°ã¯ Pod ã®å¢æžãããªã¬ãŒã«ã¿ãŒã²ããã®åé
åãè¡ããŸããSelf-hosted Runner ã¯ãžã§ãã®å¢æžã«å¿ããŠããŒããé »ç¹ã«ã¹ã±ãŒã«ãããããAlloy ã® Pod ã墿žããåé
åãé »ç¹ã«èµ·ããŸããAlloy ã®ã¯ã©ã¹ã¿ãªã³ã°ã«ããã¿ãŒã²ããã®åæ
㯠eventually consistent ãªã¢ãã«ã§ãããããåé
åã®åŒãç¶ãã®ã¿ã€ãã³ã°ã«ãã£ãŠã¯åãã¿ãŒã²ãããäžæçã«2ã€ã® Pod ãã scrape ãããããšããããŸãïŒgrafana/alloy ã® issue #1611ã»#2348 ã§ãå ±åãããŠããæ¢ç¥ã®æåã§ãïŒããã®ç¶æ
ã§åãæç³»åã®ãµã³ãã«ã2ã€å±ããšãåŸçã®ãµã³ãã«ã®ã¿ã€ã ã¹ã¿ã³ããå
çããå€ãå ŽåããããŸãããããš Prometheus ã¯ããã out-of-order ãšããŠæåŠãã該åœã®ãµã³ãã«ãæšãŠãŠããŸããŸãããã®çµæã Grafana ã§ç¢ºèªã§ããã¡ããªã¯ã¹ã«æ¬ æãçããŠããŸããŸãã 解決çïŒout-of-order time window ãèšå®ãã out_of_order_time_window ãèšå®ããããšã§ãæå®ããæéå
ã®éå»ã®ã¿ã€ã ã¹ã¿ã³ããåãå
¥ããããã«ãªããŸããåŒç€Ÿã®ç°å¢ã§ã¯ãæçµçã« 10m ã«èœã¡çããŸãããout-of-order ã®ãµã³ãã«ãä¿æããããã®ã¡ã¢ãªãè¥å¹²å¢ããŸããã storage.tsdb.min-block-duration ã 30m ã«ççž®ã㊠Head Block ã®ã¡ã¢ãªãç¯çŽã§ããŠãããããç¹ã«åé¡ãªããšå€æãã10m ã«èšå®ããŠããŸãã tsdb : out_of_order_time_window : 10m 4. Loki - chart 移管ã®çœ ãšèµ·åããªã Pod 4-1. Helm chart ãªããžããªç§»ç®¡ã®çœ Loki ã® Helm chart ã¯ããšããš grafana/helm-charts ïŒ https://grafana.github.io/helm-charts ïŒã§é
åžãããŠããŸããã ããã chart v6.55.0 ãæåŸã« OSS Loki åãã® chart 㯠grafana-community/helm-charts ïŒ https://grafana-community.github.io/helm-charts ïŒãžãã©ãŒã¯ãããv7.0.0 以é㯠community åŽãããªãªãŒã¹ãããŠããŸããåŸæ¥ã®ãªããžããªã«æ®ã£ã loki chart 㯠Grafana Enterprise LogsïŒGELïŒåãã®ã¡ã³ããã³ã¹å°çšã«ãªããŸããã ãã®ããã grafana/helm-charts ã䜿ãç¶ãããšãæ°ã¥ããªããã¡ã« GEL åãã® chart ãåŒããŠããŸããŸãã åãããŠä»¥äžã®å€æŽãããã®ã§ã泚æãå¿
èŠã§ãã ãã©ãŒã¯åŸã¯ã¡ãžã£ãŒããŒãžã§ã³ã®äžããããŒã¹ãéåžžã«éãïŒv7.0.0 ããæ°ã¶æã§ v17.xãå·çæç¹ã®ææ°ã¯ v18.xïŒ deploymentMode ã®å€ã SingleBinary â Monolithic ã«ãªããŒã ïŒchart v12.0.0ïŒããªã values ã®ããŒã¯ singleBinary ã®ãŸãŸå€ãã£ãŠããŸãã resource "helm_release" "loki" { repository = "<https://grafana-community.github.io/helm-charts>" # ãããå€ãã£ã chart = "loki" version = "17.1.6" ... } # chart v12.0.0 以é deploymentMode : Monolithic # SingleBinary ããå€æŽ 4-2. StorageClass ãèªåäœæãããªã Loki ã® Pod ãã¹ã±ãžã¥ãŒã«ãããã以äžã®ãšã©ãŒãåºãŠããŸããã eks-auto-mode/compute Failed to schedule pod, unbound pvc must define a storage class éåžžã® EKS ã§ã¯ EBS CSI driver addon ãããã©ã«ãã® StorageClass ãèªåäœæããŸãããEKS Auto Mode ã§ã¯äœæãããŸãããStorageClass ãäœæãããŠããªããšãPVC ããã€ã³ãã§ãã Pod ãèµ·åããŸããã ãã®ãããStorageClass ãæåã§äœæããLoki ã® PVC ã«æå®ããããšã§è§£æ±ºããŸããã # StorageClass ã®æåäœæ apiVersion : storage.k8s.io/v1 kind : StorageClass metadata : name : auto-ebs-sc provisioner : ebs.csi.eks.amazonaws.com volumeBindingMode : WaitForFirstConsumer parameters : type : gp3 encrypted : "true" # loki.yaml singleBinary : persistence : storageClass : auto-ebs-sc 5. Grafana - ãããã€æŠç¥ãšã³ãŒã管ç 5-1. EBSïŒRWOïŒãš RollingUpdate ã®çžæ§åé¡ Grafana ã helm upgrade ãããšããPod ãæ°ãããªãã¯ããå»¶ã
ãš Pending ã®ãŸãŸè çããŸãããåå 㯠EBS ã®å¶çŽã§ãã EBS 㯠RWOïŒReadWriteOnceïŒã®ããã1ã€ã® Node ã«ããåæã«ã¢ã¿ããã§ããŸãããããã©ã«ãã® RollingUpdate ã¯æ°ãã Pod ãèµ·åããŠããå€ã Pod ãèœãšãé çªãªã®ã§ãæ°æ§ã® Pod ãåã PVC ãåãåã£ãŠ Multi-Attach error ãçºçããŸãã 解決çïŒRecreate æŠç¥ã«å€æŽãã Recreate ã«ãããšãæ§ Pod çµäº â EBS detach â æ° Pod èµ·åããšããé çªã«ãªããŸããreplicas=1 ã®æ§æãªã®ã§æŽæ°æã«çæéã®ããŠã³ã¿ã€ã ãçºçããŸãããç£èŠåºç€ãšããç¹æ§ãèžãŸããŠç¬æã¯èš±å®¹ãã Recreate ã«èšå®ããŠããŸãã deploymentStrategy : type : Recreate 5-2. Dashboardã»Alerting ã®ã³ãŒã管ç ãªã Terraform provider ã§ã¯ãªã Helm values ã§ç®¡çããã Grafana ã®ããã·ã¥ããŒããšã¢ã©ãŒãã«ãŒã«ãã³ãŒã管çããæ¹æ³ãšããŠã grafana Terraform provider ãäœ¿ãæ¹æ³ããããŸãããããä»å㯠Helm valuesïŒYAML/JSONïŒã§ã®ç®¡çãéžã³ãŸããã çç±ã¯ã·ã³ãã«ã§ãGrafana ã® UI ã§ããã·ã¥ããŒããã¢ã©ãŒãã«ãŒã«ãäœã蟌ãã ããšããã®ãŸãŸ JSON/YAML ã§ãšã¯ã¹ããŒãã㊠Helm values ã«è²Œãä»ããã ãã§ã³ãŒã管çã§ããããã§ããTerraform ã®ãªãœãŒã¹å®çŸ©ã«èœãšã蟌ãæéãäžèŠã§ãUI ã§ç¢ºèªããªããäœã£ããã®ããã®ãŸãŸåæ ã§ããŸãã Grafana ã® Helm chart 㯠provisioning ã®ä»çµã¿ãæã£ãŠãããvalues ã«æžããããã·ã¥ããŒãå®çŸ©ãã¢ã©ãŒãã«ãŒã«ãèµ·åæã«èªåã§èªã¿èŸŒã¿ãŸãã YAML 管çã«ããå¶çŽïŒåé€ã¯ deleteRules ã«æç€ºãå¿
èŠ ã¢ã©ãŒãã«ãŒã«ã YAML ããåé€ããŠããGrafana äžã®ã«ãŒã«ã¯æ¶ããŸãããã¢ã©ãŒãã«ãŒã«ã® provisioning ã¯è¿œå ã»æŽæ°ããŠãããŸãããåé€ã¯è¡ããŸããã ãã㯠Grafana ã®èšèšäžã®å®å
šçã§ããfile-based provisioning ã¯ã¹ããŒãã¬ã¹ã§ãGrafana ã¯ããã® YAML ãã«ãŒã«ã®å
šéã§ããããšããä¿èšŒãæãŠãŸãããè€æ°ãã¡ã€ã«ããåæã«ããããžã§ãã³ã°ã§ããèšèšäžãããã¡ã€ã« A ã«ãªãã«ãŒã«ãããã¡ã€ã« B ã§ç®¡çãããŠããããããããGrafana ã«ã¯ã©ã®ãã¡ã€ã«ãäœã管çããŠããã倿ã§ããŸããããŸãèšå®ãã¡ã€ã«ã®ãã°ãããŠã³ã倱æã§äžæçã«ãã¡ã€ã«ãèªããªããªã£ããšããèªååé€ã ãšã¢ã©ãŒãã«ãŒã«ãå
šæ¶ããããªã¹ã¯ããããŸããTerraform ã管ç察象ã state ãã¡ã€ã«ã§è¿œè·¡ããŠããããèªååé€ã§ããã®ãšå¯Ÿç
§çã§ãfile-based provisioning ã«ã¯ãã® state æŠå¿µããªããããåé€ã®æå³ãæç€ºããä»çµã¿ãšã㊠deleteRules ãçšæãããŠããŸãã ã«ãŒã«ãåé€ããã«ã¯ä»¥äžã®ããã« UID ãæå®ããŸãã alerting : rules.yaml : groups : - ... deleteRules : - orgId : 1 uid : hoge-alert # åé€ãããã«ãŒã«ã® UID ãæç€º 6. å°çš NodePool ãžã®åé¢ Prometheusã»Lokiã»Grafanaã»Alloy ãšãã£ã芳枬æ§ã³ã³ããŒãã³ãããRunner ãšåã NodePool ã«æ··åšãããŠãããš2ã€ã®ãªã¹ã¯ããããŸãã 1ã€ç®ã¯ãPrometheus ã® OOM ã®ãããªèŠ³æž¬æ§ã³ã³ããŒãã³ãã®ãªãœãŒã¹ãã¬ãã·ã£ãŒãåãããŒãäžã®ã©ã³ããŒã®åäœã«æªåœ±é¿ãäžãããªã¹ã¯ã§ãã 2ã€ç®ã¯ãKarpenter ã® consolidation ã«å·»ã蟌ãŸãããªã¹ã¯ã§ããrunner ã䜿ã NodePool ã¯æ¥åæé垯㮠consolidation ãç¡å¹ã«ããŠãããæ©æã®éãããæé垯ã®ã¿æå¹ã«ãªãèšå®ã«ããŠããŸãããã®æé垯㫠consolidation ãèµ°ã£ããšããåã NodePool ã«ãã芳枬æ§ã³ã³ããŒãã³ãã® Pod ãå¥ããŒãã«ç§»åãããããã¡ããªã¯ã¹ããã°ãæ¬ æãããªã¹ã¯ããããŸãã å°çš NodePool ã«åé¢ããããšã§ããããã®åé¡ãã©ã³ããŒããåãé¢ããŸãã Karpenter ã§ dedicated=observability:NoSchedule ã® taint ãä»ããå°çš NodePool ãçšæããPrometheusã»Lokiã»Grafana ãªã©ã® Deployment / StatefulSet ç³»ã³ã³ããŒãã³ãã«å¯Ÿå¿ãã toleration ãš nodeSelector ãèšå®ããŸããããªããDaemonSet ã§ãã Alloy ãš node-exporter ã¯å
šããŒãã§åããå¿
èŠããããã toleration ã®ã¿ä»äžããŠããŸãã # NodePool taints : - key : dedicated value : observability effect : NoSchedule # åã³ã³ããŒãã³ã tolerations : - key : dedicated value : observability effect : NoSchedule nodeSelector : karpenter.sh/nodepool : observability ç£èŠçšã³ã³ããŒãã³ãã«ã€ããŠãã©ã³ããŒãšåãããarm64 on-demand ã€ã³ã¹ã¿ã³ã¹ã䜿çšããããã«ããŠããŸãã ãã ããã€ã³ã¹ã¿ã³ã¹ã¿ã€ãã®æå®ãçããš Karpenter ãã³ã³ãã¥ãŒãæé©åã€ã³ã¹ã¿ã³ã¹ïŒc6g.large / 4GB RAMïŒãéžãã§ããŸããLoki ã Grafana ã OOMKill ããããšããåé¡ãèµ·ããŠããŸããŸããobservability ã¹ã¿ãã¯ã®ã¡ã¢ãªäœ¿çšéãå®éã«ç¢ºèªããããã§ãNodePool ã® requirements ã« 8GB 以äžã®ã€ã³ã¹ã¿ã³ã¹ãæå®ãããªã©ããã¡ããšã¡ã¢ãªèŠä»¶ã«åã£ãã€ã³ã¹ã¿ã³ã¹ãéžã°ããããã«æ³šæãå¿
èŠã§ãã requirements : - key : eks.amazonaws.com/instance-memory operator : Gt values : [ "7168" ] # 8GB ä»¥äž 7. ãŸãšã Grafanaã»Prometheusã»Lokiã»Alloy ã Self-hosted Runner ã®ç£èŠåºç€ãšããŠå°å
¥ããã«ããããåã³ã³ããŒãã³ãã§æ§ã
ãªããããã€ã³ãããããŸãããç¹ã« Prometheus ã® OOM ã¯è€æ°ã®åå ãéãªã£ãŠããã1ã€è§£æ±ºããŠã次ã®åé¡ãåºãŠãã圢ã§å¯Ÿå¿ã«æéãããããŸããããŸã EKS Auto Mode ã«ã¯éåžžã® EKSïŒãããŒãžãããŒãã°ã«ãŒãçïŒãšèšå®ã«ãããŠç°ãªãéšåããããããEKS Auto Mode ã«åããŠè§Šãæ¹ãç§»è¡ãèããŠããæ¹ã¯ç¹ã«æ³šæããŠãã ããã åãæ§æãæ€èšããŠããæ¹ã®åèã«ãªãã°å¹žãã§ãã
1. ã¯ãã㫠以åã以äžã®èšäºã§AWS AI Leagueã«ã€ããŠç޹ä»ããŸããã https://zenn.dev/nttdata_tech/articles/a907eb00cbbe4b AWS AI Leagueã¯ãç«¶æåœ¢åŒã§çæAIã®æè¡ãåŠã¶ããã°ã©ã ã§ããäž»ã«ã次ã®2ã€ã®ããŒããçšæãããŠããŸãã Amazon SageMaker AIãå©çšããã¢ãã«ã®ãã¡ã€ã³ãã¥ãŒãã³ã° Amazon Bedrock AgentCoreãå©çšããAgentic AI Agentic AIãã£ã¬ã³ãžã§ã¯ãAIãšãŒãžã§ã³ãããããäžãç§»åããã³ã€ã³ã®ååŸã質åãžã®åçãªã©ãè¡ããªããã


























