
- TOP
- ã¿ã°äžèЧ
- Kubernetes
Kubernetes
ã€ãã³ã
該åœããã³ã³ãã³ããèŠã€ãããŸããã§ãã
ãã¬ãžã³
æè¡ããã°
ã¯ããã« ãªã¢ã«ã¿ã€ã æ§ã®é«ããã«ããã¬ã€ã€ãŒã²ãŒã ã¯ãè€æ°ã®ãã¬ã€ã€ãŒã®å
¥åãéçŽããã²ãŒã ã®ç¶æ
ãäžè²«ãããŠå
šå¡ãžé
ä¿¡ããä»çµã¿ãå¿
èŠãšããŸãããã®ä»çµã¿ã®äžã€ã«ãæš©åšãããµãŒããŒãã²ãŒã ã®ç¶æ
ã管çããå°çšã²ãŒã ãµãŒã㌠(Dedicated Game Server) æ¹åŒããããŸãã察æŠã®å
¬å¹³æ§ãããŒã察çãå€äººæ°ã®åææ¥ç¶ãæ±ããããã¿ã€ãã«ã§ã¯ãå°çšã²ãŒã ãµãŒããŒæ¹åŒãåºãæ¡ãããŠããŸãããã®æ¹åŒã§ã¯ããããã¡ã€ãã³ã°ãã²ãŒã ãµãŒããŒã®å²ãåœãŠãã»ãã·ã§ã³çµäºåŸã®ãµãŒããŒååãšãã£ãã©ã€ããµã€ã¯ã«ç®¡çãå¿
èŠã«ãªãããã®ãªãŒã±ã¹ãã¬ãŒã·ã§ã³åºç€ãã©ãæ§ç¯ã»éçšããããèšèšäžã®å€§ããªããŒããšãªããŸãã AWS ã§å°çšã²ãŒã ãµãŒããŒããã¹ãããäž»ãªéžæè¢ã¯ããããŒãžããµãŒãã¹ã® Amazon GameLift Servers ãå©çšãããã Amazon Elastic Compute Cloud (EC2) ã Amazon Elastic Container Service (ECS) ã Amazon Elastic Kubernetes Service (EKS) äžã§ã»ã«ããã¹ããããã® 2 ã€ã§ããGameLift Servers ã¯ã€ã³ãã©æ§ç¯ã»éçšã®å€§éšåãæ
ããŸãããèŠä»¶ã«ãã£ãŠã¯ããæè»ãªå¶åŸ¡ãå¿
èŠãªå Žåãããã Agones ã®å©çšãéžæè¢ã®äžã€ãšãªããŸããAgones ã¯ãªãŒãã³ãœãŒã¹ã®å°çšã²ãŒã ãµãŒããŒãã¹ãã£ã³ã°åã補åã§ãAmazon EKS äžã«ãããã€ã§ããæè»æ§ã享åãã€ã€å°çšã²ãŒã ãµãŒããŒã®æ§æãç°¡çŽ åã§ããŸããããã Kubernetes åºç€ã®éçšã¯é¿ãããããAgones ã®éçšã«ã¯äžå®ã®è² è·ãååšããŸããã 2024 幎 12 æã«äžè¬æäŸãå§ãŸã£ã Amazon EKS Auto Mode ã¯ãKubernetes ã®æ§ç¯ã»éçšè² æ
ãäœæžããæ©èœã§ããAgones ã®åºç€ãšãªã EKS ã¯ã©ã¹ã¿ã§ EKS Auto Mode ãå©çšã§ããã°ãå°çšã²ãŒã ãµãŒããŒãæè»ãã€ç°¡çŽ ã«æ§ç¯ã»éçšã§ããŸãããã ã EKS Auto Mode ã«ã¯ããã€ãã®å¶çŽããããŸãã æ¬èšäºã§ã¯ãEKS Auto Mode ã®å¶çŽã Agones ã®ç¹æ§ãé»å®³ããªãããã«ããããã®èšèšæ¹éã玹ä»ããŸãã Agones ãªãœãŒã¹ã®æŠèŠ æ¬èšäºã®å
容ã«é¢é£ãã Agones ã«ã¹ã¿ã ãªãœãŒã¹ã«ã€ããŠèª¬æããŸãã GameServerïŒ ãã¬ã€ã€ãŒãæ¥ç¶ããã²ãŒã ãµãŒããŒããã°ã©ã ãåäœãã Pod ã«ããããªãœãŒã¹ã§ããç¶æ
ãæã¡ãReady ç¶æ
ã¯ã²ãŒã éå§å¯èœã§åŸ
æ©ããŠããç¶æ
ãAllocated ç¶æ
ã¯ã²ãŒã ãã¬ã€ã®ããã«ãªãœãŒã¹ã確ä¿ãããç¶æ
ã衚ããŸãã FleetïŒ GameServer ã®ãŸãšãŸãã管çããããã®ãªãœãŒã¹ã§ããåžžã«èµ·åããŠããã¹ã GameServer ãªãœãŒã¹ã®æ°ãæå®ã§ããFleet Autoscaler ãªãœãŒã¹ã«ããã¹ã±ãŒãªã³ã°å¶åŸ¡ãå¯èœã§ãã GameServer AllocationïŒ ã²ãŒã ãã¬ã€ã®ããã« GameServer ãªãœãŒã¹ãéžæã確ä¿ããæäœãè¡ããªãœãŒã¹ã§ãããã®æäœã«ããéžæããã GameServer ãªãœãŒã¹ã¯ Ready ç¶æ
ãã Allocated ç¶æ
ãšãªããŸãã ã³ã³ãããŒã©ç³»ãªãœãŒã¹ïŒ Agones ã®æ©èœãå®çŸã»å¶åŸ¡ããããã®ãªãœãŒã¹çŸ€ã§ãã EKS Auto Mode ãš Agones ã®å
±åã«ãããèª²é¡ EKS Auto Mode ã¯ãã¯ã©ã¹ã¿ã®ããŒã管çã AWS ã®è²¬ä»»ç¯å²ã§èªååããŸããäŸãã° Karpenter ãããŒã¹ãšããããŒãã®ããããžã§ãã³ã°ã»ã¹ã±ãŒãªã³ã°ã»å
¥ãæ¿ããBottlerocket ã«ããæé©åããã OSãåçš®ã¢ããªã³ã®ãããŒãžã管çãšãã£ãæ©èœã«ãããKubernetes ã®éçšã倧å¹
ã«ç°¡çŽ åã§ããŸãã äžæ¹ããŠãŒã¶ãŒåŽã§èšå®ã§ããç¯å²ãçãŸããšãããã¬ãŒããªãããããŸããAgones ã®åºç€ãšããŠç¹ã«èæ
®ãå¿
èŠãªã®ã¯ãEKS Auto Mode ã以äžã®ç®çã®ããã«èªåã§è¡ãããŒãã®äžæã§ãã ConsolidationïŒ çšŒåäžã® Pod ãå°ãªãããŒããçµäºããå©çšçã®é«ãããŒããž Pod ãéçŽããŠã³ã¹ããæé©åããã DriftïŒ NodePool ã NodeClass ã®èšå®å€æŽãAMI ã®æŽæ°ãªã©ã«è¿œåŸããŠããŒããå
¥ãæ¿ããã ExpirationïŒ ã»ãã¥ãªãã£ç¶æã®ãããäžå®æéãçµéããããŒãã匷å¶çã«å
¥ãæ¿ããã InterruptionïŒ Spot Instance ã®äžæéç¥ãããŒããŠã§ã¢é害ãªã©ã«å¿ããŠããŒããçµäºããã â»EKS Auto Mode ã®é·æå®è¡ã¯ãŒã¯ããŒãã§ã®æ±ãã«ã€ããŠã¯ãbuilders.flash èšäºã Amazon EKS Auto Mode ã®ããŒãèªåæŽæ°ã Deep Dive ãã ããããããŠåç
§ãã ããã ãããã®äžæã§ãã察象ããŒãäžã® Pod ãž SIGTERM ãçºè¡ãããŸããAgonesäžã§åãããããªå°çšã²ãŒã ãµãŒããŒã«ã¯ããã¬ã€äžã®ã²ãŒã ã»ãã·ã§ã³ãäžæãããªãããšãã匷ãå¶çŽãããããã察çãããªããšãã¬ã€äžã®ã²ãŒã ãµãŒããŒã匷å¶çµäºããããªã¹ã¯ããããŸãã EKS Auto Mode ãš Agones ãå
±åãããèšå®ã®ãã€ã³ã ããã§ãAgones ã®ã³ã³ãããŒã©ç³»ãšã²ãŒã ãã¬ã€äžã®ãµãŒããŒããã°ã©ã ãããŒãã®äžæããä¿è·ããä»çµã¿ããæ¬¡ã® 6 ã€ã®ãã€ã³ãã§æ§æããŸãã以éãããããã®ãã€ã³ãã«ã€ããŠãå®éã®èšå®ãã¡ã€ã« ( nodepools.yaml / fleet.yaml / agones-values.yaml ) ãåŒçšããªãããå
·äœçãªå®è£
æ¹æ³ã説æããŸãã Point # 察象 å
容 Point 1 å
±é NodePool ã® expireAfter ã terminationGracePeriod ãé©åã«èšå®ãã Point 2 GameServer GameServer ã® Pod ã® terminationGracePeriodSeconds ãã²ãŒã ã®æå€§æç¶æéããé·ãå€ã«èšå®ãã Point 3 GameServer GameServer äžã§åããµãŒããŒã¢ããªã§ãé©åãªã·ã°ãã«ãã³ããªã³ã°ãå®è£
ãã Point 4 GameServer Eviction ãèš±å¯ããŠããŒãã®èªåæé©åãæ©èœããã ( spec.eviction.safe: Always ) Point 5 ã³ã³ãããŒã©ç³» Agones ã³ã³ãããŒã©ç³»ãš GameServer ãé
眮ããã NodePool ãåãã Point 6 ã³ã³ãããŒã©ç³» agones-allocator ãš agones-ping ã® PDB ãæå¹ã«ãã Point 1 (å
±é) ïŒNodePool ã® expireAfter / terminationGracePeriod ãé©åã«èšå®ãã EKS Auto Mode ã§ã¯ãããŒãã¯å¿
ããã€ãäžæãããŸããç¹ã« Expiration ã¯ããŒãçåæéæå€§21æ¥ã®å¶çŽããããå¿
ãçºçããŸãããã® expireAfter ã®å€ãšãããŒãã匷å¶çã«äžæããããŸã§ã®ç¶äºæéã§ãã terminationGracePeriod ããã¯ãŒã¯ããŒãç¹æ§ã«åãã㊠NodePool ã«èšå®ããããšãèµ·ç¹ã«ãªããŸãã NodePool ãšã¯ãKarpenter ããŒã¹ã®ããŒãç®¡çæ©èœã§ãPod ã®èŠæ±ã«å¿ããŠã€ã³ã¹ã¿ã³ã¹ãèªåçã«ããããžã§ãã³ã°ã»ã¹ã±ãŒãªã³ã°ããã¯ã©ã¹ã¿ãŒã®ã³ã³ãã¥ãŒããªãœãŒã¹ãæè»ã«ç®¡çããããã®ä»çµã¿ã§ãã èšå®ã«ããã£ãŠã¯ã次ã®é¢ä¿ãå®ãããã«ããŠãã ããã terminationGracePeriod ã¯ãåŸè¿°ãã Pod ã® terminationGracePeriodSeconds ããé·ãèšå®ãã (ããŒãã匷å¶åé€ãããåã«ãPod ã Graceful ã«çµäºãããããã«ãããã) ã expireAfter + terminationGracePeriod ã®åèšã¯æå€§ 21 æ¥ã«åãã (EKS Auto Mode ã®å¶çŽ) ã ãŸã disruption.budgets ã® nodes: "10%" ïŒNodePool Disruption Budgets: NDBïŒã«ãããåæã«äžæåŠçãžå
¥ãããŒãæ°ãå
šäœã® 10 % ã«å¶éããäžæäžæã®åœ±é¿ãæããŸãã consolidationPolicy: WhenEmpty ã«ã€ããŠã¯ Point 4 ã§è§ŠããŸãã Point 1 ãèžãŸãã NodePool ã®èšå®äŸã¯æ¬¡ã®éãã§ãïŒ nodepools.yaml ïŒã # GameServer çš NodePool apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: agones-worker spec: template: spec: # Point 1: ã²ãŒã ã»ãã·ã§ã³æéãèæ
®ããŠèšå® # terminationGracePeriod 㯠Pod ã® terminationGracePeriodSeconds ããé·ãããããš expireAfter: 336h # 14æ¥ terminationGracePeriod: 4h # expireAfter + terminationGracePeriod ã21æ¥(504h)ã«åãŸãããã« disruption: consolidationPolicy: WhenEmpty consolidateAfter: 30s budgets: - nodes: "10%" Point 2ïŒGameServerïŒïŒPod ã® terminationGracePeriodSeconds ãã²ãŒã æå€§æç¶æéããé·ãèšå®ãã ããŒããäžæãã Pod ã« SIGTERM ãçºè¡ãããŠãããPod ã匷å¶çµäºïŒSIGKILLïŒããããŸã§ã®ç¶äºã terminationGracePeriodSeconds ã§ãããã®å€ãã²ãŒã ã®æå€§æç¶æéããé·ãèšå®ããããšã§ãé²è¡äžã®ã²ãŒã ãéäžã§åŒ·å¶çµäºãããããšãé²ããŸãã terminationGracePeriodSeconds 㯠Pod ãªãœãŒã¹ã®èšå®å€ã§ãããAgones ã§ã¯ GameServer ãªãœãŒã¹å®çŸ©å
ã§èšå®ããããšã«ãªããŸãããŸã GameServer ã®å¶åŸ¡ã¯ Fleet ã§è¡ãããšãäžè¬çãªãããããã§ã¯ Fleet ã®èšå®äŸã次ã®éã瀺ããŸã ( fleet.yaml ) apiVersion: "agones.dev/v1" kind: Fleet metadata: name: simple-game-server-fleet spec: replicas: 3 template: spec: # ... (çç¥) template: spec: # Point 2: ã²ãŒã ã»ãã·ã§ã³æå€§æç¶æé + ãããã¡ããé·ãå€ãèšå® # äŸ: ã²ãŒã æå€§30min + ã¯ãªãŒã³ã¢ãã5min = 35min = 2100sec terminationGracePeriodSeconds: 2100 ã²ãŒã ã»ãã·ã§ã³ 1 ã€ã®æå€§æç¶æéã«ãã¯ãªãŒã³ã¢ããåŠçã®ãããã¡ãå ããå€ãèšå®ããŸããPoint 1 ã® terminationGracePeriod (ããŒãåŽã®ç¶äº) ãããã® terminationGracePeriodSeconds (Pod åŽã®ç¶äº) ããé·ãèšå®ããããšã§ã terminationGracePeriod > terminationGracePeriodSeconds > ã²ãŒã æå€§æç¶æé ã®é¢ä¿ãæç«ãããã¬ã€äžã®ã²ãŒã ãä¿è·ã§ããŸãã ãŸãšãããšã 以äžãæºããããã«åãã©ã¡ãŒã¿ãèšå®ããã°ããããšã«ãªããŸãã NodePool ã® terminationGracePeriod > Pod ã® terminationGracePeriodSeconds > ã²ãŒã ã®æå€§æç¶æé NodePool ã® expireAfter ã¯èªç± ( terminationGracePeriod ãšã®åèšã¯ 21 æ¥ä»¥äž) Point 3 (GameServer) ïŒé©åãªã·ã°ãã«ãã³ããªã³ã°ãå®è£
ãã Point 1ã»2 ã§ç¶äºæéã確ä¿ããŠããSIGTERM ãåãåã£ããµãŒããŒããã°ã©ã ãé©åã«æ¯ãèããªããã°ãã²ãŒã ãµãŒããŒã¯å®å
šã«çµäºã§ããŸãããGameServer äžã§åããµãŒããŒããã°ã©ã ã§ã¯ãSIGTERM ãåä¿¡ãããšãã«æ¬¡ã®åäœãå®è£
ããŸãã Allocated ç¶æ
(ã²ãŒã é²è¡äž) ïŒé²è¡äžã®ã²ãŒã ã®çµäºãåŸ
ã¡ãã¯ãªãŒã³ã¢ããåŠçãè¡ã£ãåŸã« Agones SDK ã® sdk.Shutdown() ãåŒã³åºãã Ready ç¶æ
(åŸ
æ©äž) ïŒæ°èŠ Allocation ãé²ãããã峿 sdk.Shutdown() ãåŒã³åºãã ãã®å®è£
ã«ãããããŒãäžæã®ã·ã°ãã«ãåããŠããé²è¡äžã®ã²ãŒã ãå®ãã€ã€ãåŸ
æ©äžã®ãµãŒããŒã¯éããã«éå»ã§ããŸãã Point 4 (GameServer) ïŒEviction ãèš±å¯ããŠããŒãã®èªåæé©åãæ©èœããã ( spec.eviction.safe: Always ) Point 1ã3 ã§ãã¬ã€äžã®ã²ãŒã ãä¿è·ã§ããããæ¬¡ã« Agones ã® spec.eviction.safe ã Always ã«èšå®ããEKS Auto Mode ã«ããããŒãã®èªåæé©åãæ©èœãããŸã ( fleet.yaml ) ã spec: template: spec: # Point 4: eviction.safe ã Always ã«èšå® # Karpenter ã® Drift/Consolidation/Expiration æã« Graceful 㪠Eviction ãèš±å¯ãã eviction: safe: Always spec.eviction.safe ã¯ãå GameServer ã® Pod ã«å¯Ÿã㊠Agones ãå
éšçã«äœæãã PDB ( maxUnavailable: 0% ) ãæå¹åãããã©ãããå¶åŸ¡ããŸããããã©ã«ãã® Never ã®å Žåãåžžã« eviction ããããã¯ããåããšãªããŸãã (衚㯠https://agones.dev/site/docs/advanced/controlling-disruption/ ãã) Never ã®ãŸãŸã§ã¯ããã® PDB ããã¹ãŠã® Eviction ããããã¯ããEKS Auto Mode ç°å¢ã§æ¬¡ã®åé¡ãçºçããŸãã Karpenter ã Consolidation / Drift ã詊ã¿ãŠã Pod ã« SIGTERM ãçºè¡ãããéå»ããªããããCordon ç¶æ
ã®ããŒããæ¢å Pod ããšæ®ãç¶ããã³ã¹ãå¹çãæªåããã Drift (AMI æŽæ°) ã PDB ã«ãããã¯ãããã»ãã¥ãªãã£ãããã®é©çšãé
å»¶ããã äžèšã®åäœã®çµæãè¿ãæé垯ã«èµ·åããå
šããŒãäžã® Pod ã Expiration ãŸã§çåãç¶ãããããã® Pod ãè¿ãæé垯ã«äžæã« SIGKILL ããããªã¹ã¯ãçããã Always ãèšå®ãããš Agones ã®ãã€ãã£ã PDB ãç¡å¹åãããEviction ãèš±å¯ãããŸããPod ã皌åäžã®ããŒãã§ã Drift ã«ãã AMI èªåæŽæ°ãšãExpiration æã® Graceful ãªããŒãå
¥ãæ¿ããæ©èœããããã«ãªããŸã (Never ã§ã¯ãããã PDB ã«ãããã¯ãããŸã) ããã¬ã€äžã®ã²ãŒã ã®ä¿è·ã¯ Point 1ã3 ã§ç¢ºä¿ããç¶äºæéãšã·ã°ãã«ãã³ããªã³ã°ãæ
ä¿ãããããAlways ã«ããŠãé²è¡äžã®ã²ãŒã ãå³åº§ã«åæãããããšã¯ãããŸããã Always ã®èšå®ã§ãå®å
šã«éçšããç®çã§ãPoint 1 ã§èšå®ãã consolidationPolicy: WhenEmpty (Pod ãæ®ã£ãŠããããŒãã§ã¯ Consolidation ãçºåãããªã) ãš NDB ( nodes: "10%" ãåæäžæããŒãæ°ã®å¶é) ãè£å®çãšããŠçµã¿åãããŸããAlways 㯠Drift / Expiration ãæå¹ã«æ©èœãããäžæ¹ãConsolidation ãŸã§ WhenEmptyOrUnderutilized ã«ãããšçšŒåäž Pod ã®ããããŒããéçŽå¯Ÿè±¡ãšãªãã²ãŒã ãµãŒããŒãé »ç¹ã« Evict ããåŸããããConsolidation 㯠WhenEmpty ã«çµãã空ããŒãã®ååã®ã¿ã«çããŸããAgones ã® Fleet èšå® spec.scheduling ã®å€ã¯ããã©ã«ãã§ Packed ãšãªã£ãŠãããã§ããã ãæ¢ã« GameServer ãåäœããŠããããŒãã«æ°èŠ GameServer ãèµ·åããããã«ã¹ã±ãžã¥ãŒã«ããããã WhenEmpty ã®èšå®ã§ãååãªã³ã¹ãæé©å广ãèŠèŸŒããŸãã Point 5 (ã³ã³ãããŒã©ç³») ïŒAgones ã³ã³ãããŒã©ç³»ãš GameServer ã® NodePool ãåãã Point 1 ã§èšå®ãã expireAfter / terminationGracePeriod 㯠NodePool åäœã®èšå®ã§ããAgones ã®ã³ã³ãããŒã©ç³» (controller / extensions / allocator / ping) ãš GameServer ã§ã¯ãå¿
èŠãªç¶äºæéãç°ãªããŸãã ã³ã³ãããŒã©ç³»ïŒ çæéã®åèµ·åã蚱容ããããããé·æå®å®çšŒåãåªå
ãã€ã€ç¶äºã¯æ¯èŒççããŠããã GameServerïŒ ã²ãŒã æå€§æç¶æéã«å¯Ÿå¿ããé·ãç¶äº ( terminationGracePeriod ) ãå¿
èŠã ããã§ãã³ã³ãããŒã©ç³»çšãš GameServer çšã§ NodePool ãåé¢ããŸã ( nodepools.yaml ) ã # ã³ã³ãããŒã©çš NodePool apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: agones-controller spec: template: metadata: labels: agones-role: controller # ã©ãã«ã§åœ¹å²ãåºå¥ spec: # ã³ã³ãããŒã©ã¯é·æå®å®çšŒåãåªå
expireAfter: 336h # 14æ¥ terminationGracePeriod: 72h # 3æ¥ --- # GameServer çš NodePool (Point 1 ã§åæ²) # labels: agones-role: gameserver / terminationGracePeriod: 4h ãã®ããã§ãGameServer åŽã¯ Fleet ã® nodeSelector ã§ GameServer çš NodePool ãæå®ããŸã ( fleet.yaml ) ã template: spec: # Point 5: GameServer çš NodePool ã«ã¹ã±ãžã¥ãŒã« nodeSelector: agones-role: gameserver ã³ã³ãããŒã©ç³»ã¯ãAgones ã® Helm Values ã§ nodeSelector ãæå®ããã³ã³ãããŒã©çš NodePool ã«é
眮ããŸã ( agones-values.yaml ) ã agones: controller: # Point 5: ã³ã³ãããŒã©çš NodePool ã«ã¹ã±ãžã¥ãŒã« nodeSelector: agones-role: controller extensions: nodeSelector: agones-role: controller allocator: nodeSelector: agones-role: controller ping: nodeSelector: agones-role: controller ããã«ãããããããã®ã¯ãŒã¯ããŒãç¹æ§ã«åã£ãç¶äºæéãäž¡ç«ã§ããŸãã Point 6 (ã³ã³ãããŒã©ç³») ïŒagones-allocator ãš agones-ping ã® PDB ãæå¹ã«ãã åè¿°ã®éã EKS Auto Mode ã§ã¯ãã€ãå¿
ãããŒãã®äžæãçºçãããããAgones ã®ã³ã³ãããŒã©ç³»ãµãŒãã¹ã§ãå¯çšæ§ç¶æã®ããã®èšå®ãéèŠã§ããagones-allocator (ã²ãŒã ãµãŒããŒã®å²ãåœãŠãªã¯ãšã¹ããåãä»ãããµãŒãã¹) ãš agones-ping (ã¬ã€ãã³ã·ãŒèšæž¬çšãµãŒãã¹) ã¯ããã©ã«ãã§ PDB ãç¡å¹ã®ãããããŒããäžæãããŠãããããå
šåæ¢ããªããããPDB ãæå¹åããŸã ( agones-values.yaml ) ã agones: allocator: replicas: 3 # Point 6: allocator ã® PDB ãæå¹å pdb: enabled: true minAvailable: 1 ping: replicas: 2 # Point 6: ping ã® PDB ãæå¹å pdb: enabled: true minAvailable: 1 minAvailable: 1 ã«ãããããŒãäžææã«ãæäœ 1 ã¬ããªã«ã皌åãç¶ããããšãä¿èšŒããã³ã³ãããŒã«ãã¬ãŒã³ã®å¯çšæ§ãç¶æããŸãã ãªããagones-controller, agones-extensions ã¯ããã©ã«ãã§ PDB ãæå¹ãšãªã£ãŠããŸãã ããã«å®å®ããéçšã®ããã®è£è¶³ (ã¬ã¢ã±ãŒã¹ãžã®å¯ŸåŠ) Point 1ã6 ã®èšèšã«ãããéåžžã®ããŒãäžæããã®ã²ãŒã ãµãŒããŒä¿è·ãå®çŸã§ããŸããããã«å®å®ããéçšã®ããã«ã¯ã次ã®ã¬ã¢ã±ãŒã¹ãžã®å¯ŸåŠãæ€èšãã€ã³ãã«ãªããŸããããããåå®è£
ã«äŸåãããããå¿
èŠã«å¿ããŠæ€èšããŠãã ããã GameServerAllocation ã®ã¬ãŒã¹ã³ã³ãã£ã·ã§ã³ Ready ç¶æ
ã® GameServer ã SIGTERM ãåä¿¡ããçŽåŸãã Shutdown å®äºãŸã§ã®éã«ãGameServerAllocation ã«ãã£ãŠ Allocate ãããå¯èœæ§ããŒãã§ã¯ãããŸããããããèµ·ãããšããã¬ã€ã€ãŒãæ¥ç¶ããããã«ã¯ Allocate ããã GameServer ãæ¢ã«åæ¢ããŠãããšã©ãŒãšãªããªã¹ã¯ããããŸãã以äžã¯ãããé²ãæ¹æ³ã®äŸã§ãã GameServer ã«ãããããã©ãã« ( allocationblock=false ç) ãä»äžããGameServerAllocation åŽã§ãã®ã©ãã«ãæã€ Pod ã®ã¿ã察象ãšããèšå®ãè¡ããSIGTERM åä¿¡æã«ã¯å³åº§ã«ã©ãã«ãå€ããã©ãã«åé€åŸãäžå®æéåŸ
æ©ãããã®éã« Allocate ãããå Žåã¯ã²ãŒã ãç¶ç¶ããçµäºåŸã« Shutdown ããã èµ·åäž Pod ãžã® SIGTERM çºè¡æã®ãã³ããªã³ã°äžå¯ ã³ã³ããèµ·åäž (Starting / ContainerCreating) ã« SIGTERM ãçºè¡ããããšãã¢ããªã±ãŒã·ã§ã³åŽã§ãã³ããªã³ã°ã§ãããGameServer ã¯æ£åžžã«èµ·åã Ready ç¶æ
ãšãªãäžæ¹ã Pod 㯠terminationGracePeriodSeconds çµéåŸã«åæ¢ããŠããŸãåäœãšãªããã²ãŒã ãä¿è·ã§ããŸããããããé²ãã«ã¯ä»¥äžã®æ¹æ³ãäŸãšããŠæããããŸãã 察çäŸ 1ïŒ GameServer ã«ãããããã©ãã« (allocationblock=false ç) ãä»äžããGameServerAllocation åŽã§ãã®ã©ãã«ãæã€ Pod ã®ã¿ã察象ãšããèšå®ãè¡ããå€éšã³ã³ãããŒã©ã§ Pod ã® deletionTimestamp ãç£èŠããé nullãã€ãŸãåé€äºå®ãããç¶æ
ãªããåè¿°ã® Allocation çšã©ãã«ãå€ããæ°èŠ Allocation ãé²ãã 察çäŸ 2ïŒ ã€ã¡ãŒãžãµã€ãºã®åæžã»DaemonSet çã«ããäºå pullãšãã£ãæ¹æ³ã§èµ·åæéççž®ãå³ããçºç確çãæå°åããã ãŸãšã æ¬èšäºã§ã¯ãAgones ã®åºç€ã« EKS Auto Mode ãæ¡çšããéãããŒãã®èªåäžæããã²ãŒã ãµãŒããŒãä¿è·ããèšèšã 6 ã€ã®ãã€ã³ãã§ç޹ä»ããŸãããå°çšã²ãŒã ãµãŒããŒã®ã»ã«ããã¹ãã§ãAgones ã®æè»æ§ã掻ããã€ã€ EKS Auto Mode ã§éçšç®¡çãã·ã³ãã«ã«ããã«ã¯ãEKS Auto Mode ã®ããŒãäžæãš Agones ã®ã²ãŒã ãµãŒããŒã©ã€ããµã€ã¯ã«ã®ç«¶åãè§£æ¶ããèšèšãéµã«ãªããŸãã æ¬èšäºã§ç޹ä»ãã Point 1ã6 ã®èšèšãã¿ãŒã³ãšå®è£
äŸ ( nodepools.yaml / fleet.yaml / agones-values.yaml ) ããEKS Auto Mode äžã§ã® Agones å°å
¥ãæ€èšããæ¹ã®åèã«ãªãã°å¹žãã§ãã
ã¯ããã« CIU (CyberAgent group Infrastructure Unit) ã® ...
ããã«ã¡ã¯ãã©ã¯ã¹æè¡åºå ±ã§ãã 2026幎7æ15æ¥ã«éå¬ããäž»å¬ã€ãã³ãããRAKUS AI Conference 2026 Summerãã®5æ¬ã®ã»ãã·ã§ã³ã®ãã¡3æ¬ç®ã«ç»å£ããã®ããAIãšãŒãžã§ã³ãéçºèª²ã®ç«¹ç°èããã§ãã ããŒãã¯ãPoCããæ¬çªãžâ楜楜粟ç®AIãšãŒãžã§ã³ããæ¯ãããLLMOpsãšã€ã³ãã©ã®éžæè¢ãã CTOã圹å¡ã«ããçµç¹æŠç¥ã®è©±ã«ç¶ããçŸå Žã®ãšã³ãžãã¢ãå®éã«äœã倿ããŠããã®ããšãããè§£å床ã®é«ãç¥èŠãèªãããã»ãã·ã§ã³ã§ããã竹ç°ãããèªã£ãã®ã¯ããAIå°çšã®ç¹æ®ãªãã®ãšããããã¯ãæ
£ããŠãããã®ãç¥èŠã®ãããã®ãåªå
ããŠçŽ æ©ãæ§ç¯ããŸããããšããæææ±ºå®ã§ãã ãã®èšäºã¯ãã®ãããªæ¹ã«ããããã§ã AIãšãŒãžã§ã³ããPoCããæ¬çªéçšãžé²ããããšããŠããããããã¯ããããé²ããããšããŠãããšã³ãžãã¢ã®æ¹ AIå°çšã®æ°ããåºç€(AgentCoreãªã©)ãæ¡çšãã¹ããã䜿ãæ
£ããæè¡ã§æ§ç¯ãã¹ããè¿·ã£ãŠããæ¹ ãç®æ¬¡ã AIå°çšã®åºç€ãæ¡çšããåã«ããŸãåãã¹ãã㚠βçããã®åŠã³ãçãããæåçã«ããã顧客å¿åãäœçŸããæ¹å ãèŠããªãAIãããèŠããåãã 顧客ã«äŸ¡å€ãå±ãç¶ããããã®ãå°ã«è¶³ã®ã€ããéžæ AIå°çšã®åºç€ãæ¡çšããåã«ããŸãåãã¹ãã㚠竹ç°ãããæ
åœããã®ã¯ã2025幎12æã«Î²çãã2026幎6æ16æ¥ã«æ£åŒçããªãªãŒã¹ãããçµè²»ç²Ÿç®ã¯ã©ãŠããµãŒãã¹ã楜楜粟ç®ãã«çµã¿èŸŒãŸãããäŒç¥šäœæAIãšãŒãžã§ã³ããã§ããé åæžãéžæããã«ãŒããäºåç³è«ãªã©ã®çŽä»ãããŒã¿ãéžã¹ã°ãããšã¯AIã«ãä»»ããäŒç¥šãã§ãããããšéç¥ãå±ããæåŸã¯äººããã§ãã¯ããŠãã®ãŸãŸç³è«ããã æ¥œæ¥œç²Ÿç®åã®AIãšãŒãžã§ã³ãæ©èœã§ãã ãã®ãšãŒãžã§ã³ããæ¯ããå®è¡åºç€ãšããŠãåœåã¯4ã€ã®éžæè¢ãæãã£ãŠãããšãããŸããLambdaãECSããããŠéçºã®éäžã§ç»å ŽããAgentCoreã®ãããªAIå°çšã®ãããŒãžããµãŒãã¹ããããŠEKSãå°äººæ°äœå¶ãšéçºé床ãæåªå
ãããšããå¶çŽã®ãªãã竹ç°ãããã¡ãéžãã ã®ã¯EKSã§ããããªãã§ãæèŠãå²ããã®ã¯ECSãšEKSã®éã§ãæçµçã«ã¯ãã£ããã¢ããã³ã¹ãã®äœããšãCIãªã©æ¢åè³ç£ããã®ãŸãŸæŽ»çšã§ããç¹ããEKSãéžãã ãšãããŸãã ãAIå°çšã®ç¹æ®ãªãã®ãšããããã¯ãæ
£ããŠãããã®ãç¥èŠã®ãããã®ãåªå
ããŠçŽ æ©ãæ§ç¯ããŸããã çç±ã¯æå¿«ã§ãã瀟å
ã«ã¯ãã§ã«Kuberneteséçšã®è³ç£ãšããŠããŠãèç©ãããŠããŠããã£ããã¢ããã«ãããæéã¯ã»ãšãã©åé¡ã«ãªããªããæ€èšŒç°å¢ããªã³ãã¬ãã¹ã§ã»ãŒåçã«åçŸã§ãããç®æ°ããAIå°çšåºç€ã«æ¹ãããå Žé¢ããã£ãã¯ãã§ãããã䜿ãæ
£ããéå
·ã§ç¢ºå®ã«åãžé²ããããšãéžãã 倿ã¯ãAIãšãŒãžã§ã³ãéçºã®æè¡éžå®ã«æ©ãèªè
ã«ãšã£ãŠãäžã€ã®åè軞ã«ãªãã®ã§ã¯ãªãã§ããããã CD(Continuous Delivery: ç¶ç¶çããªããªãŒ)åºç€ã«ãåãææ³ã貫ãããŠããŸããArgoCDãšGitHub Actionsãšããã瀟å
ã«ç¥èŠãèç©ãããçµã¿åãããæ¡çšãGitHub Actionsã«ããèªååãšArgoCDã®åãããããUIã«ãã£ãŠããæäœéã®æäœãèŠããã°ãKubernetesã«è©³ãããªããšã³ãžãã¢ã§ããªãªãŒã¹äœæ¥ãå¯èœãã«ãªãããªã³ããŒãã£ã³ã°ã³ã¹ãã®äœäžã«ãã€ãªãã£ããšãããŸããäžäžãªãªãŒã¹ã«å€±æããéããArgoCDã®UIäžã§ããã«æ°ã¥ãããããKubernetesæèè
ãžã®ãšã¹ã«ã¬ãŒã·ã§ã³ãã¹ã ãŒãºã§ãã ãµãŒãã¹æ§æã¯ããããŒãžããµãŒãã¹ãšOSSã®ãã€ããªããã§ãããµãŒãã¹åå²ã®åºæºã¯ãã¹ã±ãŒãªã³ã°ãå¿
èŠãããæè¡ã®å€åãéããããã³ã¢æ©èœãã©ãããã®3ç¹ãã¹ãã¬ãŒãžãç£èŠã®ããã«èªããŒã ã§ã®éçšè² è·ãé«ããªããã¡ãªéšåã«ã¯ããããŒãžããµãŒãã¹ãç©æ¥µçã«æ¡çšããŸããã ããããžã§ã¯ãéå§åœåã¯ãããçšåºŠã®æ£è§£ããåãããªãç¶æ
ã§ãããã ãããããåŸããã§ãèŠå ãåãåããŠå€æŽã§ããæ§æã«ããŸããã 竹ç°ããèªèº«ããåœæãã¹ãã ã£ããã¯é£ããããšãã€ã€ããããã¿ãŒãšèšãã倿ãã ã£ããšæ¯ãè¿ããŸããå®ç§ãªæ£è§£ãæåããè¿œãæ±ããã®ã§ã¯ãªãã倿Žå¯èœæ§ãæ®ããæææ±ºå®ãç©ã¿éããå§¿å¢ã¯ãæ£è§£ã®èŠããªãAIãããã¯ãéçºã«ãããå®è·µç¥ã®äžã€ã ãšæããŸããã βçããã®åŠã³ãçãããæåçã«ããã顧客å¿åãäœçŸããæ¹å βçããæ£åŒçãžã®éã®ãã§ã竹ç°ãããäžçªå€§ããªæ¹åãšããŠæããã®ããKEDA(Kubernetes Event Driven Autoscaling)ã®å°å
¥ã§ãã äŒç¥šäœæãšãŒãžã§ã³ãã¯ãLLMåŒã³åºããªã©æéã®ãããåŠçãéåæåããŠããŸããåœåã¯ãã®éåæåŠçãããã¥ãŒã«ã¡ãã»ãŒãžãä¿ç®¡ããŠããã宿çã«ã¡ãã»ãŒãžã®æç¡ã確èªããŠäžå®æ°ãã€åŠçãããããŒãªã³ã°æ¹åŒã§å®è£
ããŠããŸãããåŠçæ°ã¯äžå®æ°ã§åºå®ããŠããã1ãããã§åŠçããããªãã£ãåã¯ã次ã®ãããåŠçã«åãä»çµã¿ã§ãã ãã®éããã¡ãã»ãŒãžåŠçæ°ããšããããééãã®ãã©ã³ã¹ã®èŠæ¥µããæ©ã¿ã©ããã§ãããééãçãããã°ãã£ãã·ãã£ãªãŒããŒã®ãªã¹ã¯ãé«ãŸããé·ãããã°é¡§å®¢ãåŸ
ãããUXæªåã«ç¹ãããŸããããã«ã¿ã€ãã³ã°ã«ãã£ãŠã¯ããããã®çéã«å
¥ã£ããªã¯ãšã¹ããæ¬¡ã®ãããã®æåŸãŸã§åŸ
ããããŠããŸããUXã«ã ã©ã®ããç¶æ
ãçãŸããŠããŸããã ãç¡é§ãªããã£ãã·ãã£ãªãŒããŒããã«äœ¿ãããããªãœãŒã¹ã«äœè£ããããšãã¯ãå³åº§ã«è¿ãç¶æ
ã§ãªã¯ãšã¹ãã«åå¿ãããã ãã®èª²é¡ã«å¯ŸããŠå°å
¥ãããã®ãKEDAã§ããäŒç¥šäœæãšãŒãžã§ã³ãã§ã¯ãKubernetesã®ãžã§ãåäœã§ãã¹ã±ãŒãªã³ã°ã§ããæ§è³ªãæŽ»ããããã¥ãŒã®ä»¶æ°ã«å¿ãããžã§ãèµ·åæ°ã®å¶åŸ¡ãå®çŸããŸããã KEDAå°å
¥ã«ãããã€ã³ã ãã¥ãŒã®ä»¶æ°ã«å¿ããŠãžã§ãèµ·åæ°ãæ¯çã§å¶åŸ¡(äŸ:ãã¥ãŒ4ä»¶âããã2ã€èµ·å) 宿å®è¡ãããªã¢ã¯ãã£ããªã€ãã³ãé§åã¢ãŒããã¯ãã£ãžè»¢æ ãããã§ã¹ãã§éçšã§ãããããæ¢åã®Kuberneteséçšãšã®èŠªåæ§ãç¶æ ããã«ããããã£ãã·ãã£ãªãŒããŒã®å¯èœæ§ã¯äžããã峿åå¿ã«ãã£ãŠé¡§å®¢ãåŸ
ããããªã¹ã¯ãæžããŸãããæŽŸæãªæ©èœè¿œå ã§ã¯ãªããå°éãªå®è£
ã®å·¥å€«ã§ãŠãŒã¶ãŒäœéšã磚ãç¶ãã奜äŸã ãšèšããã§ãããã ãèŠããªãAIãããèŠããåãã 竹ç°ãããæåŸã«åŒ·èª¿ããã®ããObservability(å¯èŠ³æž¬æ§)ãžã®æè³ã§ãããç£èŠã¯åŸåãã«ãããã¡ã§ãããæåããæãã€ããŠããŸããããšããèšèã©ãããäŒç¥šäœæãšãŒãžã§ã³ãã§ã¯ãã¬ãŒã¹ã»ã¡ããªã¯ã¹ã»ãã°ãšãã3çš®é¡ã®ãã¬ã¡ããªãŒããOpenTelemetry CollectorãšFluent-bitçµç±ã§AWSã®ãããŒãžããµãŒãã¹ãžéçŽããåºç€ãæ§ç¯ããŠããŸãã ãªããããŸã§åãå
¥ããã®ãããµãŒãã¹ãåå²ãããŠãã以äžãåŠçã远ãã«ã¯åæ£ãã¬ãŒã¹ãæ¬ ãããŸããããŸããAIãšãŒãžã§ã³ãã¯è€éãã€äžç¢ºå®ã«åäœãããããäœãèµ·ããŠãããã®è©³çްã远ãå¿
èŠããããŸããAIãšãŒãžã§ã³ãã®é決å®çãªæ¯ãèãã¯ãåããšã©ãŒã§ãåå ãç°ãªããããšãçãããããŸããããã°ã®ã¡ãã»ãŒãžã ãã§ã¯å€å¥ãã¥ããäžå
·åã®åå ç©¶æã«ããã¬ãŒã¹ã圹ç«ã£ãŠãããšãããŸãã ã³ã¹ããšã®ãã©ã³ã¹ã工倫ã®ãã©ããã§ãããµã³ããªã³ã°çã¯å¯Ÿè±¡ã«ãã£ãŠäœ¿ãåããŠãããšã®ããšã§ããã 察象 ãµã³ããªã³ã°ç çç± LLMåŒã³åºãé¢é£ã®ãã¬ãŒã¹ 100% å©çšç¶æ³ã»ã¢ãã«å©çšç¶æ³ã®ææ¡ã«å¿
é ãšã©ãŒç³»ãã¬ãŒã¹(ã¹ããŒã¿ã¹ããšã©ãŒ/æªèšå®) 100% é害調æ»ã«å¿
é ãå
šäœéãšããŠã蚱容ç¯å² éåžžãã¬ãŒã¹(çŸåš) 5% ã³ã¹ããæãã€ã€åŸåãææ¡ éåžžãã¬ãŒã¹(ãµãŒãã¹ãäžå®å®ã ã£ãåæ) 70%çšåºŠ å®å®ãããŸã§ã¯æ±ºãæã¡ã§å€ãã«ååŸ æåãã5%ã ã£ãããã§ã¯ãªããå®å®ããã«ã€ããŠæ®µéçã«çµã蟌ãã§ãã£ããšããããã»ã¹ãå°è±¡çã§ãããã¡ããªã¯ã¹åéã¯ãCloudWatch Agentã§ã¯ãªãã³ã³ããã€ã³ãµã€ãã¬ã·ãŒããŒçµç±ã§OpenTelemetry Collectorãå©çšããããšã§ããã£ã«ã¿ãªã³ã°ã«ããã³ã¹ãæé©åãšãç°å¢ã«å¿ããæè»ãªãµã³ããªã³ã°èª¿æŽãäž¡ç«ãããŠããŸãããã°ã«ã€ããŠã¯ãCloudWatchãšS3ãä¿ç®¡å ŽæãšããŠäœµçšããå©äŸ¿æ§ãšã³ã¹ãã®ãã©ã³ã¹ãåã£ãŠããŸãããã®ããã«äžçš®ã®ãã¬ã¡ããªã«ã€ããŠãåæããã³ã¹ããèæ
®ããæ§æã«ããŠããŸãã ããããŠèç©ãããã¬ãŒã¹ããã°ã¯ãç£èŠã®ããã ãã§ãªã次ã®äžæã«ãã€ãªãã£ãŠããŸãã倱æå
容ãåæããæšè«ã»ããŒã«åŒã³åºãã»ããªããŒã·ã§ã³ã®ã©ãã§ã€ãŸãããã®ããç¹å®ããŸãã顧客ããã®åãåãããšé¡äŒŒã®å€±æãã¿ãŒã³ãèŠã€ãã£ãå Žåã¯ã顧客ããŒã¿ãã®ãã®ã¯äœ¿ããã«ãåå ãåçŸãããããŒããŒã¿ã»ãããäœæããŠãªãã©ã€ã³è©äŸ¡ã®æ¹åã«åœ¹ç«ãŠãŠãããšã®ããšã§ããã ããã«ãçŸåšã®ãªãã©ã€ã³è©äŸ¡ã«å ããŠãªã³ã©ã€ã³è©äŸ¡ããã®ãã£ãŒãããã¯ã«ãŒãã®æ§ç¯ããè©äŸ¡ãã©ãããã«ãããã¹ãã¹ã³ãŒãã®æŽçãè©äŸ¡è»žã®äœç³»åã«ãåãçµãã§ããããã§ãã ãããã¯AIãšãŒãžã§ã³ãéçºèª²ã®å°ä»»ã¡ã³ããŒãäžå¿ãšãªã£ãŠé²ããŠãããäŒç¥šäœæãšãŒãžã§ã³ãåäœã®æ¹åã«ãšã©ãŸããããããã¯å
šç€Ÿã§äœ¿ãããã©ãããã©ãŒã ãšããŠã®æŽåãèŠæ®ããŠãããšãããŸãã 顧客ã«äŸ¡å€ãå±ãç¶ããããã®ãå°ã«è¶³ã®ã€ããéžæ 竹ç°ããã®çºè¡šãæ¯ãè¿ããšãããã«ãã£ãã®ã¯ãAIå°çšã®ç®æ°ããæè¡ã远ãããããå§¿å¢ã§ã¯ãªããã䜿ãæ
£ããæè¡ã§ç¢ºå®ã«åã«é²ã¿ãå®è£
ã®å·¥å€«ã§é¡§å®¢äœéšã磚ããç£èŠãžã®æè³ã§ä¿¡é Œæ§ãæ
ä¿ããããšãããå°ã«è¶³ã®ã€ããæææ±ºå®ã®ç©ã¿éãã§ããã ãã®å§¿å¢ã¯ãã©ã¯ã¹éçºæ¬éšãæ²ããã顧客å¿åÃAIãã€ãã£ããªéçºçµç¹ããšããæ¹åæ§ãã®ãã®ã ãšæããŸããAIãã€ãã£ããšã¯ãç®æ°ããæè¡ããšã«ããæ¡çšããããšã§ã¯ãªãã顧客ã«äŸ¡å€ãå±ãç¶ããããã«ãæ¢åã®è³ç£ãšAIãªã©ã®æ°ããæè¡ãé©åã«çµã¿åãããŠããéžæã®é£ç¶ãšèšããŸããäŒç¥šäœæAIãšãŒãžã§ã³ãã®è£åŽã«ããæè¡éžå®ã¯ããã®å®è·µã®äžã€ã®åœ¢ã§ã¯ãªãã§ããããã ãæ¬çªéçšãä»»ããããAIãšãŒãžã§ã³ããã©ãäœãããã«æ©ããšã³ãžãã¢ã®æ¹ã«ãšã£ãŠã竹ç°ããã®å€æè»žãäœãã®ãã³ãã«ãªãã°å¹žãã§ãã åœæ¥ã®çºè¡šè³æã¯SpeakerDeckã§å
¬éããŠããŸãããã²ããããŠã芧ãã ããã çºè¡šè³æ speakerdeck.com ãªãã8æäžæ¬ããã«çºè¡šã®ã¢ãŒã«ã€ãåç»ãã©ã¯ã¹ãšã³ãžãã¢æ
å ±ããŒã¿ã«ãµã€ãã«ãŠå
¬éäºå®ã§ãã ã©ã¯ã¹ãšã³ãžãã¢æ
å ±ããŒã¿ã«ãµã€ã career-recruit.rakus.co.jp ãRAKUS AI Conference 2026 Summerãã®ä»ã¬ããŒãèšäº ã» AIãèŒããããšã¯ãŽãŒã«ã§ã¯ãªããã©ã¯ã¹CTOãšéçºå¯æ¬éšé·ãèªã£ããçµç¹ãšãããã¯ãã®å€é© 㻠顧客ã®å£°ããçãŸãããAIè¿ä¿¡è£å©æ©èœãã®éçºããã»ã¹ ã» ä»æ§é§åéçºãå°å
¥å幎ããæ¬åœã«éããªã£ãŠãã®?ãã«ããŒã¿ã§çãã ã©ã¯ã¹ã§ã¯ãããããã顧客課é¡ã®è§£æ±ºãã«çå£ã«åãåãAIãšãŒãžã§ã³ãéçºã«äžç·ã«åãçµã仲éãåéããŠããŸãããèå³ãæã£ãŠããã ããæ¹ã¯ããã²æ¡çšããŒãžããã§ãã¯ããŠã¿ãŠãã ããã æåŸãŸã§ãèªã¿ããã ãããããšãããããŸããïŒ















