ã¯ããã« ããã«ã¡ã¯ãæ ªåŒäŒç€ŸZOZOã®SREéšãã©ãããã©ãŒã SREããŒã ã«æå±ããŠãã ã¯ã£ã¡ãŒ ãšç³ããŸãã æ¬èšäºã§ã¯ãKubernetesã¯ã©ã¹ã¿ãŒäžã«ã¢ãã¯ãªãœãŒã¹ããµã¯ããšæ§ç¯ãããã¢ãã¯æ§ç¯ããŒã«ãã玹ä»ããŸããZOZOã®äºäŸãããšã«ãã説æãšãªããŸãããKubernetesã¯ã©ã¹ã¿ãŒäžã§ã®è² è·è©Šéšãããã³ããšã³ãéçºãªã©ã®å¹çåã«ãããŠåºãäžè¬çã«æŽ»çšã§ããããŒã«ã®ãããOSSãšããŠå
¬éããŠããŸããGitHubãªããžããªã¯ä»¥äžã§ãã github.com æ¬ããŒã«ã¯ãç§å人ã®OSSãšããŠç®¡çããŠããŸããZOZOã§ã¯ã瀟å¡ãOSS掻åããããããã«ããæ¥åæéäžã«æç€ºããã£ãŠæžãããœãããŠã§ã¢ã§ãèäœæš©è²æž¡ã®èš±è«Ÿã«ãã£ãŠå人ã®ãã®ã«ã§ããããšããOSSããªã·ãŒããããŸãããããããã§ãã techblog.zozo.com ç®æ¬¡ ã¯ããã« ç®æ¬¡ ã¢ãã¯æ§ç¯ããŒã«ãšã¯ éçºã®ãã£ãã ZOZOã§ã¢ãã¯æ§ç¯ããŒã«ã圹ç«ã€å Žé¢ è² è·è©Šéšã§äŸåå
ãã€ã¯ããµãŒãã¹ã«å€ãã®è² è·ããããå Žé¢ stgç°å¢ã®ç«¶ååé¿ APIã¯ã©ã€ã¢ã³ãåŽã®éçº äœ¿ãæ¹ åæïŒããŒã«å®è¡åŽïŒ Step1. OpenAPI仿§æžã®ã³ã㌠Step2. AWSãšKubernetesã®æ¥ç¶èšå® Step3. ãã©ã¡ãŒã¿ãŒèšå® Step4. ã¢ãã¯ãªãœãŒã¹ã®æ§ç¯ Step5. ïŒä»»æïŒã¬ã€ãã³ã·ãŒã®èšå® Step6. ïŒä»»æïŒãã¹ããªã¯ãšã¹ã Step7. è² è·è©Šéš Step8. ã¢ãã¯ãªãœãŒã¹ã®åé€ å®è£
çŽ¹ä» ãªãGoã§å®è£
ããã ãã£ã¬ã¯ããªæ§æ Goã³ãŒãä»¥å€ Goã³ãŒã ãã©ã¡ãŒã¿ãŒåŠçãšåæååŠç main颿° Dockerã€ã¡ãŒãžãšECRã®æ§ç¯ãšåé€ KubernetesãªãœãŒã¹ã®æ§ç¯ãšåé€ IstioãªãœãŒã¹ã®æ§ç¯ãšåé€ ä»åŸã®å±æ 远å éçº äœ¿ã£ãŠããã ãŸãšã We are hiring ã¢ãã¯æ§ç¯ããŒã«ãšã¯ ã¢ãã¯æ§ç¯ããŒã«ã¯ãKubernetesã¯ã©ã¹ã¿ãŒäžã§ãã€ã¯ããµãŒãã¹ã®ã¢ãã¯ãªãœãŒã¹ãæ§ç¯ããè² è·è©Šéšãããã³ããšã³ãéçºãªã©ãå¹çåããããŒã«ã§ããã¢ãã¯ã¯ãAPIãªã©ã®åŠçãæš¡å£ãããã®ã§ããOSSã® Prism ãå©çšããŠãããã¢ãã¯ãµãŒããŒã¯ OpenAPI ã§å®çŸ©ããã仿§æžã«ãããæ£åžžç³»ã®exampleå€ã§ã¬ã¹ãã³ã¹ãè¿ããŸããæ¬ããŒã«ã¯ãåã«Prismã®PodãKubernetesã¯ã©ã¹ã¿ãŒäžã«æ§ç¯ããã ãã§ã¯ãããŸããã Amazon Elastic Container Registry ïŒä»¥éãECRïŒãäžé£ã®KubernetesãªãœãŒã¹ãã³ãã³ãäžçºã§æ§ç¯ããŸãïŒäºåäœæ¥ããïŒã詳现ã¯åŸè¿°ããŸãããŸããã¢ãã¯ã¯1ã€ã®ãã€ã¯ããµãŒãã¹ã ãã§ãªããè€æ°æ§ç¯ã§ããŸãã éçºã®ãã£ãã ãããã€ã¯ããµãŒãã¹ã®è² è·è©Šéšã§å€§ããè² è·ããããéã«ãäŸåå
ãšãªããã€ã¯ããµãŒãã¹ã«ããã«äŸåãããã€ã¯ããµãŒãã¹ãå€éšã·ã¹ãã ãžå€§éã«ãªã¯ãšã¹ããé£ãã§ããŸããåé¡ãšãªã£ãããšããã£ããã§ããèªããŒã ã®ç®¡èœãã€ã¯ããµãŒãã¹ã§ããã°åœ±é¿ç¯å²ã¯äºæž¬ã§ããŸãããä»ããŒã ã®ç®¡èœãã€ã¯ããµãŒãã¹ãšãªããšãããã«ãã®å
ã®åœ±é¿ç¯å²ãææ¡ããã®ã¯å°é£ã§ãããŸããè² è·è©Šéšã®ãã³ã«åSREããŒã ãžåœ±é¿ç¯å²ã確èªãããã®ãçŸå®çã§ã¯ãããŸããã ããã§ãè² è·è©Šéšæã«ã¢ãã¯ã欲ãããšããèãã«ãªããŸãããããããKubernetesã¯ã©ã¹ã¿ãŒäžã«ãã€ã¯ããµãŒãã¹ã®ã¢ãã¯ãçšæããã®ã¯ãé¢åãªäœæ¥ã§ããå ããŠãè² è·è©Šéšã¯éçºå®äºåŸã®ãªãªãŒã¹æ¥ãè¿«ã£ã段éã§å®æœãããã±ãŒã¹ãã»ãšãã©ã§ãè©Šéšæéã¯éãããŠããŸãããããã£ãŠããªãã¹ãå·¥æ°ããããã«ã誰ã§ãåäžçãªæ¹æ³ã§æè»œã«ãã€ã¯ããµãŒãã¹ã®ã¢ãã¯ãçšæããä»çµã¿ã瀟å
ã«å¿
èŠã ãšæããŸãããããã§ãã¢ãã¯æ§ç¯ããŒã«ãéçºããããšã«ããŸããã ZOZOã§ã¢ãã¯æ§ç¯ããŒã«ã圹ç«ã€å Žé¢ è² è·è©Šéšã§äŸåå
ãã€ã¯ããµãŒãã¹ã«å€ãã®è² è·ããããå Žé¢ ãã£ãããšãªã£ãå Žé¢ã«ãªããŸãããããå°ã詳现ã«èª¬æããŸãã åæãšããŠãZOZOã§ã¯è² è·è©Šéšãæ€èšŒç°å¢ïŒä»¥éãstgç°å¢ïŒã§è¡ããŸããstgç°å¢ã§ã¯ããã¹ãŠã®ãã€ã¯ããµãŒãã¹ãæ¬çªç°å¢ïŒä»¥éãprdç°å¢ïŒãšåãã¹ããã¯ã®Podã§åäœããŠããŸããããããã€ã³ãã©ã³ã¹ãã®èгç¹ãããPodæ°ã¯prdç°å¢ããã倧å¹
ã«å°ãªãã§ãããŸããè² è·è©Šéšã®å¯Ÿè±¡ãšãªããã€ã¯ããµãŒãã¹ã®Podæ°ã¯1ã§åºå®ããŠããããªãŒãã¹ã±ãŒãªã³ã°ãããªãããã«ããŠããŸãã è² è·è©Šéšã§ã¯ã詊éšå¯Ÿè±¡ã®ãã€ã¯ããµãŒãã¹ãçæéã«å€§éã®ãªã¯ãšã¹ããå®è¡ããŸãã詊éšå¯Ÿè±¡ã®ãã€ã¯ããµãŒãã¹ãå¥ã®ãã€ã¯ããµãŒãã¹ã«äŸåããŠããå Žåãå®è¡ããAPIã«ãã£ãŠã¯ãã¡ãã«ããªã¯ãšã¹ããæµããŸããæµãããªã¯ãšã¹ãéãå€ããªãå Žåã¯ãstgç°å¢ã§èµ·åããŠãããäŸåå
ã®ãã€ã¯ããµãŒãã¹ã®Podã«ãã®ãŸãŸãªã¯ãšã¹ããæµããŠãæ¯éã¯ãªãã§ããäžæ¹ãæµãããªã¯ãšã¹ãéãå€ãå Žåã¯ããªã¯ãšã¹ããæããããªããªããè² è·è©Šéšã«æ¯éãã§ãŸããäŸåãã€ã¯ããµãŒãã¹ã¯ãªãŒãã¹ã±ãŒãªã³ã°ããŸãããã¹ã±ãŒãªã³ã°ãå®äºãããŸã§ã«å€å°ã®æéã¯ããããããæ£ç¢ºã«æ§èœãèšæž¬ã§ããŸããããããã£ãŠãäŸåãããã€ã¯ããµãŒãã¹ãäºåã«ã¹ã±ãŒã«ã¢ããããå¿
èŠããããŸãã ããããªããããã€ã¯ããµãŒãã¹ã®æ°ãå€ããšäºåã¹ã±ãŒãªã³ã°ã®å¯Ÿè±¡ãå€ããŠäœæ¥ã倧å€ã§ãããèªããŒã ã®ç®¡èœå€ãã€ã¯ããµãŒãã¹ã«é¢ããŠã¯ããã®æ
åœSREããŒã ãšã®èª¿æŽã³ã¹ããçºçããŸãããŸããã1PodãããCPU䜿çšç50ïŒ
ã§æããéçã®ã¹ã«ãŒããããèšæž¬ããããšãã詊éšé
ç®ãããããã詊éšåã«è² è·ã®çšåºŠãèŠç©ãã£ãŠäŒããã®ãé£ããã§ããå°ããã€å¢åŒ·äŸé Œãšãªããšèª¿æŽã倧å€ã§ããã倧éæã«å¢åŒ·ãããããšã ããªã€ã³ãã©ã³ã¹ããçºçããŠããŸããŸããããã«ãZOZOã§ãªãå€éšã®ã·ã¹ãã ã«äŸåããŠããå Žåã¯ããã調æŽãå°é£ã«ãªããŸãããã®ãããªã±ãŒã¹ã§ã¢ãã¯ã¯éåžžã«åœ¹ç«ã¡ãŸãã stgç°å¢ã®ç«¶ååé¿ ZOZOã§ã¯1ã€ã®stgç°å¢ã§è€æ°ã®SREããŒã ãè€æ°ã®ãããžã§ã¯ãã«é¢ããè² è·è©Šéšãé害詊éšã宿œããŠããŸãããããã£ãŠããã°ãã°stgç°å¢ã§ã¯è€æ°ã®è©Šéšã¿ã€ãã³ã°ãéè€ããŠããŸããŸãããããåæã«è©Šéšã宿œããŠããŸããšããäºãã®è©Šéšã圱é¿ããŠããŸããæ£ãã詊éšãã§ããŸãããçŸç¶ã¯æ
åœè
éã®èª¿æŽã§åé¿ããŠããŸãããè² è·è©Šéšã§ã¯ã¢ãã¯æ§ç¯ããŒã«ã§ã¢ãã¯ãçšæããããã«ãªãã°ã調æŽã³ã¹ããåæžã§ããŸãã APIã¯ã©ã€ã¢ã³ãåŽã®éçº ãã€ã¯ããµãŒãã¹ã®OpenAPI仿§èšèšãå®äºããã°ãKubernetesã¯ã©ã¹ã¿ãŒäžã«ã¢ãã¯ãæ§ç¯ã§ããŸãããããã£ãŠãããã³ããšã³ããªã©ã®APIã¯ã©ã€ã¢ã³ãåŽã¯ãã®ãã€ã¯ããµãŒãã¹ã®éçºå®äºãåŸ
ã€å¿
èŠãªããã¯ã©ãŠãç°å¢äžã§èªåãã¡ã®éçºãåäœç¢ºèªãé²ããããšãã§ããŸãã äœ¿ãæ¹ åæïŒããŒã«å®è¡åŽïŒ AWSãšKubernetesã®èªèšŒæ
å ±ãèšå®æžã¿ 以äžãã€ã³ã¹ããŒã«æžã¿ Go v1.22.5ã§ã®åäœã¯ç¢ºèªæžã¿ aws-cli å®å
šã« aws-sdk-go ãžç§»è¡ã§ããŠããªããã kubectl VirtualServiceããã¹ããªã¯ãšã¹ãã§å©çš å¿
é ã§ãªã Docker ã€ã¡ãŒãžãã«ããªã©ããããã Step1. OpenAPI仿§æžã®ã³ã㌠openapi.yaml ã«OpenAPIã®ä»æ§ãã³ããŒ&ããŒã¹ãããŸãã ZOZOã§ã¯ãã»ãŒãã¹ãŠã®ãã€ã¯ããµãŒãã¹ã®APIã¯OpenAPIã§å®çŸ©ãããŠããŸãã瀟å
ã§ã¯ GiHub Pages äžã§å
¬éãããŠããŸãã Step2. AWSãšKubernetesã®æ¥ç¶èšå® ãªãœãŒã¹ãæ§ç¯ããAWSãšKubernetesã®æ¥ç¶èšå®ãããŸããããšãã°ã awsp ãš kubie ã䜿ã£ãŠããå Žåã¯ä»¥äžã§ãã awsp < your_profile > kubie ctx < your_context > Step3. ãã©ã¡ãŒã¿ãŒèšå® params.go ã§ãã©ã¡ãŒã¿ãŒãèšå®ããŸããçŸç¶ã¯ãããŒãã³ãŒãã£ã³ã°ã§èšå®ããå¿
èŠããããŸããããŒã«å©çšè
ã¯æäœéã以äžã®ãã©ã¡ãŒã¿ãŒèšå®ãå¿
èŠã§ãã microserviceName ã¢ãã¯ã«ãããã€ã¯ããµãŒãã¹ã®åå e.g. zozo-member-api microserviceNamespace ã¢ãã¯ã«ãããã€ã¯ããµãŒãã¹ã®ããŒã ã¹ããŒã¹ e.g. zozo-member Step4. ã¢ãã¯ãªãœãŒã¹ã®æ§ç¯ 以äžã®ã³ãã³ããå®è¡ããŸãã make run-create ã³ãã³ãäžçºã§ã以äžã®ãªãœãŒã¹ãAWSãšKubernetesã¯ã©ã¹ã¿ãŒäžã«æ§ç¯ãããŸãã AWS ECR Kubernetes Namespace Deployment Service VirtualService Step5. ïŒä»»æïŒã¬ã€ãã³ã·ãŒã®èšå® ãã®æé ã¯ä»»æã§ãã çŸç¶ã®ãŸãŸã§ããšãã¢ãã¯ã¯å³åº§ã«APIã¬ã¹ãã³ã¹ãè¿ããŸããããã§ã¯è² è·è©Šéšã®ã¢ãã¯ãšããŠã¯äžååãªå ŽåããããŸãããªããªãã°ãå®éã®APIã§ã¯DBã¢ã¯ã»ã¹ãªã©ã®ããŸããŸãªåŠçãããåŸã«ã¬ã¹ãã³ã¹ãè¿ããããã¬ã€ãã³ã·ãŒãå°ãªãããçºçããããã§ããè² è·è©Šéšçšã®ã¢ãã¯ãšããŠã¯ããã®ã¬ã€ãã³ã·ãŒãèæ
®ããŠAPIã¬ã¹ãã³ã¹ãè¿ãããã«ããæ¹ãããã§ããããã§ãIstioã® Fault Injection æ©èœãå©çšããŠãåºå®æéã®é
å»¶ãçºçãããããã«ããŸããå
·äœçã«ã¯ãããŒã«ã§æ§ç¯ãããVirtualServiceãªãœãŒã¹ãeditããŠã spec.http.fault.delay.fixedDelay ã«ã¬ã€ãã³ã·ãŒã®å€ãèšå®ããŸãã kubectl edit VirtualService -n example-namespace example-vs apiVersion : networking.istio.io/v1alpha3 kind : VirtualService metadata : name : example-vs spec : hosts : - example-service.example-namespace.svc.cluster.local http : - name : example1 match : - uri : prefix : /example1/ method : exact : GET fault : delay : percentage : value : 100.0 fixedDelay : 0.1s # here route : - destination : host : example-service.example-namespace.svc.cluster.local - name : default route : - destination : host : example-service.example-namespace.svc.cluster.local åœç¶ãªããã¬ã€ãã³ã·ãŒã¯APIããšã«ç°ãªããããè² è·è©Šéšã®ã·ããªãªäžã§å®è¡ãããAPIããšã«äžèšã®èšå®ãããŸãããã§ã«prdç°å¢ã§ãªãªãŒã¹æžã¿ã®APIã§ããã°ãprdç°å¢ã®ããŒã¿ãåæããŠå®éã®ã¬ã€ãã³ã·ãŒãèšå®ããŸããZOZOã§ã¯ãDatadogããAPIããšã®p95ã¬ã€ãã³ã·ãŒã®å€ãç°¡åã«ç¢ºèªã§ããããã«ãªã£ãŠããããããããå©çšããŸãã æ£çŽãªãšããããã®èšå®äœæ¥ã¯è² æ
ã倧ããã®ã§ãäžéšèªååããäºå®ã§ãããªãã詊éšçã«åé¡ãªããã°ã¬ã€ãã³ã·ãŒãèšå®ããªããŠãæ§ããŸããããå
šAPIäžåŸã§defaultã»ã¯ã·ã§ã³ã«ã¬ã€ãã³ã·ãŒãèšå®ããŠãæ§ããŸããã Step6. ïŒä»»æïŒãã¹ããªã¯ãšã¹ã ãã®æé ã¯ä»»æã§ãã ã¢ãã¯ãªãœãŒã¹ãæ§ç¯ã§ãããããã¹ããªã¯ãšã¹ããããŸããããšãã°ãKubernetesã¯ã©ã¹ã¿ãŒå
ã«é©åœãªPodãèµ·åããŠãã¢ãã¯ã«å¯ŸããŠcurlã§APIãªã¯ãšã¹ããå®è¡ããŸããOpenAPIã®ä»æ§ã«å³ããã¬ã¹ãã³ã¹ãè¿ãããšã確èªã§ããŸãã kubectl run tmp- $( date " +%Y%m%d-%H%M%S " ) -hacchi --image yauritux/busybox-curl:latest -n api-gateway --annotations =" sidecar.istio.io/inject=true " --rm -it -- sh -c " curl -v -m 10 http://zozo-member-api-prism-mock.zozo-member-prism-mock.svc.cluster.local/internal/members/1 " ... * Request completely sent off < HTTP/ 1 . 1 200 OK < access-control-allow-origin: * < access-control-allow-headers: * < access-control-allow-credentials: true < access-control-expose-headers: * < content-type: application/json < content-length: 467 < date: Mon, 22 Jul 2024 10:09:48 GMT < x-envoy-upstream-service-time: 6 < server: envoy < * Connection #0 to host zozo-member-api-prism-mock.zozo-member-prism-mock.svc.cluster.local left intact { " id " :1, " email " : " taro.tanaka@zozo.com " , " zozo_id " : " tanaka " , " has_password " :false, " last_name " : " ç°äž " , " first_name " : " 倪é " , " last_name_kana " : " ã¿ãã« " , " first_name_kana " : " ã¿ã㊠" , " gender_id " :1, " birthday " : " 2004-12-15 " , " zipcode " : " 1020094 " , " prefecture_id " :1, " address " : " å代ç°åºçŽå°Ÿäºçº1-3 " , " address_building " : " æ±äº¬ã¬ãŒãã³ãã©ã¹çŽå°Ÿäºçº çŽå°Ÿäºã¿ã¯ãŒ " , " phone " : " 0120-55-0697 " , " zozo_employee_id " : " 1 " , " registered_at " : " 2004-12-15T12:00:00+00:00 " } pod " tmp-20240722-190922-hacchi " deleted Step7. è² è·è©Šé𠿥ç¶å
æ
å ±ãã¢ãã¯ã®Serviceã«åãæ¿ããŠãè² è·è©Šéšã宿œããŸãã Step8. ã¢ãã¯ãªãœãŒã¹ã®åé€ è² è·è©Šéšãå®äºããããã¢ãã¯ãªãœãŒã¹ããã¹ãŠåé€ããŸãã make run-delete 以äžã§ãäœ¿ãæ¹ã®èª¬æã¯çµäºã§ãã å®è£
çŽ¹ä» ãªãGoã§å®è£
ããã æ¬ããŒã«ã¯Goã§å®è£
ããŸãããGoãéžæããçç±ã¯ä»¥äžã®éãã§ãã 瀟å
æšå¥šããã°ã©ãã³ã°èšèªã®1ã€ã®ããã éçºè
ã§ããç§ããã£ãšã䜿ãæ
£ããèšèªã§ããããã AWSãKubernetesã®ã©ã€ãã©ãªãè±å¯ã§ããã瀟å
ã§ã䜿çšå®çžŸããã£ãããã ãªãã瀟å
ã®ã€ã³ãã©é¢é£ã®ããŒã«ã¯ã»ãšãã©ãã·ã§ã«ã¹ã¯ãªããã§å®è£
ãããŠãããããã·ã§ã«ã¹ã¯ãªããã§ã®å®è£
ãéžæè¢ã«ãããŸãããããããäžèšã®çç±ã«å ããŠãYAMLãã¡ã€ã«ãèªã¿åãæ©èœãéçºããã«ããã£ãŠã䞻芳ã§ã¯Goã®æ¹ãæžããããããšæããŸããããŸããã·ã§ã«ã¹ã¯ãªããã ãšå©çšè
ã« yq ãã€ã³ã¹ããŒã«ããŠãããå¿
èŠããããããªããããããé¿ããŸããã ãã£ã¬ã¯ããªæ§æ ãŸããå
šäœåãšããŠãã£ã¬ã¯ããªæ§æã瀺ããŸãã ãšãã«ãç¹å¥ãªç¹ã¯ãããŸããããããŸã§è€éãªããŒã«ã§ã¯ãªããããä»ã®ãšããã¯mainããã±ãŒãžã®ã¿ã«ããŠããŸããGoãã¡ã€ã«ã¯ã main.go ã ecr.go ã k8s.go ã istio.go ã params.go ã action_type.go ã lib.go ã®7ã€ã§ãããã®ä»ã«ã¯ã Makefile ã openapi.yaml ã Dockerfile.prism ããããŸãã Goã³ãŒãä»¥å€ èª¬æã®ãããããããGoã³ãŒã以å€ã®èª¬æããå§ããŸãã ç°¡åã«ããŒã«ã®å®è¡ãã§ããããã« Makefile ãçšæããŠããŸããåºæ¬çã«ã¯ã make run-create ïŒã¢ãã¯ãªãœãŒã¹ã®æ§ç¯ïŒããã³ make run-delete ïŒã¢ãã¯ãªãœãŒã¹ã®åé€ïŒãå®è¡ããŸããå®è¡æã«äœæãããã¯ã³ãã€ããªã¯æ®ããªãããã«ããŠããŸãããŸããéçºçšã« make deps ïŒäŸåã¢ãžã¥ãŒã«ã®ããŠã³ããŒãïŒãçšæããŠããŸãã BINARY_NAME =prism-mock GO =go build: $(GO) build -o $(BINARY_NAME) . run-create: build ./ $(BINARY_NAME) -action create $(MAKE) clean run-delete: build ./ $(BINARY_NAME) -action delete $(MAKE) clean clean: $(GO) clean rm -f $(BINARY_NAME) deps: $(GO) mod download ã¢ãã¯å¯Ÿè±¡ãã€ã¯ããµãŒãã¹ã®API仿§ã openapi.yaml ã«ã³ããŒ&ããŒã¹ãããŸããPrismã¯ãã®ãã¡ã€ã«ãå
ã«ã¢ãã¯ãµãŒããŒãèµ·åããŸãã Prismã® Dockerfile.prism ã¯ä»¥äžã®éãã§ããã€ã¡ãŒãžã¯ stoplight/prism:5.8.2 ãæå®ããŠããŸãã openapi.yaml ãCOPYããŸããmockã³ãã³ãã§Prismãèµ·åããŸãã FROM stoplight/prism:5.8.2 COPY ./openapi.yaml /app/openapi.yaml CMD ["mock", "-h", "0.0.0.0", "-p", "80", "/app/openapi.yaml"] Goã³ãŒã ãã©ã¡ãŒã¿ãŒåŠçãšåæååŠç ãã©ã¡ãŒã¿ãŒã®ç®¡çã¯ã params.go ã§åŠçããŠããŸããèšå®å¯èœãªãã©ã¡ãŒã¿ãŒã¯ä»¥äžã§ãã microserviceName=zozo-aggregation-api ã®å Žåãæ§ç¯ããããªãœãŒã¹å㯠zozo-aggregation-api-prism-mock ãšãªããŸãã Parameter Name Description default required microserviceName ã¢ãã¯ã«ãããã€ã¯ããµãŒãã¹å "" Yes microserviceNamespace ã¢ãã¯ã«ãããã€ã¯ããµãŒãã¹ã®ããŒã ã¹ããŒã¹å "" Yes prismMockSuffix ãªãœãŒã¹åã®ãµãã£ãã¯ã¹ "-prism-mock" Yes prismPort Prismã³ã³ãããŒã®ããŒãçªå· "80" Yes prismCPU Prismã³ã³ãããŒã®CPUãªã¯ãšã¹ã "1" Yes prismMemory Prismã³ã³ãããŒã®ã¡ã¢ãªãªã¯ãšã¹ã "1Gi" Yes istioProxyCPU istioãµã€ãã«ãŒã³ã³ãããŒã®CPUãªã¯ãšã¹ã "500m" Yes istioProxyMemory istioãµã€ãã«ãŒã³ã³ãããŒã®ã¡ã¢ãªãªã¯ãšã¹ã "512Mi" Yes timeout ããŒã«ã®å®è¡ã¿ã€ã ã¢ãŠãæé 10 * time.Minute Yes ecrTagEnv ECRã®CostEnvã¿ã°ã®å€ "stg" No main.go ã®init颿°ã§ä»¥äžã®åæååŠçãè¡ããŸãã openapi.yaml ã®ããŒã¿ååŸãšç©ºãã§ã㯠ãã©ã¡ãŒã¿ãŒã®ããªããŒã·ã§ã³ãã§ã㯠ã³ãã³ãã©ã€ã³åŒæ°ïŒã¢ã¯ã·ã§ã³ãã©ã¡ãŒã¿ãŒïŒã®ååŸ AWSãšKubernetesã®èšå® ãªãœãŒã¹åã®äœæ main.goã®ã³ãŒããèŠã var ( action actionType awsConfig aws.Config awsAccountID string kubeConfig *restclient.Config resourceName string namespaceName string ) func init() { //empty check for openapi.yaml data, err := os.ReadFile( "openapi.yaml" ) if err != nil { panic (err) } if len (data) == 0 { panic ( "openapi.yaml is empty" ) } // validation parameters err = validateParams() if err != nil { panic (err) } // action parameter var actionStr string flag.StringVar(&actionStr, "action" , "create" , "create or delete(default: create)" ) flag.Parse() parsedAction, err := validateActionType(actionStr) if err != nil { panic (err) } action = parsedAction // AWS config awsConfig, err = config.LoadDefaultConfig(context.Background()) if err != nil { panic (fmt.Errorf( "failed load AWS config: %v" , err)) } // get AWS account ID stsClient := sts.NewFromConfig(awsConfig) result, err := stsClient.GetCallerIdentity(context.Background(), &sts.GetCallerIdentityInput{}) if err != nil { panic (fmt.Errorf( "failed to get caller identity: %v" , err)) } awsAccountID = *result.Account // kube config kubeconfigPath := clientcmd.NewDefaultPathOptions().GetDefaultFilename() kubeConfig, err = clientcmd.BuildConfigFromFlags( "" , kubeconfigPath) if err != nil { panic (fmt.Errorf( "failed to build kubeconfig: %v" , err)) } // resource name resourceName = microserviceName + prismMockSuffix namespaceName = microserviceNamespace + prismMockSuffix } main颿° main.go ã®main颿°ã§ã¯ããªãœãŒã¹æ§ç¯ãšåé€ã«é¢ããããŸããŸãªåŠçãåŒã³åºããŠããŸãããªãœãŒã¹æ§ç¯ã®å Žåã¯ãbuildAndPushECR颿°ãšcreateK8sResources颿°ãšcreateIstioResources颿°ãé ã«å®è¡ããŸãããªãœãŒã¹åé€ã®å Žåã¯ãdeleteIstioResources颿°ãšdeleteK8sResources颿°ãšdeleteECR颿°ãé ã«å®è¡ããŸããããããããšã©ãŒãçºçããå Žåã¯panicããŸããcontextããã±ãŒãžã§ã¿ã€ã ã¢ãŠãèšå®ããŠããŸãã main.goã®ã³ãŒããèŠã func main() { ctx, cancel := context.WithTimeout(context.Background(), timeout) defer cancel() if action == create { err := buildAndPushECR(ctx) if err != nil { panic (err) } err = createK8sResources(ctx) if err != nil { panic (err) } err = createIstioResources(ctx) if err != nil { panic (err) } log.Println( "[INFO] All resources for prism mock are created successfully" ) } else if action == delete { err := deleteIstioResources(ctx) if err != nil { panic (err) } err = deleteK8sResources(ctx) if err != nil { panic (err) } err = deleteECR(ctx) if err != nil { panic (err) } log.Println( "[INFO] All resources for prism mock are deleted successfully" ) } } main颿°ã®åŠçã®æµããå³ã«ãããšä»¥äžã®éãã§ãã Dockerã€ã¡ãŒãžãšECRã®æ§ç¯ãšåé€ ecr.go ã§ã¯Dockerã€ã¡ãŒãžã®ãã«ããECRã®æ§ç¯ãšåé€ãããŸããbuildAndPushECR颿°ã¯ãDockerã€ã¡ãŒãžããã«ãããŠãECRãæ§ç¯ããŠããã·ã¥ããŸããdeleteECR颿°ã§ã¯ãECRãåé€ããŸãããªããçŸç¶ã¯ECRãã°ã€ã³ã®ç®æã§ aws-cli ãå®è¡ããŠããŸã£ãŠããã®ã§ã aws-sdk-go ã䜿ã£ãŠæ¹ä¿®ããäºå®ã§ãããŸããECRã ãã§ãªãä»ã®ã³ã³ããã¬ãžã¹ããªã«ã察å¿ããäºå®ã§ãã ãã§ã«ååã®ãªãœãŒã¹ãååšããç¶æ
ã§æ§ç¯ãªã¯ãšã¹ãããããšWARNãã°ãåºåããŸãããšã©ãŒã«ã¯ãªããŸãããåæ§ã«ãæå®ãªãœãŒã¹ãååšããªãç¶æ
ã§åé€ãªã¯ãšã¹ãããããšWARNãã°ãåºåããŸãããšã©ãŒã«ã¯ãªããŸãããããã¯ã ecr.go ã k8s.go ã istio.go ã®ãã¹ãŠã§åã仿§ã§ãã ecr.goã®ã³ãŒããèŠã func buildAndPushECR(ctx context.Context) error { // build Docker image imageTag := microserviceName + ":latest" cmd := exec.Command( "docker" , "build" , "-f" , "Dockerfile.prism" , "-t" , imageTag, "." ) if err := cmd.Run(); err != nil { return fmt.Errorf( "failed to build docker image: %v" , err) } log.Println( "[INFO] Docker image is built successfully" ) // create ECR repository ecrClient := ecr.NewFromConfig(awsConfig) repositoryName := resourceName input := &ecr.CreateRepositoryInput{ RepositoryName: aws.String(repositoryName), Tags: []types.Tag{ { Key: aws.String( "CostEnv" ), Value: aws.String(ecrTagEnv), }, { Key: aws.String( "CostService" ), Value: aws.String(microserviceName), }, }, } _, err := ecrClient.CreateRepository(ctx, input) if err != nil { var ecrExistsException *types.RepositoryAlreadyExistsException if !errors.As(err, &ecrExistsException) { return fmt.Errorf( "failed to create ECR repository: %v" , err) } log.Println( "[WARN] The ECR already exists" ) } else { log.Println( "[INFO] ECR is created successfully" ) } // tag Docker image for ECR ecrImageTag := fmt.Sprintf( "%s.dkr.ecr.%s.amazonaws.com/%s:latest" , awsAccountID, awsConfig.Region, repositoryName) cmdTag := exec.Command( "docker" , "tag" , imageTag, ecrImageTag) if err := cmdTag.Run(); err != nil { return fmt.Errorf( "failed to tag image: %v" , err) } log.Println( "[INFO] Docker image tagged successfully" ) // login to ECR loginCommand := fmt.Sprintf( "aws ecr get-login-password --region %s | docker login --username AWS --password-stdin %s.dkr.ecr.%s.amazonaws.com" , awsConfig.Region, awsAccountID, awsConfig.Region) cmdLogin := exec.Command( "bash" , "-c" , loginCommand) if err := cmdLogin.Run(); err != nil { return fmt.Errorf( "failed to log in ECR: %v" , err) } log.Println( "[INFO] Logged in ECR successfully" ) // push image to ECR cmdPush := exec.Command( "docker" , "push" , ecrImageTag) if err := cmdPush.Run(); err != nil { return fmt.Errorf( "failed to push image to ECR: %v" , err) } log.Println( "[INFO] Docker image is pushed to ECR successfully" ) return nil } func deleteECR(ctx context.Context) error { // Delete ECR ecrClient := ecr.NewFromConfig(awsConfig) repositoryName := resourceName input := &ecr.DeleteRepositoryInput{ RepositoryName: aws.String(repositoryName), Force: true , // Force delete to remove all images } _, err := ecrClient.DeleteRepository(ctx, input) if err != nil { var ecrNotFoundException *types.RepositoryNotFoundException if !errors.As(err, &ecrNotFoundException) { return fmt.Errorf( "failed to delete ECR: %v" , err) } log.Println( "[WARN] The ECR is not found" ) } else { log.Println( "[INFO] ECR is deleted successfully" ) } return nil } KubernetesãªãœãŒã¹ã®æ§ç¯ãšåé€ k8s.go ã§ã¯KubernetesãªãœãŒã¹ã®æ§ç¯ãšåé€ãããŸãã kubernetes/client-go ã䜿çšããŠããŸããcreateK8sResources颿°ã§ã¯ã察象Kubernetesã¯ã©ã¹ã¿ãŒã®IstioããŒãžã§ã³ãååŸããNamespaceãDeploymentãServiceãæ§ç¯ããŸããdeleteK8sResources颿°ã§ã¯ãServiceãDeploymentãNamespaceãåé€ããŸãã IstioããŒãžã§ã³ã®ååŸã¯ã istio-system ããŒã ã¹ããŒã¹äžã§åäœãã app: istiod ã©ãã«ã®ä»ããPodïŒistiodïŒã® istio.io/rev ã©ãã«å€ããååŸããŸãããã®ã©ãã«å€ã¯ 1-21-4 ãªã©ã®ãã€ãã³ç¹ãã®ãã®ã«ãªããŸãããã®æãããKubernetesã¯ã©ã¹ã¿ãŒäžã§Istioã®ããŒãžã§ã³ã¢ããã°ã¬ãŒãäœæ¥äžã§ããã°ã2ã€ã®ããŒãžã§ã³ã®PodãååšããŸããããã§ããããã®ãã¡ææ°ããŒãžã§ã³ãè¿ãgetLatestVersion颿°ãå®è£
ããŸãããgetLatestVersion颿°ã¯ãx-y-z圢åŒã®ããŒãžã§ã³ãªã¹ããåãåããparseVersion颿°ã§x-y-z圢åŒã®ããŒãžã§ã³ãæ°å€ã®ã¹ã©ã€ã¹ã«å€æããŸãããããŠãcompareVersions颿°ã§ã¡ãžã£ãŒããŒãžã§ã³ããé ã«å€§å°æ¯èŒããæçµçã«æ°ããæ¹ã®ããŒãžã§ã³ãè¿ããŸãããã®ããã«ãæ¬ããŒã«ã¯Istioã®ããŒãžã§ã³ã¢ããã°ã¬ãŒãäœæ¥äžãåé¡ãªãåäœãããã工倫ããŠããŸãããªããååŸããIstioããŒãžã§ã³ã¯ãNamespaceã® istio.io/rev ã©ãã«ã«äœ¿çšããŸããããŒãžã§ã³æ
å ±ãããŸãååŸã§ããªãã£ãå Žåã¯ãšã©ãŒã«ããã空ã§åŠçãç¶è¡ããŸãã Deploymentã§ã¯ãPodã«ã¯Istioã®ãµã€ãã«ãŒã泚å
¥ããŠããŸããã¡ã€ã³ã³ã³ãããŒã®ã€ã¡ãŒãžã¯æ§ç¯ããECRã®ãã®ãæå®ããŠããŸãã k8s.goã®ã³ãŒããèŠã func createK8sResources(ctx context.Context) error { // create clientset using kubeconfig clientset, err := kubernetes.NewForConfig(kubeConfig) // ... // get the latest istio version from istiod pod considering during upgrade podList, err := clientset.CoreV1().Pods( "istio-system" ).List(ctx, metav1.ListOptions{ LabelSelector: "app=istiod" , }) //... hyphenedVersions := [] string {} for _, item := range podList.Items { hyphenedVersions = append (hyphenedVersions, item.ObjectMeta.Labels[ "istio.io/rev" ]) } latestVersion, err := getLatestVersion(hyphenedVersions) // ... // Namespace namespace := &corev1.Namespace{ ObjectMeta: metav1.ObjectMeta{ Name: namespaceName, Labels: map [ string ] string { "istio.io/rev" : latestVersion, }, }, } _, err = clientset.CoreV1().Namespaces().Create(ctx, namespace, metav1.CreateOptions{}) //... // Deployment deployment := &appsv1.Deployment{ ObjectMeta: metav1.ObjectMeta{ Name: resourceName, }, Spec: appsv1.DeploymentSpec{ Replicas: int32Ptr( 1 ), Selector: &metav1.LabelSelector{ MatchLabels: map [ string ] string { "app" : resourceName, }, }, Template: corev1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Labels: map [ string ] string { "app" : resourceName, }, Annotations: map [ string ] string { "sidecar.istio.io/inject" : "true" , "sidecar.istio.io/proxyCPULimit" : istioProxyCPU, "sidecar.istio.io/proxyMemoryLimit" : istioProxyMemory, "traffic.sidecar.istio.io/includeOutboundIPRanges" : "*" , "proxy.istio.io/config" : `{ "terminationDrainDuration": "30s" }` , }, }, Spec: corev1.PodSpec{ Containers: []corev1.Container{ { Name: resourceName, Image: fmt.Sprintf( "%s.dkr.ecr.%s.amazonaws.com/%s" , awsAccountID, awsConfig.Region, resourceName), Ports: []corev1.ContainerPort{ { ContainerPort: int32 (prismPort), }, }, Resources: corev1.ResourceRequirements{ Limits: corev1.ResourceList{ corev1.ResourceCPU: resource.MustParse(prismCPU), corev1.ResourceMemory: resource.MustParse(prismMemory), }, }, }, }, }, }, }, } _, err = clientset.AppsV1().Deployments(namespaceName).Create(ctx, deployment, metav1.CreateOptions{}) //... // Service service := &corev1.Service{ ObjectMeta: metav1.ObjectMeta{ Name: resourceName, }, Spec: corev1.ServiceSpec{ Selector: map [ string ] string { "app" : resourceName, }, Ports: []corev1.ServicePort{ { Protocol: corev1.ProtocolTCP, Port: 80 , TargetPort: intstr.FromInt( 80 ), }, }, Type: corev1.ServiceTypeClusterIP, }, } _, err = clientset.CoreV1().Services(namespaceName).Create(ctx, service, metav1.CreateOptions{}) //... return nil } func deleteK8sResources(ctx context.Context) error { // create clientset using kubeconfig clientset, err := kubernetes.NewForConfig(kubeConfig) //... // Service err = clientset.CoreV1().Services(namespaceName).Delete(ctx, resourceName, metav1.DeleteOptions{}) //... // Deployment err = clientset.AppsV1().Deployments(namespaceName).Delete(ctx, resourceName, metav1.DeleteOptions{}) //... // Namespace err = clientset.CoreV1().Namespaces().Delete(ctx, namespaceName, metav1.DeleteOptions{}) //... return nil } func getLatestVersion(versions [] string ) ( string , error ) { if len (versions) == 0 { return "" , fmt.Errorf( "no versions provided" ) } // init max with the zero index element maxVersion := versions[ 0 ] maxVersionParts, err := parseVersion(maxVersion) if err != nil { return "" , err } // compare all versions for _, version := range versions[ 1 :] { versionParts, err := parseVersion(version) if err != nil { return "" , err } if compareVersions(versionParts, maxVersionParts) > 0 { maxVersion = version maxVersionParts = versionParts } } return maxVersion, nil } func parseVersion(version string ) ([] int , error ) { // convert "x-y-z" to [x, y, z] parts := strings.Split(version, "-" ) if len (parts) != 3 { return nil , fmt.Errorf( "invalid version format: %s" , version) } intParts := make ([] int , len (parts)) for i, part := range parts { num, err := strconv.Atoi(part) if err != nil { return nil , fmt.Errorf( "invalid number in version: %s" , part) } intParts[i] = num } return intParts, nil } func compareVersions(v1, v2 [] int ) int { // return 1 if v1 > v2, -1 if v1 < v2, 0 if v1 == v2 for i := 0 ; i < len (v1); i++ { // if just one part is greater, the version is greater if v1[i] > v2[i] { return 1 } else if v1[i] < v2[i] { return - 1 } } // if all parts are equal, the versions are equal return 0 } IstioãªãœãŒã¹ã®æ§ç¯ãšåé€ istio.go ã§ã¯IstioãªãœãŒã¹ã®æ§ç¯ãšåé€ãããŸãã istio/client-go ã䜿çšããŠããŸããcreateIstioResources颿°ã§ã¯ãVirtualServiceãæ§ç¯ããŸããdeleteIstioResources颿°ã§ã¯ãVirtualServiceãåé€ããŸãã VirtualServiceã®èšå®ã¯ã /example1/ ã®GETãªã¯ãšã¹ãã«å¯ŸããŠ100ïŒ
ã®ç¢ºçã§100msã®é
å»¶ãçºçããããã®ã§ãããŸãã /example1/ 以å€ã®ãªã¯ãšã¹ãã«å¯ŸããŠã¯é
å»¶ãèšå®ããŠããŸããããã®èšå®ã¯ããµã³ãã«ã®ãããªãã®ã§ãVirtualServiceãæ§ç¯ããŠããæåã§ç·šéããããšãæ³å®ããŠããŸããå°æ¥çã«ã¯ãconfigãã¡ã€ã«ã§APIã®ãã¹ãHTTPã¡ãœãããé
å»¶æéãªã©ãèšå®ã§ããããã«ãããã®èšå®ãå
ã«VirtualServiceãèªåã§æ§ç¯ããããã«æ¹ä¿®ããäºå®ã§ãã istio.goã®ã³ãŒããèŠã func createIstioResources(ctx context.Context) error { // Istio clientset istioClient, err := versioned.NewForConfig(kubeConfig) //... // VirtualService virtualService := &v1alpha3.VirtualService{ ObjectMeta: metav1.ObjectMeta{ Name: resourceName, }, Spec: networkingv1alpha3.VirtualService{ Hosts: [] string { resourceName + "." + namespaceName + ".svc.cluster.local" , }, Http: []*networkingv1alpha3.HTTPRoute{ { Name: "example1" , Match: []*networkingv1alpha3.HTTPMatchRequest{ { Uri: &networkingv1alpha3.StringMatch{ MatchType: &networkingv1alpha3.StringMatch_Prefix{ Prefix: "/example1/" , }, }, Method: &networkingv1alpha3.StringMatch{ MatchType: &networkingv1alpha3.StringMatch_Exact{ Exact: "GET" , }, }, }, }, Fault: &networkingv1alpha3.HTTPFaultInjection{ Delay: &networkingv1alpha3.HTTPFaultInjection_Delay{ Percentage: &networkingv1alpha3.Percent{ Value: 100.0 , }, HttpDelayType: &networkingv1alpha3.HTTPFaultInjection_Delay_FixedDelay{ FixedDelay: &duration.Duration{Nanos: int32 ( 100000000 )}, // 100ms }, }, }, Route: []*networkingv1alpha3.HTTPRouteDestination{ { Destination: &networkingv1alpha3.Destination{ Host: resourceName + "." + namespaceName + ".svc.cluster.local" , }, }, }, }, { Name: "default" , Route: []*networkingv1alpha3.HTTPRouteDestination{ { Destination: &networkingv1alpha3.Destination{ Host: resourceName + "." + namespaceName + ".svc.cluster.local" , }, }, }, }, }, }, } _, err = istioClient.NetworkingV1alpha3().VirtualServices(namespaceName).Create(ctx, virtualService, metav1.CreateOptions{}) //... return nil } func deleteIstioResources(ctx context.Context) error { // Istio clientset istioClient, err := versioned.NewForConfig(kubeConfig) //... err = istioClient.NetworkingV1alpha3().VirtualServices(namespaceName).Delete(ctx, resourceName, metav1.DeleteOptions{}) //... return nil } ä»åŸã®å±æ 远å éçº çŸæç¹ã§ã以äžã®è¿œå éçºãäºå®ããŠããŸãã ãã©ã¡ãŒã¿ãŒãããŒãã³ãŒãã£ã³ã°ïŒ params.go ïŒä»¥å€ã®æ¹æ³ã§èšå®ã§ããããã«ããã ECRãã°ã€ã³ã§ã aws-sdk-go ã䜿çšããŠã aws-cli ãããã°ã©ã äžã§äœ¿ããªãããã«ããã ECR以å€ã®ã³ã³ããã¬ãžã¹ããªã«ã察å¿ããã ECRãªãœãŒã¹ã¿ã°ïŒCostEnvãšCostServiceïŒã®ä»äžã¯ZOZOç¹æãªã®ã§ãªãã·ã§ã³ã«ããã Podã®Affinityèšå®ãå¯èœã«ããã spec.affinity.nodeAffinity ã®matchExpressionsã®key/valueèšå®ãã§ããããã«ããã Istioã®ãµã€ãã«ãŒã³ã³ãããŒã€ã³ãžã§ã¯ã·ã§ã³ããªãã·ã§ã³ã«ããïŒäžèŠãªå ŽåãããããïŒã Istioã®VirtualServiceã®èšå®ãconfigãã¡ã€ã«ã§èšå®ã§ããããã«ãããã®èšå®ãå
ã«VirtualServiceãæ§ç¯ããã ã¬ã€ãã³ã·ãŒã®èšå®å€ãDatadogããèªåã§ååŸããããã«ããã Query Time Seriesãå©çšã§ãããã Goã®ã¯ã©ã€ã¢ã³ã ãããããã openapi.yaml ããAPIãã¹ãšHTTPã¡ãœããã®æ
å ±ãæœåºããŠãDatadogã®ã¯ãšãªã«å©çšããæ³å®ã ãã¹ãã©ã¡ãŒã¿ãŒã®åãæ±ããèæ
®ããå¿
èŠãããã ããå°ãèŠèª¿æ»ã CIã远å ããã ãã¹ãã³ãŒãã远å ããã golangci-lint ã远å ããã 䜿ã£ãŠããã 瀟å
å€å«ããè²ã
ãªè² è·è©Šéšãéçºã§å©çšããã ãã䜿çšå®çžŸãç©ã¿éãããã§ãããŸãããã®äžã§åŸããããã£ãŒãããã¯ãããå¿
èŠã«å¿ããŠissueåããŠãæ©èœè¿œå ããã°ä¿®æ£ãããŸãã ãŸãšã æ¬èšäºã§ã¯ãKubernetesã¯ã©ã¹ã¿ãŒäžã«Prismã䜿ã£ãŠããã€ã¯ããµãŒãã¹ã®OpenAPI仿§ããã®ãŸãŸè¿ãã¢ãã¯ãªãœãŒã¹ããµã¯ããšæ§ç¯ãããã¢ãã¯æ§ç¯ããŒã«ãã玹ä»ããŸãããGoã®ãœãŒã¹ã³ãŒããèªãã æ¹ã¯ãæ°ã¥ãããšæããŸãããã¢ãã¯æ§ç¯ããŒã«èªäœã®å®è£
ã¯é£ãããããŸãããAWSãªãœãŒã¹ãKubernetesãªãœãŒã¹ã®æ§ç¯ãšåé€ãããŠããã ãã§ãã匷ããŠèšãã°ãIstioã®ææ°ããŒãžã§ã³ååŸã®ããžãã¯ãå°ãè€éãããããªãçšåºŠã§ããã·ã³ãã«ã§çè§£ããããããŒã«ã§ãã®ã§ããããããã°ãã²äœ¿ã£ãŠã¿ãŠãã ããããŸããå°ãã§ãè¯ããªãšæã£ãŠããã ãããã GitHubãªããžã㪠ã«ã¹ã¿ãŒãããã ãããšãšãŠãå¬ããã§ããã»ãŒã¯ãããŠã®èªäœOSSãªã®ã§å°ãç·åŒµããŠããŸãã ãªããæ¬ããŒã«ã®äžæ žã§ããPrismã¯çŽ æŽãããOSSã§ãããããPrismã®æŽ»çšããŸã ã§ããããããŒã«ã«ã®åäœç¢ºèªãçµå詊éšãªã©ã«ã䟿å©ã§ãã®ã§ãã²äœ¿ã£ãŠã¿ãŠãã ããã We are hiring ZOZOã§ã¯ãäžç·ã«ãµãŒãã¹ãäœãäžããŠãããæ¹ãåéäžã§ãããèå³ã®ããæ¹ã¯ã以äžã®ãªã³ã¯ãããã²ãå¿åãã ããã hrmos.co corp.zozo.com