åºå
žïŒ https://unsplash.com/photos/JKUTrJ4vK00 BASE BANKæ ªåŒäŒç€Ÿ ã§ãœãããŠã§ã¢ãšã³ãžãã¢ããã£ãŠããæ±å£ïŒ @hgsgtk ïŒã§ãã峿ã«è³é調éãã§ããéèãµãŒãã¹ã YELL BANK(ãšãŒã«ãã³ã¯) ããšãããããã¯ããéçºã»éçšããŠããŸãã ããŠãæ¥ã
ããŠãŒã¶ãŒã«äœ¿ã£ãŠããã ããµãŒãã¹ãéå¶ããŠããäžã§ãããµãŒãã¹ãå®å®çã«æäŸã§ããŠãããããšãã芳ç¹ã«ãããŠã ç£èŠããææ³ ã«ã€ããŠé¢å¿ããããŸãã ãããªæãã å
¥é ç£èŠââã¢ãã³ãªã¢ãã¿ãªã³ã°ã®ããã®ãã¶ã€ã³ãã¿ãŒã³ ããšããæžç±ãæè¿çºå£²ãããäžéçã«ãç£èŠã«ã€ããŠãé¢å¿ãé«ãŸã£ãŠããããšæããŸãã ä»åã¯ããã®æžç±ã®äžãããå®éã«æ¥åã§å®è·µããŠããã Health ãšã³ããã€ã³ããã¿ãŒã³ ãã«ã€ããŠã å®è·µäŸ ãš æžç±ã®å
å®¹ã®æ·±æã ãå«ããŠç޹ä»ããããšæããŸãã ãŸãã Mackerel Meetup #13 Tokyo ãšããã€ãã³ãã§ãä»åã®å
容ãçºè¡šããããŸããã®ã§ããã¡ããåãããŠãåç
§ãã ããã Health ãšã³ããã€ã³ããã¿ãŒã³ãšã¯ ã¢ããªã±ãŒã·ã§ã³ã®å¥å
šæ§ãäŒããã¢ããªã±ãŒã·ã§ã³å
ã®HTTPãšã³ããã€ã³ããäœã ãã¿ãŒã³ã§ãã ã«ããªã¢ãšã³ããã€ã³ãïŒcanary endpointïŒã»ã¹ããŒã¿ã¹ãšã³ããã€ã³ãïŒstatus endpointïŒãšãåŒã°ããç¹ã«ååãã€ããŠããããšãç¥ããã«äœ¿ã£ãŠããæ¹ãå€ãã®ã§ã¯ãªãã§ãããããïŒçè
ããã®äžäººã§ãããïŒ ãã®ãšã³ããã€ã³ãã§ã¯ãæäœéãHTTPãªã¯ãšã¹ããåããŠã¬ã¹ãã³ã¹ãè¿ãããããšããæ
å ±ã®ã¿ãè¿ãããšãã§ããã°ããããã€ãããããŒãžã§ã³ãäŸåé¢ä¿ã®ããDBãªã©ã®ãµãã³ã³ããŒãã³ãã®ã¹ããŒã¿ã¹ãšãã£ãæ
å ±ãŸã§ãã¬ã¹ãã³ã¹ã«å«ããããšãã§ããŸãã 䜿çšçšéãšããŠã¯ã次ã®ãããªã±ãŒã¹ããããããŸãã Mackerel ãªã©ç£èŠSaaSããã® å€åœ¢ç£èŠ ALBãªã© ããŒããã©ã³ãµã®ãã«ã¹ãã§ã㯠ã¢ããªã±ãŒã·ã§ã³ èµ·å確èªã®ãããã° å®éã«ãå®è·µããå Žåãã¢ããªã±ãŒã·ã§ã³ã®ç¶æ
ãäŒããHTTPãšã³ããã€ã³ããäœæããŸããäŸãã°ã /healthcheck ã /ping ãªã©ãšèšã£ãååã«ãªãã§ããããã ãã®ãšã³ããã€ã³ãã¯ã ãã«ã¹ãã§ãã¯ã«æåããã°HTTPã¹ããŒã¿ã¹ã³ãŒã200 ãã 倱æããå Žåã¯200以å€ïŒç¹ã«503ïŒ ãè¿ããšããå®è£
ã«ãªããŸãã BASE BANKã§ã®å®è·µäŸ BASE BANKã§ã¯ãïŒã€ã®ãšã³ããã€ã³ããçšéå¥ã«çšæããæ¹æ³ããšã£ãŠããŠãããããå€åœ¢ç£èŠã®å¯Ÿè±¡ãšããŠããŸãã ã²ãšã€ãã ã³ã³ããåäœã®çåç¢ºèª ãäž»ç®çãšããã /health ãããäžã€ãã äŸåããŠããDBã»Redisãªã©ãžã®æ¥ç¶ãŸã§ã確èªå¯Ÿè±¡ã«å«ãã ã /health/deep ã§ãã ãªãããã®å®è·µäŸã¯ã ã Mackerel Meetup #13 Tokyo ãã§ å±±æ ¹ ( @fumikony ) ããçºè¡šãããã BASEã«ãããMackerelå©çšäžã®å·¥å€«ãšå°ããããšã®ããçŽ¹ä» ãå
ã§ãäºäŸãšããŠèšåããŠãããŸãã Mackerel çã®ç£èŠSaaSãšã®çµã¿åãããšããç¹ã§ã¯ãåãããŠãã¡ããäžèªããã ãããšããè¯ãããªãšæããŸãã åäœã®çå確èª: /health Goèšèªã§å®éã«å®è£
ããå Žåã¯ãäŸãã°æ¬¡ã®ãããªHTTP Handlerã«ãªããŸãã // HTTPã¹ããŒã¿ã¹ããã£ãŒã«ãã«å«ãã¬ã¹ãã³ã¹ãã©ãŒããã type SimpleResponse struct { Status int `json:"status"` Message string `json:"message,omitempty"` Detail string `json:"string,omitempty"` } // 200ã¬ã¹ãã³ã¹ãè¿åŽããã func SimpleHealthCheck(w http.ResponseWriter, r *http.Request) { rs := SimpleResponse{ Status: http.StatusOK, } respondJson(w, rs, http.StatusOK) } // JSON圢åŒã§ã¬ã¹ãã³ã¹ãè¿åŽãã func respondJson(w http.ResponseWriter, body interface {}, status int ) { w.Header().Set( "Content-Type" , "application/json; charset=utf-8" ) w.WriteHeader(status) if err := json.NewEncoder(w).Encode(body); err != nil { fmt.Fprintf(os.Stderr, "failed to encode response by error '%#v'" , err) w.WriteHeader(http.StatusInternalServerError) return } } ãããã /simple/.health_check ã«å¯ŸããŠã«ãŒãã£ã³ã°èšå®ããå Žåã¯æ¬¡ã®ãããªã¬ã¹ãã³ã¹ãšãªããŸãã -> % curl -i http://localhost:8080/simple/.health_check HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 Date: Mon, 04 Mar 2019 13:24:23 GMT Content-Length: 15 {"status":200} ããã¯ãããŒããã©ã³ãµããã®å€åœ¢ç£èŠãªã©ã®éã«å©çšããŠããŸãã äŸåãµãŒãã¹èŸŒã¿ã®åäœç¢ºèª: /health/deep 次ã«ãDBãRedisãªã©äŸåããŠããå€éšãµãŒãã¹ã«æ¥ç¶ã§ããŠãããã©ããã確èªå¯Ÿè±¡ã«å«ãããšã³ããã€ã³ãã§ãã func SimpleDeepHealthCheck(w http.ResponseWriter, r *http.Request) { _, err := NewMySQL(config.DB) if err != nil { // DBãžã®ã³ãã¯ã·ã§ã³ã«ãŠãšã©ãŒãçºçããå Žåã¯503ã¬ã¹ãã³ã¹ rs := SimpleResponse{ Status: http.StatusServiceUnavailable, Message: "failed to get connection database" , Detail: err.Error(), } respondJson(w, rs, rs.Status) return } rs := SimpleResponse{ Status: http.StatusOK, Message: "success to connect server" , } respondJson(w, rs, rs.Status) } ãããã /simple/.health_check/deep ã«å¯ŸããŠã«ãŒãã£ã³ã°èšå®ããå Žåæ¬¡ã®ãããªã¬ã¹ãã³ã¹ãè¿åŽãããŸãã -> % curl -i http://localhost:8080/simple/.health_check/deep HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 Date: Mon, 04 Mar 2019 13:33:22 GMT Content-Length: 53 {"status":200,"message":"success to connect server"} äŸãã°ãããŒã¿ããŒã¹ã«å¯Ÿããæ¥ç¶ã倱æããå Žåã¯ã次ã®ãããª503ã¬ã¹ãã³ã¹ãè¿åŽãããã¢ããªã±ãŒã·ã§ã³ã®å¥åº·ç¶æ
ã«ã€ããŠç¥ãããšãã§ããŸãã -> % curl -i http://localhost:8080/simple/.health_check/deep HTTP/1.1 503 Service Unavailable Content-Type: application/json; charset=utf-8 Date: Mon, 04 Mar 2019 13:35:38 GMT Content-Length: 106 {"status":503,"message":"failed to get connection database","string":"sql: connection is already closed"} ãã®ãã¿ãŒã³ã®ãšã³ããã€ã³ãã¯ã Mackerel ããã®å€åœ¢ç£èŠã§äž»ã«å©çšããŠããŸãã å©ç¹ å€åœ¢ç£èŠã«å©çšã§ãã APIã®éä¿¡ãã§ããããšãã£ãå€åœ¢ç£èŠã¯ç¹ã«ã¢ããªã±ãŒã·ã§ã³ãéçšããŠãããšæäœéé¢å¿ããã€ãšããã ãšæããŸãããã®ãã¿ãŒã³ã§ã¯ãã¹ããŒã¿ã¹ã³ãŒãã§ç¶æ
ãå€å¥ã§ãããããå€åœ¢ç£èŠã«äœ¿ããããããšæããŸãã ãããã°ã«æå¹ èµ·åããã¢ããªã±ãŒã·ã§ã³ãæ£ããHTTPãªã¯ãšã¹ããåŠçããäºãã§ããããç¥ãäžã§ãéå®ããŠããŸãã ããã«ãäŸåé¢ä¿ã®ããDBãªã©ã®ãµãŒãã¹ãžã®ã³ãã¯ã·ã§ã³ãåãããã«ã€ããŠãæäœé確èªã§ãããããèµ·å確èªã«æçšã§ãã äœè«ïŒã³ã³ããããŒã¹ã¢ããªã±ãŒã·ã§ã³ãšã®èŠªåæ§ éçšããŠãã YELL BANK ãšãããµãŒãã¹ã¯ã以åå
¬éããã ECS(Fargate)ã§ã³ã³ããã¢ããªã±ãŒã·ã§ã³ãåããããã®èšå®æ
å ±ã®æ±ãæ¹ ããšããèšäºã§ã玹ä»ããéããã³ã³ããäžã§åäœããããšãåæãšããã¢ããªã±ãŒã·ã§ã³ãšããŠæ©èœæäŸããŠããŸãã ã³ã³ããå
ã§åããã¢ããªã±ãŒã·ã§ã³ã«ãããŠããå€åœ¢ç£èŠã¯éèŠãšæããŠããŸãããå®éãã®ãã¿ãŒã³ã¯ã³ã³ããããŒã¹ã¢ããªã±ãŒã·ã§ã³ã«ãããŠã¯ã©ã®ããã«èããããã§ããããã ã³ã³ããããŒã¹ã¢ããªã±ãŒã·ã§ã³èšèšãšããŠããHealth ãšã³ããã€ã³ããã¿ãŒã³ããã©ãè©äŸ¡ã§ããããèããã«ãããã redhat ãå
¬éããŠããã Principles of container-based application design ã ããåèã«ãªãååãäžã€èŠãŠã¿ãŸãããã ããããã HIGH OBSERVABILITY PRINCIPLE (HOP) é«èŠ³æž¬å¯èœæ§ã®åå ã ãšããèšèšååã§ãã ããã¯ãã³ã³ããå
éšããã©ãã¯ããã¯ã¹ã®ããã«æ±ãèšèšåæãæã£ãäžã§ãèªèº«ã®ã¢ããªã±ãŒã·ã§ã³ã®æŽ»åç¶æ³ãæºåç¶æ³ãªã©ãæ§ã
ãªç¶æ
ãã§ãã¯ã«ã€ããŠAPIãæäŸãããšããèšèšã«ã€ããŠèšåããŠããŸãã èªèº«ã®å¥åº·ç¶æ
ãäŒãããHealth ãšã³ããã€ã³ããã¿ãŒã³ããã芳枬床ãäžããäžã§æçšãªãã¿ãŒã³ãšèšãããã§ããã ãã©ããæ®µéã®å
±éã¬ã¹ãã³ã¹ãã©ãŒãããã«ã€ã㊠ããŠããHealth ãšã³ããã€ã³ããã¿ãŒã³ãã«ã€ããŠããã¿ãŒã³ãšå®è·µã«ã€ããŠèŠããšããã§ãå°ãè©±ãæ·±æãããŠã Health Check Response Format for HTTP APIs ããšããè°è«äžã®å
±éã¬ã¹ãã³ã¹ãã©ãŒãããã«ã€ããŠèŠãŠãããããšæããŸãã ããã¯ãã å
¥é ç£èŠââã¢ãã³ãªã¢ãã¿ãªã³ã°ã®ããã®ãã¶ã€ã³ãã¿ãŒã³ ãã®ãä»é²Cããšããç« ã§èšåãããŠãããã®ã§ãã ã©ããã£ãã¬ã¹ãã³ã¹ãè¿ãã¹ãããè°è«äžã®ãã®ãã©ãŒãããã«ã€ããŠå°ãæ·±æãããŠã¿ãŸãã Health Check Response Format for HTTP APIs ãã®ãã©ãŒãããã¯ã倧ãã以äžã®3ã€ã®ç¹åŸŽãæã¡ãŸãã JSONãã©ãŒããã ãå©çšãã media-typeã¯ã application/health+json ãšãã å¿
é ãã£ãŒã«ãã§ãã status ãš ããã€ãã®ãªãã·ã§ãã«ãªãã£ãŒã«ããå«ã å¿
é ãã£ãŒã«ã: status status ã«ã¯ã次ã®ã©ããã®å€ãèšå®ããããšãããŠããŸãã pass : healthy status code: 2xx-3xx range (MUST) ãã®ä»ãäžèšã®éžæè¢ãå¯èœ Node's Terminus ããµããŒãããããã® ok Javaã®SpringBootã®ããã® up fail : unhealthy status code: 4xx-5xx range (MUST) ãã®ä»ãäžèšã®éžæè¢ãå¯èœ Node's Terminus ããµããŒãããããã® error Javaã®SpringBootã®ããã® down warn : healthy, with some concerns status code: 2xx-3xx range (MUST) warningã¬ãã«ã®ã¿ã€ããèšå®ã§ããã®ã¯äžã€é¢çœããšããããªãšæããŸããã ãã®ä»ãã©ãŒãããäžèЧ å¿
é ãšãããŠãã status 以å€ã«ã次ã®ãã£ãŒã«ãããªãã·ã§ãã«ãªé
ç®ãšããŠæç€ºãããŠããŸãã status version (optional) - ãµãŒãã¹ã®å
¬éããŒãžã§ã³ releaseId (optional) notes (optional) - å¥åº·ç¶æ
ã«é¢ããèšè¿° output (optional) - çã®ãšã©ãŒåºåã statusã pass ã®ãšãã¯çç¥ãã¹ã details (optional) - äŸåããŠãããµãŒãã¹ãå«ãã詳现æ
å ±ã The Details Object ãšãããªããžã§ã¯ãã§å®çŸ© links (optional) - ãã詳现æ
å ±ãåŸãããã®å€éšURLãªã© serviceId (optional) - ã¢ããªã±ãŒã·ã§ã³ã¹ã³ãŒããªãŠããŒã¯èå¥å description (optional) - 人éã«åªãã説æ The Details Object äžæµã®äŸåé¢ä¿ãRedisãªã©ã®ã¢ããªã±ãŒã·ã§ã³ããèŠããµãã³ã³ããŒãã³ãã®ç¶æ
ãäŒããããã®ãªããžã§ã¯ããšããŠæç€ºãããŠããŸããå
¬åŒã®äŸã§ã¯ã以äžã®ããã«cassandraãcpuã»memoryç¶æ
ãªã©ã¢ããªã±ãŒã·ã§ã³ãäŸåãããã®ã«ã€ããŠã®å¥åº·ç¶æ
ãäŒããäŸã瀺ããŠããŸãã " details ": { " cassandra:responseTime ": [ { " componentId ": " dfd6cf2b-1b6e-4412-a0b8-f6f7797a60d2 ", " componentType ": " datastore ", " observedValue ": 250 , " observedUnit ": " ms ", " status ": " pass ", " time ": " 2018-01-17T03:36:48Z ", " output ": "" } ] , " cassandra:connections ": [ { " componentId ": " dfd6cf2b-1b6e-4412-a0b8-f6f7797a60d2 ", " type ": " datastore ", " observedValue ": 75 , " status ": " warn ", " time ": " 2018-01-17T03:36:48Z ", " output ": "", " links ": { " self ": " http://api.example.com/dbnode/dfd6cf2b/health " } } ] , " uptime ": [ { " componentType ": " system ", " observedValue ": 1209600.245 , " observedUnit ": " s ", " status ": " pass ", " time ": " 2018-01-17T03:36:48Z " } ] , } , inadarei.github.io Health Check Response Format for HTTP APIs ããåŠã¶ããš ããã¯ãçŸæç¹ã§ã¯ããã©ããçãªã®ã§æ£åŒã«å®ãã¹ãæšæºã»å¶çŽãšããããã§ã¯ãããŸãããããããå®éã«ãHealth ãšã³ããã¿ãŒã³ããå®è·µããã«ããã£ãŠã詳现ãªç¹ã«ã€ããŠè¿·ããçãŸããéã«ãã®ããã«è°è«ããŠããå Žæããããšç¥ããšãåèã«ãªãããšæããŸãã æåŸã« ç§ã¯ãPHPãGoèšèªã§ã®ã¢ããªã±ãŒã·ã§ã³éçºãã¡ã€ã³ã®ãµãŒããŒãµã€ããšã³ãžãã¢ã§ããããã®ãããªèŠç¹ã§ããã å
¥é ç£èŠââã¢ãã³ãªã¢ãã¿ãªã³ã°ã®ããã®ãã¶ã€ã³ãã¿ãŒã³ ãã¯éåžžã«å匷ã«ãªããŸãã è¿·ãããŠããæ¹ã¯ãã²ãæã«ãšã£ãŠèŠãŠã¯ãããã§ããããã ãŸãã BASEæ ªåŒäŒç€Ÿ ã¯ããµãŒãã¹ã®ç¶ç¶çãªæäŸãå®ãã»çºå±ãããŠãããããããªæ¹ãåéäžã§ãããèå³ãããã°ãã²ãæ°è»œã«éã³ã«ããããŠãã ããã binc.jp