ãã®æçš¿ã¯ã Amazon DynamoDB ã°ããŒãã«ããŒãã«ã®ãã¹ããã©ã¯ãã£ã¹ã«é¢ããã·ãªãŒãºã®ããŒã 3 ã§ãã ããŒã 1 ã§ã¯ããªãŒãžã§ã³é害ã«åããŠã¢ããªã±ãŒã·ã§ã³ãã©ã®ããã«æºåãããã«ã€ããŠè°è«ããŸããã ããŒã 2 ã§ã¯ããªãŒãžã§ã³é害æã®ãã§ã€ã«ãªãŒããŒæŠç¥ã«ã€ããŠåãäžããŸããã ãã®æçš¿ã§ã¯ã AWS Fault Injection Service (AWS FIS) ã䜿çšããŠã DynamoDB ã°ããŒãã«ããŒãã« ã«å¯Ÿããå¶åŸ¡ãããå®éšãå®è¡ããããšã§ãã¢ããªã±ãŒã·ã§ã³ããªãŒãžã§ã³é害ãæåŸ
ã©ããã«åŠçããããšãæ€èšŒããæ¹æ³ã玹ä»ããŸããAWS FIS ã¯äž¡è
ã§åäœãç°ãªãããã ãã«ããªãŒãžã§ã³åŒ·æŽåæ§ (multi-Region strong consistency) (MRSC) ãš ãã«ããªãŒãžã§ã³çµææŽåæ§ (multi-Region eventually consistent) (MREC) ã®äž¡æ¹ã®ã°ããŒãã«ããŒãã«ãåãäžããŸãã ã¬ãžãªãšã³ããªãã«ããªãŒãžã§ã³ã¢ããªã±ãŒã·ã§ã³ãæ§ç¯ããã«ã¯ã2 ã€ã®ç°ãªã課é¡ããããŸããããã¯ãéå®³ãæ³å®ããŠèšèšããããšãšãèšèšãæ©èœããããšã蚌æããããšã§ããæ¬ã·ãªãŒãºã®æåã® 2 ã€ã®ããŒãã§ã¯èšèšã«ã€ããŠæ±ããŸããããã®æçš¿ã§ã¯èšŒæã«ã€ããŠæ±ããŸãã AWS Fault Injection Service AWS Fault Injection Service ã¯ãç¹å®ã®æ¡ä»¶ãæºããããå Žåã«å®éšãèªåçã«ããŒã«ããã¯ãŸãã¯åæ¢ãããªã©ãAWS ã¯ãŒã¯ããŒãäžã§å®éšãå®è¡ããããã«ããŒã ãå¿
èŠãšããå¶åŸ¡ãšã¬ãŒãã¬ãŒã«ãæäŸããŸãã å®éšãã³ãã¬ãŒã (experiment templates) ãå®çŸ©ããŠã察象ãšãããªãœãŒã¹ã泚å
¥ããé害ãå®éšã®æéãæå®ããŸããAWS FIS ã¯ãå®çŸ©ããã圱é¿ãããå€ãè¶
ããå Žåã«å®éšãèªåçã«åæ¢ãã忢æ¡ä»¶ãå«ãããã«ãã€ã³ãããå®å
šã¬ãŒãã¬ãŒã«ãšãšãã«é害泚å
¥ãåŠçããŸãã ãªã AWS FIS ã§ãã¹ãããã®ã DynamoDB ã°ããŒãã«ããŒãã«ã¯ãMRSC ã§ã MREC ã§ããé«å¯çšæ§ãåãããã«ããªãŒãžã§ã³ã¬ããªã±ãŒã·ã§ã³ãæäŸããŸããå®åžžç¶æ
ã®ã¬ããªã±ãŒã·ã§ã³ã§ã¯ãã¬ããªã±ãŒã·ã§ã³ãäžæããããšãã«ã¢ããªã±ãŒã·ã§ã³ãã©ã®ããã«åäœãããã¯åãããŸãããæ¬¡ã®ãããªçåã¯ããã¹ããªãã§ã¯çããã®ãå°é£ã§ãã ãªãŒãžã§ã³é害æã« DynamoDB ããšã©ãŒãè¿ããå Žåãã¢ããªã±ãŒã·ã§ã³ã¯ãã§ã€ã«ãªãŒããŒããã®ãããããšãã¯ã©ãã·ã¥ããã®ã? ã¢ããªã±ãŒã·ã§ã³ã¯ããã§ã€ã«ãªãŒããŒã¢ã¯ã·ã§ã³ãå¿
èŠãšãªãéå®³ãæ©æã«æ€åºã§ããã? ãã§ã€ã«ãªãŒããŒã¡ã«ããºã ã¯ãç®æšåŸ©æ§æé (RTO) å
ã«ãã©ãã£ãã¯ãã·ããã§ããã? é害ã解決ããåŸã埩æ§ãããªãŒãžã§ã³ã§ DynamoDB ã¬ããªã±ãŒã·ã§ã³ããŸã ãã£ããã¢ããäžã§ããå¯èœæ§ããããŸããã¢ããªã±ãŒã·ã§ã³ã¯æåä»å
¥ãªãã§åŸ©æ§ããã? AWS FIS ã®å®éšã«ãããæšæž¬ã§ã¯ãªã蚌æ ããã£ãŠãããã®è³ªåã«çããããšãã§ããŸãã MRSC ã°ããŒãã«ããŒãã«ã§ã® AWS FIS å®éšã®ä»çµã¿ MRSC ã°ããŒãã«ããŒãã«ã«å¯Ÿã㊠aws:dynamodb:global-table-pause-replication ã¢ã¯ã·ã§ã³ã䜿çšã㊠AWS FIS å®éšãå®è¡ãããšãå®éšãªãŒãžã§ã³ãšä»ã®ãã¹ãŠã®ãªãŒãžã§ã³éã®ã¬ããªã±ãŒã·ã§ã³ãäžæåæ¢ãããŸããäŸãã°ãç±³åœè¥¿éš (ãªã¬ãŽã³) (us-west-2)ãç±³åœæ±éš (ããŒãžãã¢åéš) (us-east-1)ãç±³åœæ±éš (ãªãã€ãª) (us-east-2) ã«ã¬ããªã«ãæã€ MRSC ã°ããŒãã«ããŒãã«ãèããŠã¿ãŸããããus-east-2 ãå®éšãªãŒãžã§ã³ãšããŠå®éšãå®è¡ãããšãus-east-2 ãžã®åºå
¥ãã®ã¬ããªã±ãŒã·ã§ã³ã圱é¿ãåããus-west-2 ãš us-east-1 ã¯éåžžã®åäœãç¶ç¶ããŸãã å®éšäžãéé¢ãããã¬ããªã«ã¯æäœã®çš®é¡ã«å¿ããŠç°ãªãåäœãããŸãã çµææŽåæ§ã®ããèªã¿èŸŒã¿ (Eventually consistent reads) â èš±å¯ãããéé¢ãããã¬ããªã«ã«å¯ŸããŠåŒãç¶ãæ©èœããŸãã 匷æŽåæ§ã®ããèªã¿èŸŒã¿ (Strongly consistent reads) â ã¯ãã¹ãªãŒãžã§ã³ã®åæã«å°éã§ããªããããHTTP 500 ãšã©ãŒãè¿ããŸãã æžã蟌㿠(Writes) â ã¯ãã¹ãªãŒãžã§ã³ã¬ããªã±ãŒã·ã§ã³ãäžæåæ¢ãããŠãããããHTTP 500 ãšã©ãŒãè¿ããŸãã ã³ã³ãããŒã«ãã¬ãŒã³ã¢ã¯ã·ã§ã³ ( UpdateTable ãªã©) â ãã¹ãŠã®ã¬ããªã«ãªãŒãžã§ã³ã§ãããŒãã«å
šäœã«ããã£ãŠãããã¯ãããŸãã ãããã®èªçºãšã©ãŒã¯ãå°çšã® Amazon CloudWatch ã¡ããªã¯ã¹ FaultInjectionServiceInducedErrors ã«ãã£ãŠè¿œè·¡ãããŸãããã®ã¡ããªã¯ã¹ã¯ãå®éšäžããã³å®éšåŸã®ãã£ããã¢ããæéäžã«è¿ãããã·ãã¥ã¬ãŒãããã HTTP 500 ãšã©ãŒãã«ãŠã³ããã TableName ãš Operation ã§åè§£ãããŸãããããã®èªçºãšã©ãŒã¯ã SystemErrors ã¡ããªã¯ã¹ãåãã«ãŠã³ãã§ã€ã³ã¯ãªã¡ã³ãããããã SystemErrors ã«å¯Ÿããæ¢åã®ã¢ã©ãŒã ã¯å®éšäžã«çºç«ããããšã«æ³šæããŠãã ããã FaultInjectionServiceInducedErrors ã¡ããªã¯ã¹ã¯ãå®éšèªçºãšã©ãŒãšææ©çãªãµãŒãã¹ã®åé¡ãåºå¥ããã®ã«åœ¹ç«ã¡ãŸãã MRSC å®éšã®ããã® IAM æš©é MRSC å®éšã§ã¯ aws:dynamodb:global-table-pause-replication ã¢ã¯ã·ã§ã³ã䜿çšããã dynamodb:InjectError IAM æš©éãå¿
èŠã§ãããã®æš©éã¯ç¹å®ã®ããŒãã« ARN ã«ã¹ã³ãŒããèšå®ã§ããªãããã Resource: "*" ã§ä»äžããå¿
èŠããããŸããå®éšããŒã«ã«ã¯ããªãœãŒã¹ããªã·ãŒã管çããããã®æš©é (AWS FIS ã«ãã£ãŠã»ããã¢ãããšãã¢ããŠã³ã«å
éšçã«äœ¿çšãããŸã) ãšãã¿ã°ã§ããŒãã«ã察象ãšããå Žå㯠tag:GetResources ãå¿
èŠã§ãã { "Version": "2012-10-17", "Statement": [ { "Sid": "DynamoDBTableActions", "Effect": "Allow", "Action": [ "dynamodb:PutResourcePolicy", "dynamodb:DeleteResourcePolicy", "dynamodb:GetResourcePolicy", "dynamodb:DescribeTable" ], "Resource": [ "arn:aws:dynamodb:*:111122223333:table/my-mrsc-table", "arn:aws:dynamodb:*:111122223333:table/my-mrsc-table/*" ] }, { "Sid": "DynamoDBInjectError", "Effect": "Allow", "Action": "dynamodb:InjectError", "Resource": "*" }, { "Sid": "TagLookup", "Effect": "Allow", "Action": "tag:GetResources", "Resource": "*" } ] } MREC ã°ããŒãã«ããŒãã«ã§ã® AWS FIS ã®åäœ MREC ã°ããŒãã«ããŒãã«ã®å ŽåãAWS FIS ã¯ãªãœãŒã¹ããªã·ãŒããŒã¹ã®ã¢ãããŒãã䜿çšããŸããMREC ããŒãã«ã«å¯ŸããŠåã aws:dynamodb:global-table-pause-replication ã¢ã¯ã·ã§ã³ãå®è¡ãããšãAWS FIS ã¯ãDynamoDB ã¬ããªã±ãŒã·ã§ã³ ãµãŒãã¹ã«ãªã³ã¯ãããããŒã« (service-linked role) (SLR) ãã¬ããªã±ãŒã·ã§ã³æäœãå®è¡ããããšããããã¯ãã deny ã¹ããŒãã¡ã³ããããŒãã«ã®ãªãœãŒã¹ããªã·ãŒã«åçã«ã¢ã¿ããããŸããå
·äœçã«ã¯: ã¬ããªã±ãŒã·ã§ã³ SLR ã«å¯Ÿã㊠GetItem ã PutItem ã UpdateItem ã DeleteItem ã DescribeTable ã UpdateTable ã Scan ã DescribeTimeToLive ã UpdateTimeToLive ããããã¯ãã deny ã¹ããŒãã¡ã³ããããŒãã«ããªã·ãŒã«è¿œå ãããŸãã ã¬ããªã±ãŒã·ã§ã³ SLR ã«å¯Ÿã㊠GetRecords ã DescribeStream ã GetShardIterator ããããã¯ãã deny ã¹ããŒãã¡ã³ããã¹ããªãŒã ããªã·ãŒã«è¿œå ãããŸãã äž¡æ¹ã®ã¹ããŒãã¡ã³ãã«ã¯ãå®éšã®äºæ³çµäºæå»ã®ããã®ãã«ãã€ã³ãããå®å
šã¡ã«ããºã ãæäŸãããæéå¶éä»ãã® DateLessThan æ¡ä»¶ãå«ãŸããŠããŸãã MRSC ãšã®éèŠãªéã: ã¢ããªã±ãŒã·ã§ã³ã¯ãšã©ãŒãèªèããŸãããèªã¿èŸŒã¿ãšæžã蟌ã¿ã¯ããã¹ãŠã®ãªãŒãžã§ã³ã§ããŒã«ã«ã«æåãç¶ããŸãã忢ããã®ã¯ããã¯ã°ã©ãŠã³ãã®ã¬ããªã±ãŒã·ã§ã³ã§ãããããªãŒãžã§ã³ã§ã®æžã蟌ã¿ãä»ã®ãªãŒãžã§ã³ã«äŒæãããªããªããä»ã®ãªãŒãžã§ã³ããã®æžã蟌ã¿ãéé¢ããããªãŒãžã§ã³ã«äŒæãããŸãããããã¯ä»¥äžãæå³ããŸãã ã¢ããªã±ãŒã·ã§ã³ã¯ãŠãŒã¶ãŒã®èгç¹ããã¯éåžžéãæ©èœãç¶ããŸãã å®éšäžã«æžã蟌ãŸããããŒã¿ã¯ãªãŒãžã§ã³éã§ä¹é¢ããŸãã ReplicationLatency CloudWatch ã¡ããªã¯ã¹ã¯åºåãããªããªããŸãã å®éšãçµäºãããšãAWS FIS 㯠deny ã¹ããŒãã¡ã³ããåé€ããã¬ããªã±ãŒã·ã§ã³ããã£ããã¢ããããŸãã 察象ããŒãã«ã«ãŸã ãªãœãŒã¹ããªã·ãŒããªãå ŽåãAWS FIS ã¯å®éšæéäžã«ããªã·ãŒãäœæããå®éšãå®äºãããšåé€ããŸããããŒãã«ã«ãã§ã«ãªãœãŒã¹ããªã·ãŒãããå ŽåãAWS FIS 㯠deny ã¹ããŒãã¡ã³ãã®ã¿ãæ¿å
¥ããæåŸã«åé€ããŠãæ¢åã®ããªã·ãŒããã®ãŸãŸæ®ããŸãã MREC å®éšã®ããã® IAM æš©é MREC å®éšã¯ãªãœãŒã¹ããªã·ãŒããŒã¹ã®ã¡ã«ããºã ã䜿çšãã dynamodb:InjectError æš©éãå¿
èŠãšããŸãã: { "Version": "2012-10-17", "Statement": [ { "Sid": "DynamoDBActions", "Effect": "Allow", "Action": [ "dynamodb:PutResourcePolicy", "dynamodb:DeleteResourcePolicy", "dynamodb:GetResourcePolicy", "dynamodb:DescribeTable" ], "Resource": [ "arn:aws:dynamodb:*:111122223333:table/my-mrec-table", "arn:aws:dynamodb:*:111122223333:table/my-mrec-table/*" ] }, { "Sid": "TagLookup", "Effect": "Allow", "Action": "tag:GetResources", "Resource": "*" } ] } åææ¡ä»¶ éå§ããåã«ã以äžã確èªããŠãã ããã è€æ°ã®ãªãŒãžã§ã³ã«ã¬ããªã«ãæã€ DynamoDB ã°ããŒãã«ããŒãã« (MRSC ãŸã㯠MREC)ããã®ãŠã©ãŒã¯ã¹ã«ãŒã§ã¯ãus-west-2 (ãã©ã€ããª)ãus-east-1ãus-east-2 ã«ãã MRSC ããŒãã«ã䜿çšããŸãã AWS FIS å®éšçšã® IAM ããŒã«ã AWS Command Line Interface (AWS CLI) v2 ãã€ã³ã¹ããŒã«ãããèšå®ãããŠããããšã CloudWatch ããã·ã¥ããŒããŸãã¯ç£èŠãœãªã¥ãŒã·ã§ã³ãæŽã£ãŠããããšããããªãã§ã¯ãå®éšã®åœ±é¿ã枬å®ã§ããŸããã éèŠ: AWS FIS ã¯å®éã® AWS ãªãœãŒã¹ã«å¯ŸããŠå®éã®ã¢ã¯ã·ã§ã³ãå®è¡ããŸãããããåããŠã® AWS FIS å®éšã®å Žåã¯ãæ¬çªåç°å¢ãŸãã¯ãã¹ãç°å¢ã§éå§ããããšãåŒ·ãæšå¥šããŸããæ¬çªããŒãã«ã«å¯ŸããŠå®è¡ããåã«ãå®éšèšèšãšç£èŠã«å¯Ÿããèªä¿¡ãæ§ç¯ããŠãã ããã ã¹ããã 1: å®åžžç¶æ
ãå®çŸ©ãã å®éšãäœæããåã«ãã¢ããªã±ãŒã·ã§ã³ã«ãšã£ãŠãæ£åžžããã©ã®ãããªãã®ãããŸãå®éšäžã®æåŸ
ãããç¶æ
ãå®çŸ©ããŸãã äŸ: ãus-east-2 ã§ã¬ããªã±ãŒã·ã§ã³ã 10 åéäžæåæ¢ãããå Žåãã¢ããªã±ãŒã·ã§ã³ã¯ 30 ç§ä»¥å
ã« us-west-2 ã«ãã§ã€ã«ãªãŒããŒããã®ã§ããšã³ããŠãŒã¶ãŒãèªèãããšã©ãŒç㯠0.1% ãè¶
ããŠã¯ãªããªããã ãus-east-2 ã§åŒ·æŽåæ§ã®ããèªã¿èŸŒã¿ã倱æããå Žåãã¢ããªã±ãŒã·ã§ã³ã¯çµææŽåæ§ã®ããèªã¿èŸŒã¿ã«ãã©ãŒã«ããã¯ãããã®ãªãŒãžã§ã³ã®èªã¿èŸŒã¿æäœã® p99 ã¬ã€ãã³ã·ã 50ms 以äžã«ç¶æããå¿
èŠããããã ããŒã¹ã©ã€ã³ã確ç«ããããã«ãéåžžéçšäžã®ä»¥äžã®ã¡ããªã¯ã¹ãå°ãªããšã 24 æéèšé²ããŸãã æäœããšã® SuccessfulRequestLatency (p50ãp99)ã SystemErrors (åèš)ããŒãã«è¿ãå€ã§ããã¹ãã§ãã ã¢ããªã±ãŒã·ã§ã³ã¬ãã«ã®ãšã©ãŒçãšã¬ã€ãã³ã·ã¡ããªã¯ã¹ã ãããã®ããŒã¹ã©ã€ã³ãå®éšäžããã³å®éšåŸã®åãã¡ããªã¯ã¹ãšæ¯èŒããåºæºãè©äŸ¡ããŸãã ã¹ããã 2: AWS FIS å®éšããŒã«ãäœæãã AWS FIS ãã°ããŒãã«ããŒãã«ã«å¯ŸããŠå®éšãå®è¡ããããã«åŒãåããããšãã§ãã IAM ããŒã«ãäœæããŸãããã®ããŒã«ã«ã¯ããªãœãŒã¹ããªã·ãŒã管çããæš©é (AWS FIS ã«ãã£ãŠå
éšçã«äœ¿çšãããŸã) ãšãMRSC ããŒãã«åºæã® dynamodb:InjectError æš©éãå¿
èŠã§ãã dynamodb:InjectError ã«ã¯ Resource: "*" ãå¿
èŠã§ããããšã«æ³šæããŠãã ãããç¹å®ã®ããŒãã« ARN ã«ã¹ã³ãŒãããããšã¯ã§ããŸããã å®éšããŒã«ããªã·ãŒ: { "Version": "2012-10-17", "Statement": [ { "Sid": "DynamoDBTableActions", "Effect": "Allow", "Action": [ "dynamodb:PutResourcePolicy", "dynamodb:DeleteResourcePolicy", "dynamodb:GetResourcePolicy", "dynamodb:DescribeTable" ], "Resource": [ "arn:aws:dynamodb:*:111122223333:table/my-mrsc-table", "arn:aws:dynamodb:*:111122223333:table/my-mrsc-table/*" ] }, { "Sid": "DynamoDBInjectError", "Effect": "Allow", "Action": "dynamodb:InjectError", "Resource": "*" } ] } ä¿¡é Œããªã·ãŒ: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "fis.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } ã¹ããã 3: å®éšãã³ãã¬ãŒããäœæãã 以äžã®ã³ãã³ãã¯ãã¬ããªã±ãŒã·ã§ã³ã 10 åéäžæåæ¢ããå®éšãã³ãã¬ãŒããäœæããŸãã忢æ¡ä»¶ã§ãã CloudWatch ã¢ã©ãŒã ã«æ³šç®ããŠãã ãããã¢ããªã±ãŒã·ã§ã³ã®ãšã©ãŒçã蚱容ãããå€ãè¶
ããå Žåã«ãå®éšãèªåçã«åæ¢ããŸãã aws fis create-experiment-template \ --description "MRSC replication pause - 10 minute resilience test" \ --targets '{ "myGlobalTable": { "resourceType": "aws:dynamodb:global-table", "resourceArns": [ "arn:aws:dynamodb:us-east-2:111122223333:table/my-mrsc-table" ], "selectionMode": "ALL" } }' \ --actions '{ "pauseReplication": { "actionId": "aws:dynamodb:global-table-pause-replication", "parameters": { "duration": "PT10M" }, "targets": { "Tables": "myGlobalTable" } } }' \ --stop-conditions '[{ "source": "aws:cloudwatch:alarm", "value": "arn:aws:cloudwatch:us-east-2:111122223333:alarm:MRSC-Critical-Error-Rate" }]' \ --role-arn "arn:aws:iam::111122223333:role/FIS-DynamoDB-Experiment-Role" ARN ã®ä»£ããã«ã¿ã°ã§ããŒãã«ã察象ã«ããããšãã§ããããã¯å
±éã®ã¿ã° (äŸãã° Environment: staging ) ãå
±æããè€æ°ã®ããŒãã«éã§å®éšãå®è¡ãããå Žåã«äŸ¿å©ã§ãããããè¡ãã«ã¯ã察象å®çŸ©ã® resourceArns ã resourceTags ã«çœ®ãæããŸã: "myGlobalTable": { "resourceType": "aws:dynamodb:global-table", "resourceTags": { "Environment": "staging" }, "selectionMode": "ALL" } ã¹ããã 4: å®éšãå®è¡ãã å®éšãéå§ããŸã: aws fis start-experiment \ --experiment-template-id EXT1234567890abcdef0 å®éšã¹ããŒã¿ã¹ãç£èŠããŸã: aws fis get-experiment \ --id EXP1234567890abcdef0 å®éšãå®è¡ãããŠããéãCloudWatch ããã·ã¥ããŒããšãéé¢ããããªãŒãžã§ã³ã ãã§ãªããã¹ãŠã®ã¬ããªã«ãªãŒãžã§ã³ã§ã®ã¢ããªã±ãŒã·ã§ã³ã®åäœã芳å¯ããŸãã å®éšäž: MRSC ããŒãã«ã®å Žå: FaultInjectionServiceInducedErrors ã¯ãéé¢ããããªãŒãžã§ã³å
ã§èªçºããã 500 ãšã©ãŒã®å®å®ããã«ãŠã³ããæäœ ( GetItem ã PutItem ã Query ãªã©) ããšã«è¡šç€ºããã¯ãã§ãã SystemErrors ã¯åãã«ãŠã³ãã ãå¢å ããŸãããã®ã¡ããªã¯ã¹ã«å¯Ÿããæ¢åã®ã¢ã©ãŒã ãçºç«ããŸãã çµææŽåæ§ã®ããèªã¿èŸŒã¿ã¯ãéé¢ããããªãŒãžã§ã³ã§æåãç¶ããã¯ãã§ãããããæäœã§ãã£ã«ã¿ãªã³ã°ããã SuccessfulRequestLatency ã® SampleCount ã§ç¢ºèªããŸãã éé¢ããããªãŒãžã§ã³å
ã®åŒ·æŽåæ§ã®ããèªã¿èŸŒã¿ãšæžã蟌ã¿ã¯ãHTTP 500 ãšã©ãŒãè¿ãã¯ãã§ãã å¥å
šãªãªãŒãžã§ã³ã¯éåžžéãåäœãç¶ããã¯ãã§ãããããã®ãªãŒãžã§ã³ã§ SuccessfulRequestLatency ã§ç¢ºèªããŸãã ãªãŒãžã§ãã«ãã§ã€ã«ãªãŒããŒãèšå®ãããŠããå Žå (äŸãã°ãAmazon Route 53 ãã«ã¹ãã§ãã¯ãéããŠ)ããã©ãã£ãã¯ãç®æš RTO å
ã«å¥å
šãªãªãŒãžã§ã³ã«ã·ããããããšã確èªããŸãã MREC ããŒãã«ã®å Žå: ã¢ããªã±ãŒã·ã§ã³ã¯ãã¹ãŠã®ãªãŒãžã§ã³ã§éåžžéãåäœãç¶ããå¿
èŠããããŸããèªã¿èŸŒã¿ãšæžã蟌ã¿ã¯ããŒã«ã«ã§æåããŸãã å¥å
šãªãªãŒãžã§ã³éã® ReplicationLatency ãç£èŠããŠãããããäºãã®éã§ã¬ããªã±ãŒã·ã§ã³ãç¶ç¶ããŠããããšã確èªããŸãã å®éšãªãŒãžã§ã³ãžã® ReplicationLatency ã¡ããªã¯ã¹ã¯åºåãããªããªããŸãã ã¢ããªã±ãŒã·ã§ã³ãã°ã«ãšã©ãŒã衚瀺ãããªãã¯ãã§ããã¬ããªã±ãŒã·ã§ã³ã®äžæåæ¢ã¯ã¢ããªã±ãŒã·ã§ã³ã«ã¯èŠããŸããã å®éšçµäºåŸ: MRSC ããŒãã«ã®å Žå: DynamoDB ãã¬ããªã±ãŒã·ã§ã³ãåéããŠåæããéã FaultInjectionServiceInducedErrors ã¯ãã£ããã¢ããæéäžã«çæéç¶ç¶ããå¯èœæ§ããããŸãã SuccessfulRequestLatency ã® SampleCount ã¯ã以åã«éé¢ãããŠãããªãŒãžã§ã³ã§å®éšåã®ã¬ãã«ã«å埩ããã¯ãã§ãã ã¯ã©ã€ã¢ã³ãã忥ç¶ããéã®ãªãã©ã€ã¹ããŒã ã«ã€ã㊠ThrottledRequests ã芳å¯ããŸãã MREC ããŒãã«ã®å Žå: å®éšãªãŒãžã§ã³ãžã® ReplicationLatency ã¡ããªã¯ã¹ãåºåãéå§ããã¬ããªã±ãŒã·ã§ã³ã®ããã¯ãã°ãæåºãããã«ã€ããŠæžå°ããã¯ãã§ãã å®éšäžã«å¥å
šãªãªãŒãžã§ã³ã«æžã蟌ãŸããããŒã¿ãã以åã«éé¢ãããŠãããªãŒãžã§ã³ã§èªã¿åããããã«ãªã£ãŠããããã®éãåæ§ã§ããããšã確èªããŸãã äž¡æ¹ã®ããŒãã«ã¿ã€ãã®å Žå: ã¢ããªã±ãŒã·ã§ã³ãæåä»å
¥ãªãã§å®åžžç¶æ
ã«æ»ãããšã確èªããŸãã ã³ã³ãããŒã«ãã¬ãŒã³æäœãåã³æåããã¯ãã§ãã ã¹ããã 5: çµæãè©äŸ¡ãã å®éšåŸãã¹ããã 1 ã§å®çŸ©ããåºæºã«æ»ããåéããããŒã¿ãšç
§ããåãããŠè©äŸ¡ããŸããæ¬¡ã®è³ªåãããŠãã ããã ã¢ããªã±ãŒã·ã§ã³ã¯ 500 ãšã©ãŒãé©åã«åŠçããã? ã¢ããªã±ãŒã·ã§ã³ãã°ã§æªåŠçã®äŸå€ã確èªããŸãããšã©ãŒããšã³ããŠãŒã¶ãŒã«äŒæããå Žåã¯ãSDK ã®ãªãã©ã€èšå®ã調æŽãããããµãŒããããã¬ãŒã«ãŒã䜿çšããã¢ããªã±ãŒã·ã§ã³ã¬ãã«ã®ãªãã©ã€ããžãã¯ã远å ããããšãæ€èšããŠãã ããã çµææŽåæ§ã®ããèªã¿èŸŒã¿ã¯æçšãªãã©ãŒã«ããã¯ãæäŸããã? å®éšäžãçµææŽåæ§ã®ããèªã¿èŸŒã¿ã¯éé¢ãããã¬ããªã«ã«å¯ŸããŠåäœãç¶ããŸãããã¢ããªã±ãŒã·ã§ã³ãç¹å®ã®æäœã§å€ãèªã¿èŸŒã¿ã蚱容ã§ããå Žåã¯ã匷æŽåæ§ã®ããèªã¿èŸŒã¿ã 500 ãšã©ãŒãè¿ãããšãã« ConsistentRead=True ãã ConsistentRead=False ã«åãæ¿ãããã©ãŒã«ããã¯ã®å®è£
ãæ€èšããŠãã ãããããã«ããããã©ãã£ãã¯ãå¥å
šãªãªãŒãžã§ã³ã«ã·ããããŠããéããªãŒãžã§ã³é害äžã«éšåçãªå¯çšæ§ãç¶æã§ããŸãã ãã§ã€ã«ãªãŒããŒã¡ã«ããºã ã¯äœåããã? Route 53 ãã«ã¹ãã§ãã¯ãŸãã¯ã«ã¹ã¿ã ãã§ã€ã«ãªãŒããŒã¡ã«ããºã ã䜿çšããŠããå Žåã¯ãæåã®èªçºãšã©ãŒãããã©ãã£ãã¯ã·ãããŸã§ã®æéãæž¬å®ããŸãããããç®æšåŸ©æ§æé (RTO) ãšæ¯èŒããŸãã ã¢ã©ãŒãã¯çºç«ããã? ãšã©ãŒã¡ããªã¯ã¹ã«å¯Ÿãã CloudWatch ã¢ã©ãŒã ãããå Žåã¯ãæåŸ
ãããã¿ã€ã ãã¬ãŒã å
ã«ããªã¬ãŒãããããšã確èªããŸããMRSC ããŒãã«ã§ã¯ãAWS FIS ã«ãã£ãŠèªçºããããšã©ãŒã¯ FaultInjectionServiceInducedErrors ã¡ããªã¯ã¹ãš SystemErrors ã¡ããªã¯ã¹ã®äž¡æ¹ãåãã«ãŠã³ãã§ã€ã³ã¯ãªã¡ã³ãããããšã«æ³šæããŠãã ãããããã¯ã SystemErrors ã«å¯Ÿããæ¢åã®ã¢ã©ãŒã ã AWS FIS å®éšäžã«çºç«ããããšãæå³ããŸããAWS FIS èªçºãšã©ãŒãšææ©çãªãšã©ãŒãåºå¥ãããå Žåã¯ã FaultInjectionServiceInducedErrors ã¡ããªã¯ã¹ã䜿çšããŸãããšã©ãŒãœãŒã¹ã«é¢ä¿ãªããšã³ããŠãŒã¶ãŒãžã®åœ±é¿ãææãããããã¢ããªã±ãŒã·ã§ã³ã¬ãã«ã®ãšã©ãŒçã«ãã¢ã©ãŒã ãäœæããããšãæ€èšããŠãã ããã 埩æ§ã¯ã¯ãªãŒã³ã«å®äºããã? å®éšåŸãã¢ããªã±ãŒã·ã§ã³ãæåä»å
¥ãªãã§éåžžéçšã«æ»ã£ãããšã確èªããŸããå®éšäžã®æžã蟌ã¿ããçããå¯èœæ§ã®ããããŒã¿ã®äžæŽåã確èªããŸãã çµæãè©äŸ¡åºæºãšäžèŽããªãã£ãå Žåãããã¯æåããå®éšã§ããã客æ§ã«åœ±é¿ãäžããåã«ãã¬ãžãªãšã³ã¹ã®ã®ã£ãããç¹å®ããããšã«ãªããŸãã ãã¹ããã©ã¯ãã£ã¹ DynamoDB ã°ããŒãã«ããŒãã«ã«å¯Ÿã㊠AWS FIS å®éšãå®è¡ããã客æ§ãšååããçµéšã«åºã¥ããŠã以äžãæšå¥šããŸãã å°ããå§ããŠãå埩ããèªååããã ãã¹ãç°å¢å
ã®åäžããŒãã«ã«å¯ŸããŠçãå®éš (10ã15 å) ããå§ããŸããèªä¿¡ãæ§ç¯ããã«ã€ããŠãæéãå¢ãããã¹ããŒãžã³ã°ç°å¢ã§ãã¹ãããæçµçã«ã¯ããŒã ã芳å¯ã§ããå¶æ¥æéäžã«æ¬çªç°å¢ã§å®éšãå®è¡ããŸããããŒã¹ã©ã€ã³ã確ç«ãããããå®éšãã³ãã¬ãŒããä¿åããŠå®æçã«å®è¡ããŸããå€ãã®ããŒã ã¯ãAWS FIS å®éšãææ¬¡ãŸãã¯ååæããšã®éçšæºåã¬ãã¥ãŒã«çµã¿èŸŒãã§ããŸããã¬ãžãªãšã³ã¹ã¯äžåºŠéãã®éæã§ã¯ãããŸãããã¢ããªã±ãŒã·ã§ã³ãé²åããã«ã€ããŠãç¶ç¶çãªæ€èšŒãå¿
èŠã§ãã å®éã®ãªã¹ã¯ã«åãããŠå®éšãèšèšããã éå»ã®ã€ã³ã·ãã³ããšãã¢ãã¹ã確èªããæåã«ãã¹ãããé害ã·ããªãªãåªå
ããŸããä»®å® (ããªãã©ã€ããžãã¯ã 500 ãåŠçããããšã«èªä¿¡ãããã) ã«ææŠããå®éšã¯ããã°ãã°æã䟡å€ã®ããçºèŠããããããŸããDynamoDB ã¬ããªã±ãŒã·ã§ã³ã®äžæåæ¢ãšãä»ã® AWS ãµãŒãã¹ã«å¯Ÿããä»ã® AWS FIS ã¢ã¯ã·ã§ã³ãçµã¿åãããŠãããçŸå®çãªé害ã·ããªãªããã¹ãããã·ããªãªããŒã¹ã®ãã¹ããæ€èšããŠãã ãããäŸãã°ã ã¯ãã¹ãªãŒãžã§ã³æ¥ç¶ã·ããªãª (Cross-Region Connectivity scenario) ã¯ãã¢ããªã±ãŒã·ã§ã³ãããã¯ãŒã¯ãã©ãã£ãã¯ãS3ãDynamoDBããã®ä»ã®ãµãŒãã¹ããããã¯ããŸãã ãã£ããã¢ãããå«ã埩æ§ãã§ãŒãºãç£èŠããã å®éšãçµäºãããããšãã£ãŠèгå¯ãæ¢ããªãã§ãã ãããDynamoDB ãã¬ããªã±ãŒã·ã§ã³ãåéããä¿çäžã®æžã蟌ã¿ãåŠçããå®éšåŸã®ãã£ããã¢ããæéã¯ãã¢ããªã±ãŒã·ã§ã³ã®åŸ©æ§ããžãã¯ã®åé¡ãæããã«ããå¯èœæ§ããããŸãã FaultInjectionServiceInducedErrors ã¡ããªã¯ã¹ã¯ããã®æéäžã®ãšã©ãŒã远跡ããŸããå®éšãæåãšå®£èšããåã«ãã¢ããªã±ãŒã·ã§ã³ãæåä»å
¥ãªãã§å®åžžç¶æ
ã«æ»ãããšã確èªããŸãã FIS å®éšãã®ã³ã°ãæå¹ã«ãã: å®éšãã®ã³ã° (experiment logging) ãæå¹ã«ããŠãå®éšã®å®è¡äžã«è©³çŽ°ãªæ
å ±ãåéããŸããå®éšãã®ã³ã°ã¯ããã©ã«ãã§ç¡å¹ã«ãªã£ãŠããããšã«æ³šæããŠãã ããã ãŸãšã ãã®ã·ãªãŒãºã® ããŒã 1 ãš ããŒã 2 ã§ã¯ãDynamoDB ã°ããŒãã«ããŒãã«ã䜿çšãããªãŒãžã§ãã«ã¬ãžãªãšã³ã¹åãã®ã¢ããªã±ãŒã·ã§ã³èšèšã«ã€ããŠåãäžããŸããããã®æçš¿ã§ã¯ã AWS Fault Injection Service ã䜿çšããŠãã®èšèšãæ€èšŒããæ¹æ³ã«ã€ããŠèª¬æããŸãããMRSC ããŒãã«ã®å ŽåãFIS ã¯å®éã®é害æã«ã¢ããªã±ãŒã·ã§ã³ãèŠãã®ãšåã HTTP 500 ãšã©ãŒãçæããŸããMREC ããŒãã«ã®å ŽåãFIS ã¯ããã¯ã°ã©ãŠã³ãã¬ããªã±ãŒã·ã§ã³ãäžæåæ¢ããŸããåäžã®ããŒãã«ãšçãæéããå§ããŠãèªä¿¡ãåŸãã«ã€ããŠç¯å²ãæ¡å€§ããããšããå§ãããŸããå®éšãå®å
šã«ã¹ã³ãŒãããããšã«é¢ãã詳现㯠FIS èšç»ã¬ã€ã ãããããªã詳现㯠DynamoDB éçºè
ã¬ã€ãã® é害泚å
¥ãã¹ã (Fault Injection Testing) ãåç
§ããŠãã ããã æ¬èšäºã¯ 2026 幎 05 æ 20 æ¥ ã«å
¬éããã âBest practices for Amazon DynamoDB Global Tables â Part 3: Validating regional resilience with AWS Fault Injection Serviceâ ã翻蚳ãããã®ã§ãã åæ: https://aws.amazon.com/blogs/database/best-practices-for-amazon-dynamodb-global-tables-part-3-validating-regional-resilience-with-aws-fault-injection-service/ èè
ã«ã€ã㊠Lee Hannigan Lee ã¯ã¢ã€ã«ã©ã³ããæ ç¹ãšãã AWS DynamoDB ããŒã ã®ã·ãã¢ããŒã¿ããŒã¹ãšã³ãžãã¢ã§ããããŒã¿ã¢ããªã³ã°ã忣ã·ã¹ãã ãéçºè
ããŒãªã³ã°ã«ããã 7 幎ã®çµéšãæã¡ãå€§èŠæš¡ã«æ§ç¯ããã客æ§ã«ãšã£ãŠ DynamoDB ãããã¢ã¯ã»ã¹ããããããããšã«çŠç¹ãåœãŠãŠããŸãã Shiladitya Mandal Shiladitya ã¯ãAWS DynamoDB ããŒã ã®ãœãããŠã§ã¢éçºãããŒãžã£ãŒã§ãããŒã¿ã ãŒãã¡ã³ãã°ã«ãŒããçããŠããŸããAmazon ã« 10 幎以äžåšç±ãã忣ã·ã¹ãã ã®æ§ç¯ãšã¹ã±ãŒãªã³ã°ã«åãçµãã§ããŸãããã·ã¢ãã«ãæ ç¹ãšãã Shiladitya ã¯ãã°ããŒãã«ããŒãã«ããã®ä»ã®ã¯ãã¹ãªãŒãžã§ã³æ©èœãæ¯ãã DynamoDB ã®ã¬ããªã±ãŒã·ã§ã³ããã³ããŒã¿ã ãŒãã¡ã³ãæ©èœã«çŠç¹ãåœãŠãŠããŸãã