æ¬ããã°ã¯ â Capacity-aware inference: Automatic instance fallback for SageMaker AI endpoints â ã翻蚳ãããã®ã§ãã çµç¹ãæ¬çªç°å¢ã§çæ AI ã¯ãŒã¯ããŒããã¹ã±ãŒã«ãããŠããäžã§ãä¿¡é Œæ§ã®é«ã GPU ã³ã³ãã¥ãŒãã確ä¿ããããšã¯ãæãæ ¹åŒ·ãéçšäžã®èª²é¡ã® 1 ã€ã«ãªã£ãŠããŸããå€§èŠæš¡èšèªã¢ãã« (LLM) ããã«ãã¢ãŒãã«ã¢ãŒããã¯ãã£ã¯ç¹å®ã®ã€ã³ã¹ã¿ã³ã¹ã¿ã€ããå¿
èŠãšãããã®ãã£ãã·ãã£ãå©çšã§ããªãå Žåããšã³ããã€ã³ã㯠1 ä»¶ã®ãªã¯ãšã¹ããåŠçããåã«å€±æããŠããŸããŸãã Amazon SageMaker AI ã§ãªã¢ã«ã¿ã€ã æšè«ãšã³ããã€ã³ããæ§ç¯ããéã¯ããããŸã§äœææã«åäžã®ã€ã³ã¹ã¿ã³ã¹ã¿ã€ããæå®ããå¿
èŠããããŸããããã®ã€ã³ã¹ã¿ã³ã¹ã¿ã€ãã®ãã£ãã·ãã£ãäžè¶³ããŠãããšããšã³ããã€ã³ãã¯å®è¡ç¶æ
ã«å°éã§ããŸãããèšå®ãæŽæ°ããå¥ã®ã€ã³ã¹ã¿ã³ã¹ã¿ã€ããéžæããŠå詊è¡ããããšãããµã€ã¯ã«ãããããžã§ãã³ã°ãæåãããŸã§ç¹°ãè¿ãããšã«ãªããŸãã æ¬æ¥ãAmazon SageMaker AI ã¯ãæ°èŠããã³æ¢åã®æšè«ãšã³ããã€ã³ãåãã« ãã£ãã·ãã£å¯Ÿå¿ã€ã³ã¹ã¿ã³ã¹ããŒã« ãå°å
¥ããŸããåªå
é äœä»ãã®ã€ã³ã¹ã¿ã³ã¹ã¿ã€ããªã¹ããå®çŸ©ããŠããã°ãSageMaker AI ã¯ãã£ãã·ãã£ãå¶çŽãããŠãããšã (äœææãã¹ã±ãŒã«ã¢ãŠãæãã¹ã±ãŒã«ã€ã³æ) ã«ãèªåçã«ãã®ãªã¹ããé ã«åŠçããŸãããšã³ããã€ã³ãã¯æåã®ä»å
¥ãªãã§ãå©çšå¯èœãª AI ã€ã³ãã©äžã«ããããžã§ãã³ã°ãããŸãããã®æ©èœã¯ãã·ã³ã°ã«ã¢ãã«ãšã³ããã€ã³ããæšè«ã³ã³ããŒãã³ãããŒã¹ã®ãšã³ããã€ã³ããéåææšè«ãšã³ããã€ã³ãã§å©çšã§ããŸãã æ¬èšäºã§ã¯ãã€ã³ã¹ã¿ã³ã¹ããŒã«ã®ä»çµã¿ãšãæ°èŠãšã³ããã€ã³ãã®äœæãæ¢åãšã³ããã€ã³ãã®ç§»è¡ãå«ãã䜿ãå§ãæ¹ã«ã€ããŠèª¬æããŸãã ãããŸã§ã®èª²é¡ SageMaker AI æšè«ãšã³ããã€ã³ã (ãªã¢ã«ã¿ã€ã ãŸãã¯éåæ) ã«ã¢ãã«ããããã€ãããšããåäžã®ã€ã³ã¹ã¿ã³ã¹ã¿ã€ããæå®ããŸãããã®ã€ã³ã¹ã¿ã³ã¹ã¿ã€ãã®å©çšå¯èœãªãã£ãã·ãã£ããªãå Žåããšã³ããã€ã³ãã®äœæã¯å€±æããŸãããã®å¶çŽã¯ããšã³ããã€ã³ãã®ã©ã€ããµã€ã¯ã«ã®ããããæ®µéã§çŸããŸãã ãã£ãã·ãã£äžè¶³ã«ãããšã³ããã€ã³ãäœæå€±æïŒ åžæããã€ã³ã¹ã¿ã³ã¹ã¿ã€ããå©çšã§ããªãå ŽåãSageMaker AI 㯠Insufficient Capacity ãšã©ãŒãè¿ããŸãããšã³ããã€ã³ãã皌åç¶æ
ã«ããã«ã¯ä»£æ¿ã®ã€ã³ã¹ã¿ã³ã¹ã¿ã€ããæåã§è©Šè¡ãç¶ããå¿
èŠããããçµæãããããŸã§æ¯åããªãã®æéãèŠããŸãã Auto Scaling ãããªãŒããæ¡åŒµã§ããªãïŒ ã¹ã±ãŒã«ã¢ãŠãã€ãã³ããçºçããã€ã³ã¹ã¿ã³ã¹ã¿ã€ãã«ãã£ãã·ãã£ãäžè¶³ããŠããå ŽåãAuto Scaling ã¯åãã€ã³ã¹ã¿ã³ã¹ã¿ã€ããééãªããªãã©ã€ããŸãããã©ãã£ãã¯ãå¢å ãç¶ããäžæ¹ã§ããšã³ããã€ã³ãã®ãµã€ãºã¯çŸç¶ã®ãŸãŸã§ãã ã¹ã±ãŒã«ããŠã³ã«åªå
é äœã®æŠå¿µããªãïŒ åäžã®ã€ã³ã¹ã¿ã³ã¹ã¿ã€ãã§ã¯ãåªå
(preferred) ãšãã©ãŒã«ãã㯠(fallback) ã®ããŒããŠã§ã¢ãšããæŠå¿µããããŸããããã¹ãŠã®ã€ã³ã¹ã¿ã³ã¹ãåºå¥ãªãåé€åè£ãšãªããŸãã ãªãã¶ãŒãããªãã£ãéçŽãããŠããŸãã察åŠã«ã€ãªãã«ããïŒ Amazon CloudWatch ã¡ããªã¯ã¹ã¯ãšã³ããã€ã³ãã¬ãã«ã§éçŽãããŸããã¬ã€ãã³ã·ããã£ãã·ãã£ã®åé¡ã調æ»ããéãã¡ããªã¯ã¹ã¯ãäœãããããããããšã¯ç€ºããŠãããã©ã®ã€ã³ã¹ã¿ã³ã¹ã¿ã€ããåå ãããŸã§ã¯ç€ºããŠãããŸããã åªå
é äœããŒã¹ã®ã€ã³ã¹ã¿ã³ã¹ããŒã«ã®ä»çµã¿ ãšã³ããã€ã³ãèšå®ã®äžã§ã instance pools ãšåŒã°ããã€ã³ã¹ã¿ã³ã¹ã¿ã€ãã®åªå
é äœä»ããªã¹ããå®çŸ©ããŸããSageMaker AI ã¯ãã£ãã·ãã£ãå¶çŽããããšãã«ãèªåçã«ãã®ãªã¹ããé çªã«åŠçããŸãã ãšã³ããã€ã³ããç«ã¡äžããïŒ SageMaker AI ã¯æåã®éžæè¢ã®ã€ã³ã¹ã¿ã³ã¹ã¿ã€ãã詊ããŸãããã£ãã·ãã£ãå©çšã§ããªãå Žåããã ã¡ã« 2 çªç®ã®éžæè¢ã次㫠3 çªç®ã®éžæè¢ã詊ããŸããæåã§ãªãã©ã€ããå¿
èŠã¯ãããŸããããšã³ããã€ã³ãã¯æ°å以å
ã«ãæåã«å©çšå¯èœãª AI ã€ã³ãã©äžã§ InService ã«å°éããŸãã ãšã³ããã€ã³ãã皌åãç¶ããïŒ Auto Scaling ãããªã¬ãŒãããåªå
ããã€ã³ã¹ã¿ã³ã¹ã¿ã€ããå¶çŽãããŠããå ŽåãSageMaker AI ã¯åªå
é äœãªã¹ãã®æ¬¡ã«å©çšå¯èœãªã€ã³ã¹ã¿ã³ã¹ã¿ã€ãã§ã¹ã±ãŒã«ã¢ãŠãããããããã©ãã£ãã¯ãæµãç¶ããŸãã ããªãŒãã¯åªå
ããŒããŠã§ã¢ã«åæããåŸåãæã€ïŒ ã¹ã±ãŒã«ã€ã³æãSageMaker AI ã¯æãåªå
床ã®äœã (ãã©ãŒã«ããã¯) ã€ã³ã¹ã¿ã³ã¹ããå
ã«åé€ããŸãããã®åŸã®ã¹ã±ãŒã«ã¢ãŠãã€ãã³ãã§ã¯ãåã³æãåªå
床ã®é«ãã¿ã€ããã詊è¡ããŸããåªå
ããããŒããŠã§ã¢ãå©çšå¯èœã«ãªãã«ã€ããŠãããªãŒãã¯æéãšãšãã«èªç¶ã«ãã¡ããžæ»ããæåã®ä»å
¥ã¯å¿
èŠãããŸããã ãã¹ãŠãå¯èŠåã§ããïŒ æ¢åã®ãã¹ãŠã® CloudWatch ã¡ããªã¯ã¹ã« InstanceType ãã£ã¡ã³ã·ã§ã³ã远å ãããŠããããã1 ã€ã®ãšã³ããã€ã³ãå
ã®ã€ã³ã¹ã¿ã³ã¹ã¿ã€ãããšã«ãã¬ã€ãã³ã·ãã¹ã«ãŒããããGPU 䜿çšçãã€ã³ã¹ã¿ã³ã¹æ°ã远跡ã§ããŸãã 詳现ã«ã€ããŠã¯ã Amazon SageMaker AI ã®ããã¥ã¡ã³ã ãåç
§ãã GitHub ã®ãµã³ãã«ããŒããã㯠ã詊ããŠã¿ãŠãã ããã åã€ã³ã¹ã¿ã³ã¹ã¿ã€ãã«é©åãªã¢ãã«ãåœãŠã ãã©ãŒã«ããã¯å
ã®ã€ã³ã¹ã¿ã³ã¹ã¿ã€ãã¯ãGPU ã¡ã¢ãªãã³ã³ãã¥ãŒãæ§èœãã¢ãŒããã¯ãã£ãç°ãªãããšããããããŸããé«ã¡ã¢ãªã®ãã«ã GPU ã€ã³ã¹ã¿ã³ã¹åãã«æé©åãããã¢ãã«ããããå°ããªã·ã³ã°ã« GPU ã®ãã©ãŒã«ããã¯ã§å¿
ãããåäœãããšã¯éããŸãããããŒã«ãªã¹ãå
ã®åã€ã³ã¹ã¿ã³ã¹ã¿ã€ãããæ£ããæ§æãããã¢ãã«ãšãããã³ã°ãããæ¹æ³ã¯ 2 ã€ãããŸãã ãªãã·ã§ã³ 1: èªåã§æé©åããã¢ãã«ãæã¡èŸŒã ã¿ãŒã²ããã®ã€ã³ã¹ã¿ã³ã¹ã¿ã€ãããã§ã«åãã£ãŠããå Žåãããããã«å¯ŸããŠã¢ãã«ã¢ãŒãã£ãã¡ã¯ããæºåããŸãããã©ã€ããªã®ãã€ãšã³ãã€ã³ã¹ã¿ã³ã¹ã§ã¯ãè€æ° GPU ã«ããããã³ãœã«äžŠåã䜿ããããããŸãããäžäœã®ãã©ãŒã«ããã¯ã§ã¯ãæšè«ãé«éåããããã«ææ©çãã³ãŒãã£ã³ã° (speculative decoding) ãé©çšããããšãèããããŸããæãåªå
床ã®äœããã©ãŒã«ããã¯ã§ã¯ãã¡ã¢ãªäºç®ã«åããããã« INT4 éååã䜿ããããããŸããã åæ§æã«ã€ããŠåå¥ã® SageMaker AI ã¢ãã«ãäœæããããããã® InstancePools ãšã³ã㪠(Single Model Endpoint ã®å Žå) ãŸãã¯ã€ã³ã¹ã¿ã³ã¹ã¿ã€ãããšã® Specifications (InferenceComponent ããŒã¹ã®ãšã³ããã€ã³ãã®å Žå) ã§ã ModelNameOverride ã䜿ã£ãŠåç
§ããŸããSageMaker AI ãåªå
床ã®äœãããŒã«ã«ãã©ãŒã«ããã¯ãããšããã®ããŒããŠã§ã¢çšã«æºåããã¢ãã«ããããã€ãããŸãã ãªãã·ã§ã³ 2: SageMaker AI æšè«ã¬ã³ã¡ã³ããŒã·ã§ã³æ©èœã䜿ã åããŒããŠã§ã¢ã¿ãŒã²ãããæåã§æé©åããã®ãæéãªå Žåã¯ã SageMaker AI æšè«ã¬ã³ã¡ã³ããŒã·ã§ã³ ã«ãã£ãŠããŒããŠã§ã¢åºæã®æ§æãçæã§ããŸããããŒã¹ã¢ãã«ãäžãããšãææ©çãã³ãŒãã£ã³ã°ãéååãªã©ã®æè¡ã䜿ã£ãŠãã¿ãŒã²ããã®ã€ã³ã¹ã¿ã³ã¹ã¿ã€ãå
šäœã«ãããæé©åãããæ§æã SageMaker AI ãçæããŸãã ã¬ã³ã¡ã³ããŒã·ã§ã³ãžã§ãã¯ã¿ãŒã²ããã®ã€ã³ã¹ã¿ã³ã¹ã¿ã€ãããšã« 1 ã€ã®çµæãè¿ããŸããåçµæã«ã¯ AIRecommendationModelDetails ã®ã¬ã¹ãã³ã¹å
ã« ModelPackageArn ãš InferenceSpecificationName ãå«ãŸããŠãããããããç¹å®ã®ããŒããŠã§ã¢åãã®æ§æã瀺ããŠããŸããäž¡æ¹ã®ãã£ãŒã«ãã䜿ã£ãŠçµæããšã« 1 ã€ã® SageMaker AI ã¢ãã«ãäœæãã察å¿ããããŒã«ãšã³ããªã§ ModelNameOverride ãä»ããŠåç
§ããŸããããã¯ãªãã·ã§ã³ 1 ãšåããã¿ãŒã³ã§ãããæé©åäœæ¥ã¯ãµãŒãã¹åŽãåŠçããŸãã MODEL_PACKAGE_ARN = "arn:aws:sagemaker:us-west-2:123456789012:model-package/MyModelPkgGroup/1" # AIRecommendationModelDetails ã®äž¡ãã£ãŒã«ãã䜿ã£ãŠã€ã³ã¹ã¿ã³ã¹ã¿ã€ãããšã« 1 ã€ã®ã¢ãã«ãäœæã sm.create_model( ModelName="my-llm-for-p5", PrimaryContainer={ "ModelPackageName": MODEL_PACKAGE_ARN, "InferenceSpecificationName": "p5-48xlarge-optimized", }, ExecutionRoleArn="arn:aws:iam::123456789012:role/SageMakerRole", ) sm.create_model( ModelName="my-llm-for-g6", PrimaryContainer={ "ModelPackageName": MODEL_PACKAGE_ARN, "InferenceSpecificationName": "g6-48xlarge-optimized", }, ExecutionRoleArn="arn:aws:iam::123456789012:role/SageMakerRole", ) # ãã®åŸãåŸè¿°ã®ãã»ããã¢ãããã®ãšãããããŒã«ãšã³ããªããšã« ModelNameOverride ã§åã¢ãã«ãåç
§ããã æ··åšããªãŒãã§ã® Auto Scaling Auto Scaling ã¯ãäœææã«å®çŸ©ããã®ãšåãåªå
é äœããžãã¯ã«åŸããŸããã¹ã±ãŒã«ã¢ãŠãã¯ãŸãæãåªå
床ã®é«ãããŒã«ã詊ãããã£ãã·ãã£ãå©çšã§ããªãå Žåã¯æ¬¡ã®ããŒã«ã«ãã©ãŒã«ããã¯ããŸããã¹ã±ãŒã«ã€ã³ã¯æãåªå
床ã®äœãã€ã³ã¹ã¿ã³ã¹ããå
ã«åé€ããããªãŒããçž®å°ããŠãåªå
ããããŒããŠã§ã¢ãæž©åããŸãã å éã¹ã±ãŒãªã³ã°ã¡ããªã¯ã¹ãæ§ç¯ãã ããªãŒãã«ã¯ç°ãªãã¹ã«ãŒããããã£ãã·ãã£ãæã€ã€ã³ã¹ã¿ã³ã¹ã¿ã€ããå«ãŸããŠãããããããã©ã«ãã®éçŽã¡ããªã¯ã¹ã¯å®éã®äœ¿çšç¶æ³ãæ£ãã衚çŸã§ããªãããšããããŸããããšãã° p5 ã€ã³ã¹ã¿ã³ã¹ã 18 ä»¶ã®åæãªã¯ãšã¹ããåŠçããg6 ã 7 ä»¶åŠçããŠãããšãããããã®çã®æ°å€ãå¹³åã㊠12.5 ã«ããŠããã©ã¡ãã®ã€ã³ã¹ã¿ã³ã¹ã¿ã€ãã®è² è·ãæ£ç¢ºã«ã¯åæ ãããŸããã CloudWatch ã®ã¡ããªã¯ã¹èšç® (metric math) ã䜿ããšãã¿ã€ãããšã®äœ¿çšç( per-type utilization ratios )ã«åºã¥ããå éã¡ããªã¯ã¹ãæ§ç¯ã§ããŸããåé
ã¯ãã®ã¿ã€ãã§èŠ³æž¬ããã䞊åå®è¡æ°ãæå€§ãã£ãã·ãã£ã§å²ã£ãŠã0.0 ã 1.0 ã®å€ãçæããŸãããããã®æ¯çãå¹³åããããšã§ã TargetValue ãšåã 0.0 ã 1.0 ã®ã¹ã±ãŒã«ã§ããªãŒãã¬ãã«ã®äœ¿çšã·ã°ãã«ãåŸãããŸãã TargetValue ã 0.7 ã«èšå®ãããšããããªãŒãå
ã®ãã¹ãŠã®ã€ã³ã¹ã¿ã³ã¹ã¿ã€ãã«ãããå éå¹³åããã£ãã·ãã£ã® 70% ãè¶
ãããã¹ã±ãŒã«ã¢ãŠãããããšããæå³ã«ãªããŸãã aas = boto3.client("application-autoscaling") aas.put_scaling_policy( PolicyName="weighted-utilization-scaling", ServiceNamespace="sagemaker", ResourceId="endpoint/my-heterog-endpoint/variant/primary", ScalableDimension="sagemaker:variant:DesiredInstanceCount", PolicyType="TargetTrackingScaling", TargetTrackingScalingPolicyConfiguration={ "TargetValue": 0.7, # å éããªãŒã䜿çšçã 70% ãè¶
ãããã¹ã±ãŒã«ã¢ãŠã "CustomizedMetricSpecification": { "Metrics": [ { "Id": "p5_concurrency", "MetricStat": { "Metric": { "Namespace": "AWS/SageMaker", "MetricName": "ConcurrentRequestsPerModel", "Dimensions": [ {"Name": "EndpointName", "Value": "my-heterog-endpoint"}, {"Name": "VariantName", "Value": "primary"}, {"Name": "InstanceType", "Value": "ml.p5.48xlarge"}, ], }, "Stat": "Average", }, "ReturnData": False, }, { "Id": "g6_concurrency", "MetricStat": { "Metric": { "Namespace": "AWS/SageMaker", "MetricName": "ConcurrentRequestsPerModel", "Dimensions": [ {"Name": "EndpointName", "Value": "my-heterog-endpoint"}, {"Name": "VariantName", "Value": "primary"}, {"Name": "InstanceType", "Value": "ml.g6.48xlarge"}, ], }, "Stat": "Average", }, "ReturnData": False, }, { "Id": "weighted_utilization", # ã¿ã€ãããšã®äœ¿çšçæ¯ = èŠ³æž¬å€ / æå€§ãã£ãã·ãã£ããããå¹³åãã "Expression": "(p5_concurrency / 20 + g6_concurrency / 8) / 2", "ReturnData": True, }, ], }, }, ) ãã®åŒã® 20 ãš 8 ã¯ãåã€ã³ã¹ã¿ã³ã¹ã¿ã€ãã§æž¬å®ãããæå€§åæäžŠåæ°ã§ãããã®äŸã§ã¯ p5 ã¯æå€§ 20 ãªã¯ãšã¹ããg6 ã¯æå€§ 8 ãªã¯ãšã¹ããåŠçããŸãããããã®å€ã¯ãè² è·ãã¹ãã§ã䜿ãã®ã¢ãã«ã«ã€ããŠæž¬å®ããæå€§å€ã«çœ®ãæããŠãã ãããæ¬¡ã®è¡šã¯ããã©ãã£ãã¯ã¬ãã«ããšã«ãã®ã¡ããªã¯ã¹ãã©ãåå¿ãããã瀺ããŠããŸãã ãã©ãã£ãã¯ã¬ãã« p5 ãªã¯ãšã¹ã g6 ãªã¯ãšã¹ã å é䜿çšç ã¢ã¯ã·ã§ã³ äœ 5 2 (0.25 + 0.25) / 2 = 0.25 ã¹ã±ãŒã«ã€ã³ äž 12 5 (0.60 + 0.63) / 2 = 0.61 ç¶æ é« 18 7 (0.90 + 0.88) / 2 = 0.89 ã¹ã±ãŒã«ã¢ãŠã ã¿ãŒã²ããä»è¿ 14 6 (0.70 + 0.75) / 2 = 0.73 ã¿ãŒã²ããä»è¿ â ç¶æ 泚 : ãã¹ãŠã®ã€ã³ã¹ã¿ã³ã¹ã¿ã€ãã®ã¹ã«ãŒããããã£ãã·ãã£ãåçšåºŠã®ã¯ãŒã¯ããŒãã§ã¯ãæ¢åã®ã¹ã±ãŒãªã³ã°ããªã·ãŒã倿Žããã«ãã®ãŸãŸäœ¿çšã§ããŸããå é䜿çšçã¡ããªã¯ã¹ã¯ãããŒã«ã¡ã³ããŒã® GPU ãã£ãã·ãã£ã倧ããç°ãªãå Žåã«æã䟡å€ãçºæ®ããŸãã ããªãŒããã¢ãã¿ãªã³ã°ãã æ¢åã®ãã¹ãŠã® CloudWatch ã¡ããªã¯ã¹ã« InstanceType ãšããæ°ãããã£ã¡ã³ã·ã§ã³ã远å ãããŸããã ModelLatency ã ConcurrentRequestsPerModel ã GPUUtilization ã InstanceCount ã InvocationsPerInstance ãã1 ã€ã®ãšã³ããã€ã³ãå
ã®ããŒããŠã§ã¢ã¿ã€ãããšã«åè§£ã§ããŸããåã€ã³ã¹ã¿ã³ã¹ã¿ã€ããç¬ç«ããŠè¿œè·¡ããããã·ã¥ããŒããã¢ã©ãŒã ãæ§ç¯ã§ããŸãã DescribeEndpoint ã¯ããŒã«ããšã®çŸåšã®ã€ã³ã¹ã¿ã³ã¹æ°ãè¿ããããããªãŒãã®æ§æãåžžã«ææ¡ã§ããŸãã response = sm.describe_endpoint(EndpointName="my-heterog-endpoint") pools = response["ProductionVariants"][0]["InstancePools"] # åºåäŸ: # [ # {"InstanceType": "ml.p5.48xlarge", "CurrentInstanceCount": 4}, # {"InstanceType": "ml.g6.48xlarge", "CurrentInstanceCount": 2}, # ] ãã©ãã£ãã¯ã«ãŒãã£ã³ã° ã€ã³ã¹ã¿ã³ã¹ããŒã«ã䜿ããšã³ããã€ã³ãã§ã¯ã ProductionVariant ã® RoutingConfig ãèšå®ããããšã§ãLeast Outstanding Requests (LOR) ã«ãŒãã£ã³ã°ãæå¹åããããšãæšå¥šããŸããLOR ã¯åä¿¡ãªã¯ãšã¹ãããšã«ãã¢ãã«ã³ããŒãããåŠçäžã®ãªã¯ãšã¹ããæãå°ãªãã€ã³ã¹ã¿ã³ã¹ãžã«ãŒãã£ã³ã°ããŸãããã£ãã·ãã£ã®å€§ããã€ã³ã¹ã¿ã³ã¹ã¯ãªã¯ãšã¹ããé«éã«åŠçãããããã¥ãŒãéããã«è§£æ¶ãããå®åžžç¶æ
ã§ã¯åŠçäžã®ãªã¯ãšã¹ãæ°ãå°ãªãä¿ãããŸããããã«ãããæåã®éã¿ä»ãèšå®ãªãã§ããã£ãã·ãã£ã®å€§ããã€ã³ã¹ã¿ã³ã¹ã¯èªç¶ã«å€ãã®ãã©ãã£ãã¯ãåãåãããã«ãªããŸãã "RoutingConfig": {"RoutingStrategy": "LEAST_OUTSTANDING_REQUESTS"} ãã®èšå®ããªãå Žåããšã³ããã€ã³ãã¯ããã©ã«ãã§ RANDOM ã«ãŒãã£ã³ã°ã䜿çšããã€ã³ã¹ã¿ã³ã¹ã®è² è·ã«é¢ä¿ãªããªã¯ãšã¹ããåçã«åæ£ããŸããããŒã«ã¡ã³ããŒéã§ã¹ã«ãŒããããã£ãã·ãã£ã倧ããç°ãªãå Žåãããã¯æé©ã§ã¯ãããŸããã詳现ã«ã€ããŠã¯ã ProductionVariant API ãªãã¡ã¬ã³ã¹ã® RoutingConfig ãåç
§ããŠãã ããã æŽæ°ãšããŒã«ãã㯠ã€ã³ã¹ã¿ã³ã¹ããŒã«ã¯ãBlue/Green ããã〠㚠ããŒãªã³ã°ããã〠ã®äž¡æ¹ããµããŒãããŠããŸãã Blue/Green ããã〠ã§ã¯ããã©ãã£ãã¯ãåãæ¿ããåã«ãåãåªå
é äœããŒã¹ã®ãã©ãŒã«ããã¯ããžãã¯ã䜿ã£ãŠãæ°ãã (ã°ãªãŒã³) ããªãŒãå
šäœãããããžã§ãã³ã°ããŸãããã«ã¹ãã§ãã¯ããã¹ããããã©ãã£ãã¯ãã«ãããªãŒããŒãããŸãã倱æããå Žåã¯èªåããŒã«ããã¯ã«ãã£ãŠãã«ãŒããªãŒããä¿æããããšã³ããã€ã³ãã¯çµå§ InService ãç¶æããŸãã ããŒãªã³ã°ããã〠ã§ã¯ãèšå®å¯èœãªããã (äžåºŠã« 5ã50% ã®ã€ã³ã¹ã¿ã³ã¹) ã§ããªãŒããæŽæ°ããŸããBlue/Green å
šäœã»ã©ã®è¿œå ãã£ãã·ãã£ãå¿
èŠãšããªããããç¹ã«å€§èŠæš¡ã¢ãã«ãéèŠã®é«ã GPU ã€ã³ã¹ã¿ã³ã¹ã¿ã€ãã§äŸ¡å€ããããŸããSageMaker AI ã¯æ°ããåããããããããžã§ãã³ã°ããéã«ãåªå
é äœããŒã¹ã®ãã©ãŒã«ããã¯ããžãã¯ãé©çšããŸããããŒãã³ã°æéäžã« CloudWatch ã®ã¢ã©ãŒã ãçºç«ããå Žåããã©ãã£ãã¯ã¯èªåçã«ããŒã«ããã¯ãããŸããèšå®ã®è©³çŽ°ã¯ Use rolling deployments ãåç
§ããŠãã ããã åææ¡ä»¶ éå§ããåã«ã以äžã確èªããŠãã ããã sagemaker:CreateEndpointConfig ã sagemaker:CreateEndpoint ã sagemaker:UpdateEndpoint ã® IAM æš©éãæã€ AWS ã¢ã«ãŠã³ã Amazon S3 ã«ã¢ãŒãã£ãã¡ã¯ããæã€å°ãªããšã 1 ã€ã® SageMaker ã¢ãã« Boto3 1.43.1 以é (Python SDK ã§ã® InstancePools ãµããŒãã®ãã) (ä»»æ) ã¿ãŒã²ããã®ã€ã³ã¹ã¿ã³ã¹ã¿ã€ãããšã«æé©åãããåå¥ã®ã¢ãã«ã¢ãŒãã£ãã¡ã¯ãããŸã㯠SageMaker AI æšè«ã¬ã³ã¡ã³ããŒã·ã§ã³ ããã® ModelPackage SageMaker AI æšè«ãšã³ããã€ã³ãã®ã€ã³ã¹ã¿ã³ã¹ããŒã«ãµããŒãã¯ããã¹ãŠã®åçš AWS ãªãŒãžã§ã³ã§å©çšå¯èœã§ãã AWS ãããžã¡ã³ãã³ã³ãœãŒã« ãAWS Command Line Interface (AWS CLI)ããŸã㯠AWS SDK ããå§ããããŸãã ã€ã³ã¹ã¿ã³ã¹ããŒã«ã§ãšã³ããã€ã³ããæ§æããã¯ãŒã¯ãã㌠ã€ã³ã¹ã¿ã³ã¹ããŒã«ãæ§æããæ¹æ³ã¯ 2 ã€ãããŸããAmazon SageMaker AI ã§æ°èŠãšã³ããã€ã³ããäœæããå Žåãšãæ¢åã®ãšã³ããã€ã³ããç§»è¡ããå Žåã® 2 éãã§ãã æ°èŠãšã³ããã€ã³ããäœæããå Žåã以äžã®å³ãã¯ãŒã¯ãããŒã説æããŸãã ã€ã³ã¹ã¿ã³ã¹ã¿ã€ããéžæããåªå
é äœãå²ãåœãŠã (1 ãæé«)ã åã€ã³ã¹ã¿ã³ã¹ã¿ã€ãåãã«æé©åããã¢ãã«ãæºåããããŸã㯠SageMaker AI æšè«ã¬ã³ã¡ã³ããŒã·ã§ã³ãå®è¡ããŠçæããã åªå
é äœãæã€ InstancePools ããªã¹ãåãããšã³ããã€ã³ãèšå®ãäœæããã ãšã³ããã€ã³ããäœæãããSageMaker AI ãèªåçã«ãã£ãã·ãã£ã®ç¢ºä¿ãåŠçããã æ°ãã InstanceType ãã£ã¡ã³ã·ã§ã³ã䜿ã£ãŠãã€ã³ã¹ã¿ã³ã¹ã¿ã€ãããšã® CloudWatch ã¢ãã¿ãªã³ã°ãèšå®ããã æ¢åã®ãšã³ããã€ã³ããç§»è¡ããå Žåã以äžã®å³ãã¯ãŒã¯ãããŒã説æããŸãã æ°ãããšã³ããã€ã³ãèšå®ãäœæããã InstanceType ã InstancePools ã«çœ®ãæããçŸåšã®ã€ã³ã¹ã¿ã³ã¹ã¿ã€ãã Priority: 1 ã«ä¿ã€ã UpdateEndpoint ãåŒã³åºãããšã³ããã€ã³ã㯠Blue/Green ç§»è¡äžã InService ãç¶æããã ãã©ãŒã«ããã¯å
ã®ã€ã³ã¹ã¿ã³ã¹ã¿ã€ãéã§ã¹ã«ãŒããããã£ãã·ãã£ã倧ããç°ãªãå Žåã¯ãå é䜿çšçã¹ã±ãŒãªã³ã°ã¡ããªã¯ã¹ãä»»æã§è¿œå ããã ã»ããã¢ãã ã€ã³ã¹ã¿ã³ã¹ããŒã«ã®å°å
¥ã¯ããšã³ããã€ã³ãèšå®ã® 1 ãã£ãŒã«ãã倿Žããã ãã§æžã¿ãŸãã ProductionVariant ã®åäžã® InstanceType ãã£ãŒã«ãã InstancePools ãªã¹ãã«çœ®ãæããŠãã ãããã¢ãã«ãã¹ã±ãŒãªã³ã°ããªã·ãŒãã¢ãã¿ãªã³ã°ããã·ã¥ããŒãã¯å€æŽãªãã§åäœãç¶ããŸãã æ¢åãšã³ããã€ã³ãã®ç§»è¡ ç§»è¡å: åäžã€ã³ã¹ã¿ã³ã¹ã¿ã€ã import boto3 sm = boto3.client("sagemaker") sm.create_endpoint_config( EndpointConfigName="my-config", ProductionVariants=[{ "VariantName": "primary", "ModelName": "my-llm", "InitialInstanceCount": 2, "InstanceType": "ml.g6e.48xlarge", # åäžã¿ã€ã â ãã£ãã·ãã£ãã©ãŒã«ããã¯ãªã }], ) ç§»è¡åŸ: åªå
é äœä»ãã€ã³ã¹ã¿ã³ã¹ããŒã« sm.create_endpoint_config( EndpointConfigName="my-config-v2", ProductionVariants=[{ "VariantName": "primary", "ModelName": "my-llm", "InitialInstanceCount": 2, "VariantInstanceProvisionTimeoutInSeconds": 1200, # åŸè¿°ã®æ³šãåç
§ "InstancePools": [ {"InstanceType": "ml.g6e.48xlarge", "Priority": 1}, # çŸåšã®ã¿ã€ã {"InstanceType": "ml.g6.48xlarge", "Priority": 2}, # åãã¡ããªãæåã®ãã©ãŒã«ãã㯠{"InstanceType": "ml.p4d.24xlarge", "Priority": 3}, # ããåºããã©ãŒã«ãã㯠], }], ) Blue/Green ç§»è¡äžããšã³ããã€ã³ã㯠InService ãç¶æããŸãã sm.update_endpoint( EndpointName="my-endpoint", EndpointConfigName="my-config-v2", ) 泚 : VariantInstanceProvisionTimeoutInSeconds ã¯ãã€ã³ã¹ã¿ã³ã¹ããŒã«ãµããŒãã§å°å
¥ãããæ°ãããã£ãŒã«ãã§ããããŒã«ããã€ã³ã¹ã¿ã³ã¹ã調éããå
šäœã®æéæ ãèšå®ããŸããSageMaker AI ã¯ãã®æéæ ã®äžã§ Insufficient Capacity ãšã©ãŒã«å¯ŸããŠãªãã©ã€ãç¶ããã¿ã€ã ã¢ãŠãåŸã«æ¬¡ã®ããŒã«ãžç§»ããŸããæå¹ãªç¯å²ã¯ 300 ã 3600 ç§ã§ããå€§èŠæš¡ GPU ã€ã³ã¹ã¿ã³ã¹ã¿ã€ãã§ã¯ã1200 ç§ã劥åœãªéå§å€ã§ãããã®ã¿ã€ããŒã¯ã€ã³ã¹ã¿ã³ã¹èª¿éã®ã¿ãã«ããŒããŸããã¢ãã«ããŠã³ããŒããšã³ã³ããèµ·åæéã¯ãæ¢åã® ModelDataDownloadTimeoutInSeconds ãš ContainerStartupHealthCheckTimeoutInSeconds ãã£ãŒã«ãã§åå¥ã«ç®¡çãããŸããã€ã³ã¹ã¿ã³ã¹ã¿ã€ãããšã«ç°ãªãæé©åæžã¿ã¢ãã«ããããã€ããã«ã¯ãä»»æã®ããŒã«ãšã³ããªã« ModelNameOverride ã远å ããŸããã¢ãã«èšå®ãªãã·ã§ã³ã¯åã®ã»ã¯ã·ã§ã³ã§ç¢ºèªã§ããŸãã InferenceComponent ããŒã¹ã®ãšã³ããã€ã³ã sm.create_inference_component( InferenceComponentName="my-ic", EndpointName="my-heterogeneous-endpoint", VariantName="primary", Specifications=[ { "InstanceType": "ml.p5.48xlarge", "ModelName": "my-model-p5-optimized", "ComputeResourceRequirements": { "NumberOfAcceleratorDevicesRequired": 8, "MinMemoryRequiredInMb": 65536, }, }, { "InstanceType": "ml.g6.48xlarge", "ModelName": "my-model-g6-optimized", "ComputeResourceRequirements": { "NumberOfAcceleratorDevicesRequired": 8, "MinMemoryRequiredInMb": 32768, }, }, ], RuntimeConfig={"CopyCount": 4}, ) éåææšè«ãšã³ããã€ã³ã éåææšè« ãšã³ããã€ã³ãã§ãã€ã³ã¹ã¿ã³ã¹ããŒã«ã¯åãããã«åäœããŸãã InstancePools ã®å®çŸ©ãšäžŠã¹ãŠ CreateEndpointConfig ã®åŒã³åºãã« AsyncInferenceConfig ãããã¯ã远å ããã ãã§ãåªå
é äœããŒã¹ã®ããããžã§ãã³ã°ãšãã©ãŒã«ããã¯ããžãã¯ããã®ãŸãŸé©çšãããŸããããã¯ãã€ã³ã¹ã¿ã³ã¹æ° 0 ãŸã§ã¹ã±ãŒã«ããŠã³ããéåæã¯ãŒã¯ããŒãã§ç¹ã«æçšã§ãããšã³ããã€ã³ããåã³ã¹ã±ãŒã«ã¢ããããŠãã¥ãŒã€ã³ã°ããããªã¯ãšã¹ããåŠçããéãSageMaker AI ã¯ãŸãæãåªå
床ã®é«ãå©çšå¯èœãªããŒã«ã䜿ã£ãŠããããžã§ãã³ã°ããæåä»å
¥ãªãã§èé害æ§ã®ããã³ãŒã«ãã¹ã¿ãŒãæåãæäŸããŸãã ãŸãšã Amazon SageMaker AI Instance Pools ã¯ãæšè«ãšã³ããã€ã³ãåãã«ã€ã³ã¹ã¿ã³ã¹ã¿ã€ãã®åªå
é äœä»ããªã¹ããå®çŸ©ããããšãå¯èœã«ããSageMaker AI ããã®é åºã«åºã¥ããŠãã£ãã·ãã£ãèªåçã«ç®¡çããŸãã ãšã³ããã€ã³ãäœææãã¹ã±ãŒã«ã¢ãŠãæãã¹ã±ãŒã«ã€ã³æã«ããã£ãŠãSageMaker AI ã¯åªå
ããã€ã³ã¹ã¿ã³ã¹ã¿ã€ããé ã«åŠçããããã第 1 åè£ã®ããŒããŠã§ã¢ãå©çšã§ããªããšãã§ããããã€ãæåã§ãªãã©ã€ããå¿
èŠããããŸãããå§ãæ¹ã¯ç°¡åã§ãããšã³ããã€ã³ãèšå®ã® InstanceType ã InstancePools ã«çœ®ãæã㊠UpdateEndpoint ãåŒã³åºããŸããæ¢åã®ã¢ãã«ãAuto Scaling ããªã·ãŒãã¢ãã¿ãªã³ã°ããã·ã¥ããŒãã¯å€§ããªå€æŽãªãã«åäœãç¶ããŸãã ã€ã³ã¹ã¿ã³ã¹ã¿ã€ãããšã® CloudWatch ã¡ããªã¯ã¹ãš DescribeEndpoint ããã®è©³çްãªããŒã«æ°ã«ãã£ãŠãã©ã®ã€ã³ã¹ã¿ã³ã¹ã¿ã€ããããªãŒããæ¯ããŠãããããªã¢ã«ã¿ã€ã ã«æç¢ºã«ææ¡ã§ããŸããã³ã¹ãæé©åãGPU ãã£ãã·ãã£å¶çŽãžã®å¯Ÿå¿ããŒãããã³ãŒã«ãã¹ã¿ãŒãå¯èœãªèé害æ§ã®é«ãéåæãã€ãã©ã€ã³ã®æ§ç¯ãªã©ãã©ã®ç®çã§ãã£ãŠããã€ã³ã¹ã¿ã³ã¹ããŒã«ã¯éçšè² è·ãæããªãã ML æšè«ãã¹ã ãŒãºã«åããç¶ããããã®æè»æ§ãšèªååãæäŸããŸãã ãã®æ©èœã¯æ¬æ¥ãã远å è²»çšãªãã§å©çšå¯èœã§ããå®éã«ããããžã§ãã³ã°ãããã€ã³ã¹ã¿ã³ã¹ã¿ã€ãã«å¯ŸããŠã¯ãæšæºã®åäžã¿ã€ããšã³ããã€ã³ããšåãæçã§èª²éãããŸãã詳现ã«ã€ããŠã¯ã Amazon SageMaker AI ã®ããã¥ã¡ã³ã ãš GitHub ã®ãµã³ãã«ããŒããã㯠ãåç
§ããŠãã ããã èè
ã«ã€ã㊠Kareem Syed-Mohammed Kareem Syed-Mohammed 㯠AWS ã®ãããã¯ããããŒãžã£ãŒã§ããSageMaker HyperPod äžã§ã®çæ AI ã¢ãã«éçºãšã¬ããã³ã¹ã®å®çŸã«æ³šåããŠããŸãããã以å㯠Amazon QuickSight ã§çµã¿èŸŒã¿åæãšéçºè
ãšã¯ã¹ããªãšã³ã¹ããªãŒãããŸãããQuickSight ã«å ããŠãAWS Marketplace ãš Amazon Retail ã§ããããã¯ããããŒãžã£ãŒãåããŠããŸãããKareem ã¯ã³ãŒã«ã»ã³ã¿ãŒæè¡ã®éçºè
ãšããŠãã£ãªã¢ãã¹ã¿ãŒãããExpedia ã® Local Expert ãšåºåãMcKinsey ã®çµå¶ã³ã³ãµã«ã¿ã³ããšããŠã®çµæŽãæã¡ãŸãã Dmitry Soldatkin Dmitry Soldatkin 㯠AWS ã® SageMaker Inference ã«ãããã¹ãã·ã£ãªã¹ããœãªã¥ãŒã·ã§ã³ã¢ãŒããã¯ãã£ã®ã¯ãŒã«ãã¯ã€ããªãŒããŒã§ãããšã³ã¿ãŒãã©ã€ãºå
šäœã«ãããçæ AI ããã³ AI/ML ãœãªã¥ãŒã·ã§ã³ã®èšèšãæ§ç¯ãæé©åãæ¯æŽããåãçµã¿ãçããŠããŸãã圌ã®ä»äºã¯å¹
åºã ML ãŠãŒã¹ã±ãŒã¹ã«åã³ãçæ AIããã£ãŒãã©ãŒãã³ã°ãå€§èŠæš¡ãª ML ã®ãããã€ã¡ã³ãã«éç¹ã眮ããŠããŸããéèãµãŒãã¹ãä¿éºãéä¿¡ãªã©ã®æ¥çã®äŒæ¥ãšåæ¥ããŠããŸãããDmitry ãšã¯ LinkedIn ã§ã€ãªããããšãã§ããŸãã Johna Liu Johna Liu 㯠Amazon SageMaker ããŒã ã®ãœãããŠã§ã¢éçºãšã³ãžãã¢ã§ããå¹çãé«ãæ°ããªæ©èœãå¯èœã«ãã AI/LLM é§åã®ããŒã«ãæ§ç¯ãæ¢æ±ããŠããŸããä»äºä»¥å€ã§ã¯ãããã¹ããã¹ã±ããããŒã«ãéçãæ¥œããã§ããŸãã Xu Deng Xu Deng 㯠SageMaker ããŒã ã®ãœãããŠã§ã¢ãšã³ãžãã¢ãªã³ã°ãããŒãžã£ãŒã§ããAmazon SageMaker äžã§ã客æ§ã® AI/ML æšè«äœéšã®æ§ç¯ãšæé©åãæ¯æŽããããšã«æ³šåããŠããŸããäœæã«ã¯æ
è¡ãšã¹ããŒããŒããæ¥œããã§ããŸãã Mona Mona Mona Mona ã¯çŸåšãAmazon ã§ã·ã㢠AI/ML ã¹ãã·ã£ãªã¹ããœãªã¥ãŒã·ã§ã³ã¢ãŒããã¯ããšããŠå€åããŠããŸãã以å㯠Google ã§ãªãŒãçæ AI ã¹ãã·ã£ãªã¹ããšããŠåããŠããŸããããNatural Language Processing with AWS AI Services: Derive strategic insights from unstructured data with Amazon Textract and Amazon Comprehendãããã³ãGoogle Cloud Certified Professional Machine Learning Study Guideãã® 2 åã®èè
ã§ãããAI/ML ãšã¯ã©ãŠãæè¡ã«é¢ãã 19 æ¬ã®ããã°ãå·çãCORD19 Neural Search ã«é¢ããç ç©¶è«æã®å
±èè
ãšããŠãæš©åšãã AAAI (Association for the Advancement of Artificial Intelligence) ã«ã³ãã¡ã¬ã³ã¹ã§ Best Research Paper è³ãåè³ããŠããŸãã 翻蚳㯠Solutions Architect çå±±æŽå¹³ ãæ
åœããŸãããåæã¯ ãã¡ã ã§ãã