ããã¹ãæ€çŽ¢ãš ã»ãã³ãã£ãã¯æ€çŽ¢ ãšã³ãžã³ã®å°é ã«ãããeã³ããŒã¹ãå°å£²æ¥ã¯æ¶è²»è
ã«ãšã£ãŠããç°¡åã«æ€çŽ¢ã§ããããã«ãªããŸãããæ€çŽ¢ããéã«ããã¹ããšç»åã®äž¡æ¹ãã¯ãšãªã«å«ãããšãã§ããæ€çŽ¢ãšã³ãžã³ã¯ãæ€çŽ¢ãœãªã¥ãŒã·ã§ã³ã®æè»æ§ãéåžžã«é«ããããšãã§ããŸããããšãã°ãã©ãããããã«äœçŸãã®å®¶æã®åçãå
¥ã£ããã©ã«ãããããããªããšããªãã®èŠªåãå€ãå®¶ã®ããŒã«ã®åã«ãããšãã«æ®ã£ãåçãããã«èŠã€ããããšããå Žåãä»®å®ããŸãããã®å ŽåããããŒã«ã®åã«äºäººãç«ã£ãŠãããã®ãããªäŒè©±åœ¢åŒã®æç« ãã¯ãšãªãšããŠå
¥åããããšã§ãããã¹ããšç»åã®çµ±åæ€çŽ¢ãšã³ãžã³ãããç®åœãŠã®ç»åãæ€çŽ¢ããããšãã§ããŸããç»åã¿ã€ãã«ã«é©åãªããŒã¯ãŒããå
¥åããªããŠããã¯ãšãªãå®è¡ã§ããŸãã çŸåšã Amazon OpenSearch Service 㯠k-NN ã€ã³ããã¯ã¹ã® ã³ãµã€ã³é¡äŒŒåºŠ ã¡ããªã¯ã¹ããµããŒãããŠããŸããã³ãµã€ã³é¡äŒŒåºŠã¯ã2 ã€ã®ãã¯ãã«éã®è§åºŠã®ã³ãµã€ã³ã枬å®ããŸããã³ãµã€ã³è§åºŠãå°ããã»ã©ããã¯ãã«éã®é¡äŒŒæ§ãé«ããªããŸããã³ãµã€ã³é¡äŒŒåºŠã䜿çšãããšã2 ã€ã®ãã¯ãã«éã®åããæž¬å®ã§ãããããç¹å®ã®ã»ãã³ãã£ãã¯æ€çŽ¢ã¢ããªã±ãŒã·ã§ã³ã«é©ããŠããŸãã Contrastive Language-Image Pre-Training (CLIP) ã¯ãããŸããŸãªç»åãšããã¹ãã®ãã¢ã§ãã¬ãŒãã³ã°ããããã¥ãŒã©ã«ãããã¯ãŒã¯ã§ããCLIP ãã¥ãŒã©ã«ãããã¯ãŒã¯ã¯ãç»åãšããã¹ãã®äž¡æ¹ãåã æœåšç©ºé ã«æåœ±ã§ãããããã³ãµã€ã³é¡äŒŒåºŠãªã©ã®é¡äŒŒåºŠå°ºåºŠã䜿çšããŠç»åãšããã¹ããæ¯èŒããããšãã§ããŸããCLIP ã䜿çšããŠè£œåã®ç»åã説æã Embedding ã« ãšã³ã³ãŒã ããOpenSearch Service k-NN ã€ã³ããã¯ã¹ã«ä¿åããããšãã§ããŸããããããã°ã顧客ã¯ã€ã³ããã¯ã¹ãã¯ãšãªããŠãé¢å¿ã®ããååãæ€çŽ¢ã§ããŸãã Amazon SageMaker ãçšããããšã§ãäžèšã® CLIP ããšã³ã³ãŒãã£ã³ã°ãããããã«å©çšããããšãã§ããŸãã Amazon SageMaker Serverless Inference ã¯ãæ©æ¢°åŠç¿ (ML) ã¢ãã«ã®ãããã€ãšã¹ã±ãŒãªã³ã°ã容æã«ããå°çšã®æšè«ãµãŒãã¹ã§ããSageMaker ã䜿çšãããšãéçºãšãã¹ãçšã«ãµãŒããŒã¬ã¹ããããã€ããæ¬çªç°å¢ã§ã¯ ãªã¢ã«ã¿ã€ã æšè« ã«ç§»è¡ããããšãå¯èœã§ããAmazon SageMaker Serveless Inference ã§ã¯ãã¢ã€ãã«æã«ã€ã³ãã©ã¹ãã©ã¯ãã£ã 0 ã«ã¹ã±ãŒã«ããŠã³ããããšã§ã³ã¹ããç¯çŽã§ããŸããããã¯ãéçºãµã€ã¯ã«éã®ã¢ã€ãã«æéãé·ã POC ãæ§ç¯ããå Žåã«æé©ã§ãããã®ä»ã Amazon SageMaker Batch transform ã䜿çšããŠãå€§èŠæš¡ãªããŒã¿ã»ããããæšè«ãååŸããããšãã§ããŸãã ãã®èšäºã§ã¯ãSageMakerãšOpenSearch Serviceã§CLIPã䜿çšããŠæ€çŽ¢ã¢ããªã±ãŒã·ã§ã³ãæ§ç¯ããæ¹æ³ã瀺ããŸããã³ãŒãã¯ãªãŒãã³ãœãŒã¹ã§ã ãã¡ãã®GitHub ã§ãã¹ããããŠããŸãã ãœãªã¥ãŒã·ã§ã³æŠèŠ Amazon OpenSearch Service ã¯ãããã¹ããããã³ã°ãšEmbedding k-NN æ€çŽ¢ãæäŸããŸãããã®ãœãªã¥ãŒã·ã§ã³ã§ã¯ Embedding k-NN æ€çŽ¢ãå©çšããŸããç»åãšããã¹ãã®äž¡æ¹ãã¯ãšãªãšããŠäœ¿çšããŠãã€ã³ãã³ããªããã¢ã€ãã ãæ€çŽ¢ã§ããŸãããã®çµ±åç»åããã³ããã¹ãæ€çŽ¢ã¢ããªã±ãŒã·ã§ã³ã®å®è£
ã¯ã次㮠2 ã€ã®ãã§ãŒãºã§æ§æãããŸãã k-NN åç
§ã€ã³ããã¯ã¹ â ãã®ãã§ãŒãºã§ã¯ãäžé£ã®ã³ãŒãã¹ããã¥ã¡ã³ããŸãã¯è£œåç»åãCLIPã¢ãã«ã«æž¡ããŠããããã Embedding ã«ãšã³ã³ãŒãããŸããããã¹ããšç»åã® Embedding ã¯ãããããã³ãŒãã¹ãŸãã¯ç»åã®æ°å€è¡šçŸã衚ããŸãããããã® Embedding ã OpenSearch Service ã® k-NN ã€ã³ããã¯ã¹ã«ä¿åããŸããk-NNãæ¯ããæŠå¿µã¯ãEmbedding 空éã«é¡äŒŒããããŒã¿ãã€ã³ããè¿æ¥ããŠååšãããšãããã®ã§ããããšãã°ããèµ€ãè±ããããã©ããšãã£ãããã¹ããšèµ€ããã©ã®ç»åã¯äŒŒãŠããããããããã®ããã¹ããšç»åã® Embedding 㯠Embedding 空éå
ã§äºãã«è¿æ¥ããŠããŸãã k-NN ã€ã³ããã¯ã¹ã¯ãšãª â ããã¯ã¢ããªã±ãŒã·ã§ã³ã®æšè«ãã§ãŒãºã§ãããã®ãã§ãŒãºã§ã¯ããã£ãŒãã©ãŒãã³ã°ã¢ãã« (CLIP) ãä»ããŠããã¹ãæ€çŽ¢ã¯ãšãªãŸãã¯ç»åæ€çŽ¢ã¯ãšãªãéä¿¡ããEmbedding ã«ãšã³ã³ãŒãããŸããæ¬¡ã«ããããã® Embedding ã䜿çšããŠãOpenSearch Service ã«ä¿åãããŠããåç
§ k-NN ã€ã³ããã¯ã¹ãã¯ãšãªããŸããk-NN ã€ã³ããã¯ã¹ã¯ãemnbedding 空éããåæ§ã® Embedding ãè¿ããŸããããšãã°ããèµ€ãè±ãã®ããã¹ããæž¡ããšãèµ€ããã©ã®ç»åã® Embedding ãé¡äŒŒã®ã¢ã€ãã ãšããŠè¿ãããŸãã 以äžã®ç»åã¯ãœãªã¥ãŒã·ã§ã³ã®ã¢ãŒããã¯ãã£å³ã§ãã 以äžãã¯ãŒã¯ãããŒãšãªããŸãã äºååŠç¿æžã¿ã® CLIP ã¢ãã«ãããããæšè«ãšãªã¢ã«ã¿ã€ã æšè«çšã® SageMaker ã¢ãã« ããããã€ããŸãã SageMaker batch transform ãžã§ãã䜿çšããŠè£œåç»åã® Embedding ãçæããŸãã SageMaker Serverless Inference ã䜿çšããŠãã¯ãšãªç»åãšããã¹ãã Embedding ãžãªã¢ã«ã¿ã€ã ã«ãšã³ã³ãŒãããŸãã Amazon Simple Storage Service (Amazon S3) ã䜿çšããŠãæªå å·¥ã®ããã¹ã (補å説æ) ãšç»å (補åç»å)ãããã³ SageMaker batch transform ãžã§ãã«ãã£ãŠçæãããç»åã® Embedding ãä¿åããŸãã OpenSearch Service ãæ€çŽ¢ãšã³ãžã³ãšããŠäœ¿çšããŠãEmbeddingãä¿åããé¡äŒŒã®Embeddingãæ€çŽ¢ããŸãã ã¯ãšãªé¢æ°ã䜿çšããŠã¯ãšãªã®ãšã³ã³ãŒãã£ã³ã°ã調æŽããk-NN æ€çŽ¢ãå®è¡ããŸãã ãã®ãœãªã¥ãŒã·ã§ã³ãéçºããããã®çµ±åéçºç°å¢ (IDE) ãšããŠãAmazon SageMaker Studio Notebooks ã䜿çšããŠããŸãïŒã¢ãŒããã¯ãã£å³ã«ã¯ç€ºãããŠããŸããïŒ ã ãœãªã¥ãŒã·ã§ã³ã®ã»ããã¢ãã ãœãªã¥ãŒã·ã§ã³ãã»ããã¢ããããã«ã¯ãæ¬¡ã®æé ãå®è¡ããŸãã SageMaker ãã¡ã€ã³ãšãŠãŒã¶ãŒãããã¡ã€ã«ãäœæããŸããæé ã«ã€ããŠã¯ãã ã¯ã€ãã¯ã»ããã¢ããã䜿çšã㊠Amazon SageMaker ãã¡ã€ã³ã«ãªã³ããŒãã£ã³ã°ãã ããåç
§ããŠãã ããã OpenSearch Service ãã¡ã€ã³ãäœæããŸããæé ã«ã€ããŠã¯ãã Amazon OpenSearch ãµãŒãã¹ãã¡ã€ã³ã®äœæãšç®¡ç ããåç
§ããŠãã ããã äžèšã®æé 以å€ã«ãã GitHub ã®èšèŒå
容 ã«åŸã AWS CloudFormation ãã³ãã¬ãŒãã䜿çšããŠãã¡ã€ã³ãäœæããããšãå¯èœã§ããçŸç¶ãã®ãã³ãã¬ãŒãã§ã¯ã€ã³ã¿ãŒãããçµç±ã§æ¥ç¶ãã圢ã«ãªã£ãŠããŸãããVPC ã®ã€ã³ã¿ãŒãã§ã€ã¹ãšã³ããã€ã³ãã䜿çšããŠãAmazon ä»®æ³ãã©ã€ããŒãã¯ã©ãŠã (Amazon VPC) ãã SageMaker Studio ã Amazon S3 ã«æ¥ç¶ããããšãå¯èœã§ããVPC ãšã³ããã€ã³ã (ã€ã³ã¿ãŒãã§ã€ã¹ãšã³ããã€ã³ã) ã䜿çšããããšã§ãVPC ãš SageMaker Studio éã®é信㯠AWS ãããã¯ãŒã¯å
ã§å®å
šãã€å®å
šã«è¡ãããŸããSageMaker Studio Notebookã¯ããã©ã€ããŒã VPC çµç±ã§ OpenSearch Serviceã«æ¥ç¶ã§ãããããéä¿¡ã®ã»ãã¥ãªãã£ã確ä¿ãããŸããOpenSearch Service ãã¡ã€ã³ã¯ãä¿åäžã®ããŒã¿ãæå·åããŸããããã¯ãããŒã¿ãžã®äžæ£ã¢ã¯ã»ã¹ãé²ãã®ã«åœ¹ç«ã€ã»ãã¥ãªãã£æ©èœã§ããããŒãéæå·åã¯ãOpenSearch Service ã®ããã©ã«ãæ©èœã«å ããŠã»ãã¥ãªãã£ãããã«åŒ·åããŸããAmazon S3 ã¯ãå¥ã®æå·åãªãã·ã§ã³ãæå®ããªãéããæ°ãããªããžã§ã¯ãããšã«ãµãŒããŒåŽã®æå·å (SSE-S3) ãèªåçã«é©çšããŸãã OpenSearch Service ãã¡ã€ã³ã§ã¯ãã¢ã€ãã³ãã£ãã£ããŒã¹ã®ããªã·ãŒãã¢ã¿ããããŠããµãŒãã¹ã«ã¢ã¯ã»ã¹ã§ãããŠãŒã¶ãŒãå®è¡ã§ããã¢ã¯ã·ã§ã³ãããã³è©²åœããå Žåã¯ãããã®ã¢ã¯ã·ã§ã³ãå®è¡ã§ãããªãœãŒã¹ãå®çŸ©ã§ããŸãã ã€ã¡ãŒãžãšããã¹ãã®ãã¢ã Embedding ã«ãšã³ã³ãŒããã ãã®ã»ã¯ã·ã§ã³ã§ã¯ãç»åãšããã¹ããåã蟌ã¿ã«ãšã³ã³ãŒãããæ¹æ³ã«ã€ããŠèª¬æããŸããããã§ã¯ãããŒã¿ã®æºåãSageMaker ã§ã®ã¢ãã«ã®äœæãããã³ã¢ãã«ã䜿çšãã Batch Transformã®å®è¡ãå«ãŸããŸãã ïŒèš³è
远èšïŒæ¬èšäºã«ããããŠããã³ãŒãã®å
容ã¯ãã¹ãŠãã¡ãã® GitHub ãªããžã㪠㮠blog_clip.ipynb ã«èšèŒãããŠããŸããSageMaker Studio ã«Git ãªããžããªãã¯ããŒã³ããæ¹æ³ã«ã€ããŠã¯ ãã¡ãã®ããã¥ã¡ã³ã ãåç
§ããŠãã ããã ïŒ ããŒã¿ã®æŠèŠãšæºåæ¹æ³ Python 3 (ããŒã¿ãµã€ãšã³ã¹) ã«ãŒãã«ãæèŒãã SageMaker Studio ããŒãããã¯ã䜿çšããŠãµã³ãã«ã³ãŒããå®è¡ã§ããŸãã ãã®èšäºã§ã¯ã Amazon Berkeley Object Dataset ã䜿çšããŸãããã®ããŒã¿ã»ããã¯ãå€èšèªã¡ã¿ããŒã¿ãš 398,212 æã®ãŠããŒã¯ãªã«ã¿ãã°ç»åãå«ã 147,702 åã®è£œåãªã¹ãã®ã³ã¬ã¯ã·ã§ã³ã§ããååã®ç»åãšååå㯠US è±èªã§ã®ã¿äœ¿çšããŠããŸãããã¢çšã«ã¯ãçŽ 1,600 åã®è£œåã䜿çšããŠããŸãããã®ããŒã¿ã»ããã®è©³çްã«ã€ããŠã¯ã README ãåç
§ããŠãã ãããããŒã¿ã»ããã¯å€éšå
¬éãããŠãããããªãã¯ãª S3 ãã±ããã§ãã¹ããããŠããŸããAmazonååã®åå説æãšã¡ã¿ããŒã¿ãå«ã16åã®ãã¡ã€ã«ã®ãã©ãŒããã㯠listings/metadata/listings_<i>.json.gz ãšãªã£ãŠããŸãã ãã®ãã¢ã§ã¯ãæåã®ã¡ã¿ããŒã¿ãã¡ã€ã«ã䜿çšããŸãã Pandas ã䜿çšããŠã¡ã¿ããŒã¿ãèªã¿èŸŒã¿ãããŒã¿ãã¬ãŒã ãã US è±èªã®ã¿ã€ãã«ãå«ã補åãéžæããŸããPandasã¯ãPythonããã°ã©ãã³ã°èšèªäžã«æ§ç¯ããããªãŒãã³ãœãŒã¹ã®ããŒã¿åæããã³æäœããŒã«ã§ãã main_image_id ãšãã屿§ã䜿çšããŠã€ã¡ãŒãžãèå¥ããŸããæ¬¡ã®ã³ãŒããåç
§ããŠãã ããã meta = pd.read_json("s3://amazon-berkeley-objects/listings/metadata/listings_0.json.gz", lines=True) def func_(x): us_texts = [item["value"] for item in x if item["language_tag"] == "en_US"] return us_texts[0] if us_texts else None meta = meta.assign(item_name_in_en_us=meta.item_name.apply(func_)) meta = meta[~meta.item_name_in_en_us.isna()][["item_id", "item_name_in_en_us", "main_image_id"]] print(f"#products with US English title: {len(meta)}") meta.head() ããŒã¿ãã¬ãŒã ã«ã¯ 1,639 åã®è£œåããããŸããæ¬¡ã«ãã¢ã€ãã åãšç»åãçµåãããŸãã images/metadata/images.csv.gz ã«ã¯ç»åã®ã¡ã¿ããŒã¿ãå«ãŸããŠããŸãããã®ãã¡ã€ã«ã¯ gzip ã§å§çž®ããã CSV ãã¡ã€ã«ã§ã image_id , height , width , path åããããŸããã¡ã¿ããŒã¿ãã¡ã€ã«ãèªã¿èŸŒãã§ããããã¢ã€ãã ã¡ã¿ããŒã¿ãšçµåããŠã¿ãŸããããæ¬¡ã®ã³ãŒããåç
§ããŠãã ããã image_meta = pd.read_csv("s3://amazon-berkeley-objects/images/metadata/images.csv.gz") dataset = meta.merge(image_meta, left_on="main_image_id", right_on="image_id") dataset.head() SageMaker Studio Notebook ã® Python 3 ã«ãŒãã«ã«çµã¿èŸŒãŸããŠãã PIL library ã䜿çšããŠãããŒã¿ã»ããã®ãµã³ãã«ç»åã衚瀺ããŠã¿ãŸãããã from sagemaker.s3 import S3Downloader as s3down from pathlib import Path from PIL import Image def get_image_from_item_id(item_id = "B0896LJNLH", return_image=True): s3_data_root = "s3://amazon-berkeley-objects/images/small/" item_idx = dataset.query(f"item_id == '{item_id}'").index[0] s3_path = dataset.iloc[item_idx].path local_data_root = f'./data/images' local_file_name = Path(s3_path).name s3down.download(f'{s3_data_root}{s3_path}', local_data_root) local_image_path = f"{local_data_root}/{local_file_name}" if return_image: img = Image.open(local_image_path) return img, dataset.iloc[item_idx].item_name_in_en_us else: return local_image_path, dataset.iloc[item_idx].item_name_in_en_us image, item_name = get_image_from_item_id() print(item_name) image ã¢ãã«ã®æºå 次ã«ãäºååŠç¿æžã¿ã® CLIP ã¢ãã«ãã SageMaker ã§æšè«çšã«ã¢ãã«ãããã〠ããŸããæåã®ã¹ãããã¯ãäºååŠç¿æžã¿ã®ã¢ãã«ã®éã¿ãã¡ã€ã«ãããŠã³ããŒãã㊠model.tar.gz ãã¡ã€ã«ã«å
¥ããS3 ãã±ããã«ã¢ããããŒãããããšã§ããäºååŠç¿æžã¿ã¢ãã«ã®ãã¹ã¯ CLIP ãªããžã㪠ã«ãããŸãããã®ãã¢ã§ã¯ãäºååŠç¿æžã¿ã® ResNet-50 (RN50) ã¢ãã«ã䜿çšããŸããæ¬¡ã®ã³ãŒããåç
§ããŠãã ããã %%writefile build_model_tar.sh #!/bin/bash MODEL_NAME=RN50.pt MODEL_NAME_URL=https://openaipublic.azureedge.net/clip/models/afeb0e10f9e5a86da6080e35cf09123aca3b358a0c3e3b6c78a7b63bc04b6762/RN50.pt BUILD_ROOT=/tmp/model_path S3_PATH=s3:////model.tar.gz rm -rf $BUILD_ROOT mkdir $BUILD_ROOT cd $BUILD_ROOT && curl -o $BUILD_ROOT/$MODEL_NAME $MODEL_NAME_URL cd $BUILD_ROOT && tar -czvf model.tar.gz . aws s3 cp $BUILD_ROOT/model.tar.gz $S3_PATH !bash build_model_tar.sh ãã®åŸãæšè«ã® entry point ãšããŠæå®ãã CLIP ã¢ãã«ã®ã¹ã¯ãªãããæäŸããå¿
èŠããããŸãã CLIP 㯠PyTorch ã䜿ã£ãŠå®è£
ãããŠããããã SageMaker PyTorch Framework ã䜿çšããŸãã PyTorch ã¯ãªãŒãã³ãœãŒã¹ã® ML ãã¬ãŒã ã¯ãŒã¯ã§ãç ç©¶ãããã¿ã€ãã³ã°ããæ¬çªãããã€ãŸã§ã®ãã¹ãå éãããŸãã SageMaker ã§ PyTorch ã¢ãã«ããããã€ããæ¹æ³ã«ã€ããŠã¯ã Deploy PyTorch Models ãåç
§ããŠãã ããã æšè«ã³ãŒãã¯ã MODEL_NAME ãš ENCODE_TYPE ã®2ã€ã®ç°å¢å€æ°ãåãå
¥ããŸãã ããã«ãããç°ãªã CLIP ã¢ãã«ãç°¡åã«åãæ¿ããããšãã§ããŸãã ENCODE_TYPE ã䜿çšããŠãã€ã¡ãŒãžãŸãã¯ããã¹ãã®ã©ã¡ãããšã³ã³ãŒãããããæå®ããŸããããã§ã¯ã ããã©ã«ãã® PyTorch æšè«ãã³ãã© ããªãŒããŒã©ã€ãããããã«ã model_fn ã input_fn ã predict_fn ã output_fn 颿°ãå®è£
ããŸãã æ¬¡ã®ã³ãŒããåç
§ããŠãã ãã: !mkdir -p code %%writefile code/clip_inference.py import io import torch import clip from PIL import Image import json import logging import sys import os import torch import torch.nn as nn import torch.nn.functional as F from torchvision.transforms import ToTensor logger = logging.getLogger(__name__) logger.setLevel(logging.DEBUG) logger.addHandler(logging.StreamHandler(sys.stdout)) MODEL_NAME = os.environ.get("MODEL_NAME", "RN50.pt") # ENCODE_TYPE could be IMAGE or TEXT ENCODE_TYPE = os.environ.get("ENCODE_TYPE", "TEXT") device = torch.device("cuda" if torch.cuda.is_available() else "cpu") # defining model and loading weights to it. def model_fn(model_dir): model, preprocess = clip.load(os.path.join(model_dir, MODEL_NAME), device=device) return {"model_obj": model, "preprocess_fn": preprocess} def load_from_bytearray(request_body): return image # data loading def input_fn(request_body, request_content_type): assert request_content_type in ( "application/json", "application/x-image", ), f"{request_content_type} is an unknown type." if request_content_type == "application/json": data = json.loads(request_body)["inputs"] elif request_content_type == "application/x-image": image_as_bytes = io.BytesIO(request_body) data = Image.open(image_as_bytes) return data # inference def predict_fn(input_object, model): model_obj = model["model_obj"] # for image preprocessing preprocess_fn = model["preprocess_fn"] assert ENCODE_TYPE in ("TEXT", "IMAGE"), f"{ENCODE_TYPE} is an unknown encode type." # preprocessing if ENCODE_TYPE == "TEXT": input_ = clip.tokenize(input_object).to(device) elif ENCODE_TYPE == "IMAGE": input_ = preprocess_fn(input_object).unsqueeze(0).to(device) # inference with torch.no_grad(): if ENCODE_TYPE == "TEXT": prediction = model_obj.encode_text(input_) elif ENCODE_TYPE == "IMAGE": prediction = model_obj.encode_image(input_) return prediction # Serialize the prediction result into the desired response content type def output_fn(predictions, content_type): assert content_type == "application/json" res = predictions.cpu().numpy().tolist() return json.dumps(res) ãã®ãœãªã¥ãŒã·ã§ã³ã§ã¯ãã¢ãã«ã®æšè«æã«è¿œå ã® Python ããã±ãŒãžãå¿
èŠã§ãããã®ãããSageMaker ãã¢ãã«ããã¹ãã£ã³ã°ããéã«è¿œå ã®ããã±ãŒãžãã€ã³ã¹ããŒã«ã§ããããã«ã requirements.txt ãã¡ã€ã«ãæäŸããããšãã§ããŸãã %%writefile code/requirements.txt ftfy regex tqdm git+https://github.com/openai/CLIP.git PyTorchModel ã¯ã©ã¹ ã䜿çšãããšãã¢ãã«ææç©ã®Amazon S3 ã®å Žæãšæšè«ãšã³ããªãã€ã³ãã®è©³çްãå«ããªããžã§ã¯ããäœæã§ããŸãããã®ãªããžã§ã¯ãã䜿çšããŠãBatch Transform ãžã§ããäœæãããããšã³ããã€ã³ãã«ãããã€ããŠãªã³ã©ã€ã³æšè«ãè¡ãããšãã§ããŸãã以äžã®ã³ãŒããåç
§ããŠãã ãã: from sagemaker.pytorch import PyTorchModel from sagemaker import get_execution_role, Session role = get_execution_role() shared_params = dict( entry_point="clip_inference.py", source_dir="code", role=role, model_data="s3://<your-bucket>/<your-prefix-for-model>/model.tar.gz", framework_version="1.9.0", py_version="py38", ) clip_image_model = PyTorchModel( env={'MODEL_NAME': 'RN50.pt', "ENCODE_TYPE": "IMAGE"}, name="clip-image-model", **shared_params ) clip_text_model = PyTorchModel( env={'MODEL_NAME': 'RN50.pt', "ENCODE_TYPE": "TEXT"}, name="clip-text-model", **shared_params ) ã¢ã€ãã ç»åã Embeding ã«ãšã³ã³ãŒãããããã® Batch Transform ãå®è¡ãã æ¬¡ã«ãCLIP ã¢ãã«ã䜿çšããŠãã¢ã€ãã ç»åã Embedding ã«ãšã³ã³ãŒãããSageMaker Batch Transformã䜿çšããŠãããæšè«ãå®è¡ããŸãã ãžã§ããäœæããåã«ã以äžã®ã³ãŒãã¹ããããã䜿çšããŠãAmazon Berkeley Objects Dataset ã®ãããªã㯠S3 ãã±ããããã¢ã€ãã ç»åãèªåã®ãã±ããã«ã³ããŒããŸãããã®æäœã¯10åæªæºã§çµäºããŸãã from multiprocessing.pool import ThreadPool import boto3 from tqdm import tqdm from urllib.parse import urlparse s3_sample_image_root = "s3://<your-bucket>/<your-prefix-for-sample-images>" s3_data_root = "s3://amazon-berkeley-objects/images/small/" client = boto3.client('s3') def upload_(args): client.copy_object(CopySource=args["source"], Bucket=args["target_bucket"], Key=args["target_key"]) arugments = [] for idx, record in dataset.iterrows(): argument = {} argument["source"] = (s3_data_root + record.path)[5:] argument["target_bucket"] = urlparse(s3_sample_image_root).netloc argument["target_key"] = urlparse(s3_sample_image_root).path[1:] + record.path arugments.append(argument) with ThreadPool(4) as p: r = list(tqdm(p.imap(upload_, arugments), total=len(dataset))) 次ã«ããããåŠçã§ã¢ã€ãã ç»åã®æšè«ãè¡ããŸããSageMakerã® Batch Transform ãžã§ãã¯ãAmazon S3 ã®å
¥åçšãã£ã¬ã¯ããªã«æ ŒçŽãããŠãããã¹ãŠã®ç»åããšã³ã³ãŒãããããã«CLIPã¢ãã«ã䜿çšããåºåããã Embedding ãåºåS3ãã©ã«ãã«ã¢ããããŒãããŸãã ãã®ãžã§ãã«ã¯çŽ10åããããŸãã batch_input = s3_sample_image_root + "/" output_path = f"s3://<your-bucket>/inference/output" clip_image_transformer = clip_image_model.transformer( instance_count=1, instance_type="ml.c5.xlarge", strategy="SingleRecord", output_path=output_path, ) clip_image_transformer.transform( batch_input, data_type="S3Prefix", content_type="application/x-image", wait=True, ) Amazon S3 ãã Embeddings ã倿°ã«èªã¿èŸŒã¿ãåŸã»ã©OpenSearch Service ã«ããŒã¿ãæ ŒçŽã§ããããã«ããŸãã embedding_root_path = "./data/embedding" s3down.download(output_path, embedding_root_path) embeddings = [] for idx, record in dataset.iterrows(): embedding_file = f"{embedding_root_path}/{record.path}.out" embeddings.append(json.load(open(embedding_file))[0]) æ©æ¢°åŠç¿ã掻çšããçµ±åæ€çŽ¢ãšã³ãžã³ãäœæãã ãã®ã»ã¯ã·ã§ã³ã§ã¯ãEmbeddings ã䜿çšãã k-NN æ€çŽ¢ãå®è¡ããæ€çŽ¢ãšã³ãžã³ã®äœææ¹æ³ã«ã€ããŠèª¬æããŸããããã«ã¯ãOpenSearch Serviceã¯ã©ã¹ã¿ãŒã®æ§æãEmbeddings ã®åã蟌ã¿ãããªãŒããã¹ãããã³ç»åæ€çŽ¢ã¯ãšãªã®å®è¡ãå«ãŸããŸãã k-NN ã䜿ã£ãŠ OpenSeach Service ãã¡ã€ã³ãèšå®ãã 以åã«ãOpenSearch ã¯ã©ã¹ã¿ãŒãäœæããŸãããããã§ã¯ãã«ã¿ãã°ããŒã¿ãšåã蟌ã¿ãæ ŒçŽããããã®ã€ã³ããã¯ã¹ãäœæããŸããã€ã³ããã¯ã¹èšå®ãæ¬¡ã®æ§æã§æ§æããŠãk-NNæ©èœãæå¹ã«ã§ããŸãã index_settings = { "settings": { "index.knn": True, "index.knn.space_type": "cosinesimil" }, "mappings": { "properties": { "embeddings": { "type": "knn_vector", "dimension": 1024 #Make sure this is the size of the embeddings you generated, for RN50, it is 1024 } } } } ãã®äŸã§ã¯ã Python Elasticsearch Client ã䜿çšããŠOpenSearch ã¯ã©ã¹ã¿ãŒãšéä¿¡ããããŒã¿ããã¹ãããã€ã³ããã¯ã¹ãäœæããŸãã ããŒãããã¯ã§ %pip install elasticsearch ãå®è¡ããŠã©ã€ãã©ãªãã€ã³ã¹ããŒã«ã§ããŸãã æ¬¡ã®ã³ãŒããåç
§ããŠãã ãã: import boto3 import json from requests_aws4auth import AWS4Auth from elasticsearch import Elasticsearch, RequestsHttpConnection def get_es_client(host = "<your-opensearch-service-domain-url>", port = 443, region = "<your-region>", index_name = "clip-index"): credentials = boto3.Session().get_credentials() awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, 'es', session_token=credentials.token) headers = {"Content-Type": "application/json"} es = Elasticsearch(hosts=[{'host': host, 'port': port}], http_auth=awsauth, use_ssl=True, verify_certs=True, connection_class=RequestsHttpConnection, timeout=60 # for connection timeout errors ) return es es = get_es_client() es.indices.create(index=index_name, body=json.dumps(index_settings)) ç»åã® Embedding ããŒã¿ã OpenSearch Service ãžæå
¥ãã ããã§ãããŒã¿ã»ãããã«ãŒãåŠçããŠãã¯ã©ã¹ã¿ãŒã«ã¢ã€ãã ããŒã¿ãåã蟌ã¿ãŸãã ãã®ç·Žç¿ã®ããã®ããŒã¿åã蟌ã¿ã¯60ç§ä»¥å
ã«å®äºããã¯ãã§ãã ãŸããããŒã¿ãã€ã³ããã¯ã¹ã«æ£åžžã«åã蟌ãŸããããšã確èªããããã®ã·ã³ãã«ãªã¯ãšãªãå®è¡ããŸãã æ¬¡ã®ã³ãŒããåç
§ããŠãã ãã: # ingest_data_into_es for idx, record in tqdm(dataset.iterrows(), total=len(dataset)): body = record[['item_name_in_en_us']].to_dict() body['embeddings'] = embeddings[idx] es.index(index=index_name, id=record.item_id, doc_type='_doc', body=body) # Check that data is indeed in ES res = es.search( index=index_name, body={ "query": { "match_all": {} }}, size=2) assert len(res["hits"]["hits"]) > 0 ãªã¢ã«ã¿ã€ã ã§ã¯ãšãªãå®è¡ãã ããã§åšåº«ã®ç»åã¢ã€ãã ã® Embeddings ãå«ãŸããOpenSearch Service Index ãçšæã§ããã®ã§ã次ã¯ã¯ãšãªã® Embeddings ãçæããæ¹æ³ãèŠãŠãããŸããããããã¹ããšç»åã® Embeddings ãããããåŠçããããã«ã2 ã€ã®SageMaker æšè«ãšã³ããã€ã³ããäœæããå¿
èŠããããŸãã ãŸãããšã³ããã€ã³ãã䜿çšããŠç»åãšããã¹ãããšã³ã³ãŒããã 2 ã€ã®é¢æ°ãäœæããŸãã encode_text 颿°ã®å Žåã this is ãã¢ã€ãã åã®åã«è¿œå ããŠãã¢ã€ãã åãã¢ã€ãã ã®èª¬ææãšããŠç¿»èš³ããŸãã Transformer ãš ResNet ã¢ãã«ããµããŒãããããã«ã memory_size_in_mb 㯠6 GB ã«èšå®ãããŠããŸãã以äžã®ã³ãŒããåç
§ããŠãã ãã: text_predictor = clip_text_model.deploy( instance_type='ml.c5.xlarge', initial_instance_count=1, serverless_inference_config=ServerlessInferenceConfig(memory_size_in_mb=6144), serializer=JSONSerializer(), deserializer=JSONDeserializer(), wait=True ) image_predictor = clip_image_model.deploy( instance_type='ml.c5.xlarge', initial_instance_count=1, serverless_inference_config=ServerlessInferenceConfig(memory_size_in_mb=6144), serializer=IdentitySerializer(content_type="application/x-image"), deserializer=JSONDeserializer(), wait=True ) def encode_image(file_name="./data/images/0e9420c6.jpg"): with open(file_name, "rb") as f: payload = f.read() payload = bytearray(payload) res = image_predictor.predict(payload) return res[0] def encode_name(item_name): res = text_predictor.predict({"inputs": [f"this is a {item_name}"]}) return res[0] ãŸã䜿çšããç»åãããããããŸãã item_image_path, item_name = get_image_from_item_id(item_id = "B0896LJNLH", return_image=False) feature_vector = encode_image(file_name=item_image_path) print(feature_vector.shape) Image.open(item_image_path) ç°¡åãªã¯ãšãªã®çµæãèŠãŠã¿ãŸããããOpenSearch ServiceããçµæãååŸããåŸã dataset ããã¢ã€ãã åãšç»åã®ãªã¹ããååŸããŸãã def search_products(embedding, k = 3): body = { "size": k, "_source": { "exclude": ["embeddings"], }, "query": { "knn": { "embeddings": { "vector": embedding, "k": k, } } }, } res = es.search(index=index_name, body=body) images = [] for hit in res["hits"]["hits"]: id_ = hit["_id"] image, item_name = get_image_from_item_id(id_) image.name_and_score = f'{hit["_score"]}:{item_name}' images.append(image) return images def display_images( images: [PilImage], columns=2, width=20, height=8, max_images=15, label_wrap_length=50, label_font_size=8): if not images: print("No images to display.") return if len(images) > max_images: print(f"Showing {max_images} images of {len(images)}:") images=images[0:max_images] height = max(height, int(len(images)/columns) * height) plt.figure(figsize=(width, height)) for i, image in enumerate(images): plt.subplot(int(len(images) / columns + 1), columns, i + 1) plt.imshow(image) if hasattr(image, 'name_and_score'): plt.title(image.name_and_score, fontsize=label_font_size); images = search_products(feature_vector) 2 ã€ã®ç»åãåãã§ãããããæåã®é
ç®ã®ã¹ã³ã¢ã¯ 1.0 ã§ãããã®ä»ã®é
ç®ã¯ãOpenSearch Service Index ã«ããããŸããŸãªçš®é¡ã®ãŠã©ãŒã¿ãŒã°ã©ã¹ã«ãªããŸãã ããã¹ãã䜿çšããŠã€ã³ããã¯ã¹ãã¯ãšãªããããšãã§ããŸãã feature_vector = encode_name("drinkware glass") images = search_products(feature_vector) display_images(images) ããã§ãã€ã³ããã¯ã¹ãããŠã©ãŒã¿ãŒã°ã©ã¹ã®åçã 3 æååŸã§ããããã«ãªããŸãããCLIP ãšã³ã³ãŒããŒã䜿çšãããšãåãæœåšç©ºéå
ã®ç»åãšããã¹ããæ€çŽ¢ã§ããŸããããã®ãã 1 ã€ã®äŸã¯ã玢åŒã§ããã¶ããšããåèªãæ€çŽ¢ããããšã§ãã feature_vector = encode_name("pizza") images = search_products(feature_vector) display_images(images) Clean up åŸé課éå¶ã¢ãã«ã® Serverless Inference ã¯ãé »åºŠã®äœããã©ãã£ãã¯ãã¿ãŒã³ãäºæž¬äžå¯èœãªãã©ãã£ãã¯ãã¿ãŒã³ã«å¯ŸããŠè²»çšå¯Ÿå¹æã®é«ããªãã·ã§ã³ã§ãã峿 Œãª ãµãŒãã¹ã¬ãã«å¥çŽ (SLA) ãç· çµããŠããå Žåããã³ãŒã«ãã¹ã¿ãŒãã«èããããªãå Žåã¯ããªã¢ã«ã¿ã€ã ãšã³ããã€ã³ãã®æ¹ãé©ããŠããŸãã ãã«ãã¢ãã« ãŸã㯠ãã«ãã³ã³ãã ã®ãšã³ããã€ã³ãã䜿çšãããšã倿°ã®ã¢ãã«ããããã€ããããã®ã¹ã±ãŒã©ãã«ã§è²»çšå¯Ÿå¹æã®é«ããœãªã¥ãŒã·ã§ã³ãåŸãããŸãã詳现ã«ã€ããŠã¯ã Amazon SageMaker ã®æéããŒãž ãåç
§ããŠãã ããã ãµãŒããŒã¬ã¹ãšã³ããã€ã³ãã¯ãäžèŠã«ãªã£ããåé€ããããšããå§ãããŸãããã®æŒç¿ãçµäºããããæ¬¡ã®æé ã§ãªãœãŒã¹ãåé€ã§ããŸã (ãããã®ãªãœãŒã¹ã¯ã AWS ãããžã¡ã³ãã³ã³ãœãŒã« ããããŸã㯠AWS SDK ãŸã㯠SageMaker SDK ã䜿çšããŠåé€ã§ããŸã)ã äœæãããšã³ããã€ã³ããåé€ããŸãã (ãªãã·ã§ã³) ç»é²ãããã¢ãã«ãåé€ããŸãã (ãªãã·ã§ã³) SageMaker å®è¡ããŒã«ãåé€ããŸãã (ãªãã·ã§ã³) S3 ãã±ããã空ã«ããŠåé€ããŸãã Summary ãã®èšäºã§ã¯ãSageMaker ãš OpenSearch Service ã® k-NN ã€ã³ããã¯ã¹æ©èœã䜿çšã㊠k-NN æ€çŽ¢ã¢ããªã±ãŒã·ã§ã³ãäœæããæ¹æ³ã瀺ããŸãããOpenAI å®è£
ã®äºåãã¬ãŒãã³ã°æžã¿ã® CLIP ã¢ãã«ã䜿çšããŸããã æçš¿ã® OpenSearch ãµãŒãã¹ã®ã€ã³ãžã§ã¹ãå®è£
ã¯ããããã¿ã€ãã³ã°ã«ã®ã¿äœ¿çšãããŸããAmazon S3 ãã OpenSearch ãµãŒãã¹ã«å€§èŠæš¡ã«ããŒã¿ãåã蟌ã¿ããå Žåã¯ãé©åãªã€ã³ã¹ã¿ã³ã¹ã¿ã€ããšã€ã³ã¹ã¿ã³ã¹æ°ã§ Amazon SageMaker åŠçãžã§ããèµ·åã§ããŸããå¥ã®ã¹ã±ãŒã©ãã«ãªåã蟌ã¿åã蟌ã¿ãœãªã¥ãŒã·ã§ã³ã«ã€ããŠã¯ãã Novartis AG uses Amazon OpenSearch Service K-Nearest Neighbor (KNN) and Amazon SageMaker to power search and recommendation (Part 3/4) ããåç
§ããŠãã ããã CLIP㯠ãŒãã·ã§ãã æ©èœãåããŠããããã 転移åŠç¿ ã䜿çšããŠã¢ãã«ã埮調æŽããªããŠããäºåã«ãã¬ãŒãã³ã°ãããã¢ãã«ãçŽæ¥æ¡çšã§ããŸããããã«ãããCLIPã¢ãã«ã®é©çšãç°¡åã«ãªããŸãã補åç»åãšèª¬ææã®äž¡æ¹ãããå Žåã¯ã転移åŠç¿ã䜿çšããŠç¬èªã®ããŒã¿ã§ã¢ãã«ã埮調æŽããã¢ãã«ã®ããã©ãŒãã³ã¹ãããã«åäžãããããšãã§ããŸãã詳现ã«ã€ããŠã¯ãã Learning Transferable Visual Models From Natural Language Supervision ããšã CLIP GitHub ãªããžã㪠ããåç
§ããŠãã ããã èè
ã«ã€ã㊠Kevin Du 㯠AWS ã®ã·ãã¢ããŒã¿ã©ãã¢ãŒããã¯ãã§ãã客æ§ã®æ©æ¢°åŠç¿ (ML) 補åãš MLOps ãã©ãããã©ãŒã ã®éçºãä¿é²ã§ããããæ¯æŽããããšã«å°å¿µããŠããŸããã¹ã¿ãŒãã¢ãããšäŒæ¥ã®äž¡æ¹ã察象㫠ML 察å¿è£œåãéçºããŠãã 10 幎以äžã®çµéšãæã€åœŒã¯ãã客æ§ã ML ãœãªã¥ãŒã·ã§ã³ã®æ¬çªåãåçåã§ããããæ¯æŽããããšã«éç¹ã眮ããŠããŸããèªç±æéã«ã¯ãã±ãã³ã¯æçããã¹ã±ããããŒã«èгæŠã楜ããã§ããŸãã Ananya Royã¯ããªãŒã¹ãã©ãªã¢ã®ã·ãããŒãæ ç¹ãšããAIãšæ©æ¢°åŠç¿ãå°éãšããã·ãã¢ããŒã¿ã©ãã¢ãŒããã¯ãã§ãã圌女ã¯ããŸããŸãªé¡§å®¢ãšååããŠã¢ãŒããã¯ãã£ã®ã¬ã€ãã³ã¹ãæäŸããããŒã¿ã©ããšã®é£æºãéããŠå¹æçãªAI/MLãœãªã¥ãŒã·ã§ã³ãæäŸã§ããããæ¯æŽããŠããŸãããAWS ã«å
¥ç€Ÿããåã¯ãã·ãã¢ããŒã¿ãµã€ãšã³ãã£ã¹ããšããŠåããéä¿¡äŒç€Ÿãéè¡ããã£ã³ããã¯ãªã©ã®ããŸããŸãªæ¥çã®å€§èŠæš¡ãª ML ã¢ãã«ãæ±ã£ãŠããŸãããAI/ML ã®çµéšã«ãããè€éãªããžãã¹äžã®åé¡ã«å¯ŸããŠå¹æçãªãœãªã¥ãŒã·ã§ã³ãæäŸã§ããæå
端ã®ãã¯ãããžãŒã掻çšããŠããŒã ãç®æšãéæã§ããããæ¯æŽããããšã«æ
ç±ã泚ãã§ããŸãã 翻蚳ã¯ãœãªã¥ãŒã·ã§ã³ã¢ãŒããã¯ãã®èŸ» 浩å£ãæ
åœããŸãããåæã¯ ãã¡ã ã§ãã