æ¬èšäºã¯ 2026 幎 8 æ 25 æ¥ ã«å
¬éãããã PythonOperator and BashOperator Now Available on Amazon Managed Workflows for Apache Airflow (Amazon MWAA) Serverless ãã翻蚳ãããã®ã§ãã翻蚳ã¯ã¯ã©ãŠããµããŒããšã³ãžãã¢ã®å±±æ¬ãæ
åœããŸããã Amazon MWAA Serverless ã§ Apache Airflow ã¯ãŒã¯ãã㌠ãå®è¡ããŠããå ŽåãPythonOperator ãš BashOperator ã䜿ã£ãŠã«ã¹ã¿ã ã³ãŒãããµãŒããŒã¬ã¹ã©ã³ã¿ã€ã äžã§çŽæ¥å®è¡ã§ããããã«ãªããŸããããããŸã§ Amazon Managed Workflows for Apache Airflow (Amazon MWAA) Serverless ã§ã¯ããªãã¬ãŒã¿ãŒçµç±ã§ AWS ãµãŒãã¹ããªãŒã±ã¹ãã¬ãŒã·ã§ã³ããã¿ã¹ã¯ã®ã¹ã±ãžã¥ãŒãªã³ã°ãäŸåé¢ä¿ã®ç®¡çããªãã©ã€åŠçãè¡ãããšããã§ãããç¬èªã® Python 颿°ãã·ã§ã«ã¹ã¯ãªããããã€ãã£ãã«å®è¡ã§ããŸããã§ãããã«ã¹ã¿ã ã® Python ããžãã¯ãã·ã§ã«ã³ãã³ããå¿
èŠãªå Žåã¯ãã³ãŒãã AWS Lambda 颿°ã«ã©ããããããAmazon Elastic Container Service (Amazon ECS) ã¿ã¹ã¯ãèµ·åããããã»ãã® AWS ã³ã³ãã¥ãŒãã£ã³ã°ãµãŒãã¹ã䜿ãå¿
èŠããããŸãããããããä»£æ¿ææ®µã§ã¯ããªãŒã±ã¹ãã¬ãŒã·ã§ã³ãã€ãã©ã€ã³ã®è€éããã³ã¹ããã¬ã€ãã³ã·ãŒãå¢ããŸãã ä»åã®æ©èœè¿œå ã«ãããã€ã³ãã©ã¹ãã©ã¯ãã£ã远å ããã«ããµãŒããŒã¬ã¹ã¿ã¹ã¯ã©ã³ã¿ã€ã å
ã§ã«ã¹ã¿ã ã® Python 颿°ãã·ã§ã«ã¹ã¯ãªãããçŽæ¥å®è¡ã§ããŸããã€ãŸããå€ãã®ããŒã¿ãšã³ãžãã¢ãªã³ã°ããŒã ã ETL ãã€ãã©ã€ã³ãããŒã¿å質ãã§ãã¯ã§å©çšããŠãã PythonOperator ãš BashOperator ããã³ã³ãã¥ãŒãã£ã³ã°ãªãœãŒã¹ã远å ã§ããããžã§ãã³ã°ããã«äœ¿ããŸãã æ¬èšäºã§ã¯ãæ°æ©èœã®ä»çµã¿ã解説ããå®è·µçãªäŸã瀺ããŸããPythonOperator ã§ CSV ãã¡ã€ã«ã JSON 圢åŒã«å€æããBashOperator ã§åºåãæ€èšŒãããµãŒããŒã¬ã¹ãã€ãã©ã€ã³ãæ§ç¯ããŸããèªã¿çµãããšã次ã®ããšãã§ããããã«ãªããŸãã äŸåé¢ä¿ãå«ã Python ã¢ãžã¥ãŒã«ãããã±ãŒãžåããã³ãŒããã³ãã«ãšã㊠Amazon Simple Storage Service (Amazon S3) ãã±ããã«ã¢ããããŒããã dag-factory äºæã® YAML ã§è€æ°ã¿ã¹ã¯ã®ã¯ãŒã¯ãããŒãå®çŸ©ãã AWS Command Line Interface (AWS CLI) ã§ã¯ãŒã¯ãããŒãäœæããŠå®è¡ãã ãã€ãã©ã€ã³ãæåŸ
ã©ããã®åºåãçæããããšãæ€èšŒãã ä»çµã¿ MWAA Serverless ã§ã¯ãã«ã¹ã¿ã ã³ãŒããããã±ãŒãžåã㊠Amazon S3 ãã±ããã«ã¢ããããŒãããã¯ãŒã¯ãããŒäœææã«åç
§ããŸãããµãŒãã¹ã¯ã¯ãŒã¯ãããŒäœææç¹ã®ã³ãŒããã¹ãããã·ã§ãããšããŠååŸãã以éã¯åãã¯ãŒã¯ãããŒããŒãžã§ã³ã®ãã¹ãŠã®å®è¡ã§ãã®ã¹ãããã·ã§ããã䜿ããŸãã ã³ãŒããã³ãã« ã³ãŒããã³ãã«ã¯ãã«ã¹ã¿ã ããžãã¯ãå«ãããã±ãŒãžã§ããPython ã¢ãžã¥ãŒã«ãã·ã§ã«ã¹ã¯ãªãããããã±ãŒãžåã㊠Amazon S3 ãã±ããã«ã¢ããããŒãããŸããã³ãŒããã³ãã«ã®åœ¢åŒã¯æ¬¡ã®ããããã§ãã åäžã® .py ãã¡ã€ã«ãŸã㯠.sh ã® bash ã¹ã¯ãªãã (Amazon S3 ãã±ããã«ã¢ããããŒã) è€æ°ã®ã·ã§ã«ã¹ã¯ãªãããPython ã¢ãžã¥ãŒã«ãäŸåé¢ä¿ãå«ã ZIP ã¢ãŒã«ã€ã (æå€§ 250 MB) å®è¡ã¢ãã« ã¯ãŒã¯ãããŒãäœæãŸãã¯æŽæ°ãããšãMWAA Serverless ã¯æå®ãã Amazon S3 ãã±ããããã³ãŒããã³ãã«ã®ã¹ãããã·ã§ãããååŸãããµãŒãã¹åŽã«ä¿åããŸããã¿ã¹ã¯å®è¡æã«ã¯ãAmazon S3 ãã±ããã«çŸåšçœ®ãããŠãããªããžã§ã¯ãã§ã¯ãªãããã®ã¹ãããã·ã§ããã䜿ã£ãŠéé¢ãããã©ã³ã¿ã€ã ç°å¢ã§ã³ãŒããå®è¡ããŸãã Python ã¿ã¹ã¯ãš Bash ã¿ã¹ã¯ã¯ã€ã³ã¿ãŒãããã«ã¢ã¯ã»ã¹ã§ããŸãããå°éã§ããã®ã¯ãã©ã³ã¿ã€ã ã®åäœã«å¿
èŠãª Amazon S3ãAmazon Elastic Container Registry (Amazon ECR)ãAmazon CloudWatch ã ãã§ããã€ã³ã¿ãŒãããã¢ã¯ã»ã¹ãå¿
èŠãªå Žåã¯ã ã¯ãŒã¯ãããŒã« Amazon VPC ãèšå® ããŠããã® VPC çµç±ã§éä¿¡ãããŠãã ããã ãµããŒãããããªãã¬ãŒã¿ãŒ MWAA Serverless ã§å©çšã§ããããã«ãªã£ã 2 ã€ã®ãªãã¬ãŒã¿ãŒã¯æ¬¡ã®ãšããã§ãã ãªãã¬ãŒã¿ãŒ 説æ PythonOperator ã³ãŒããã³ãã«å
ã® Python ã®åŒã³åºãå¯èœãªããžã§ã¯ã (颿°) ãå®è¡ããŸã BashOperator ã·ã§ã«ã³ãã³ããã¹ã¯ãªãããå®è¡ããŸã ã»ãã¥ãªã㣠ã³ãŒããã³ãã«ã¯ AWS Key Management Service (AWS KMS) ã§ä¿åæã«æå·åãããŸããã¯ãŒã¯ãããŒãäœæãæŽæ°ãããªã¬ãŒã§ãããŠãŒã¶ãŒã¯ IAM ããªã·ãŒã§å¶åŸ¡ããŸããå®è¡æã«ã³ãŒããã¢ã¯ã»ã¹ã§ãã AWS ãªãœãŒã¹ã®ç¯å²ã¯å®è¡ããŒã«ã§æ±ºãŸããŸãã åææ¡ä»¶ å§ããåã«ã次ã®ãªãœãŒã¹ãšããŒã«ã AWS ã¢ã«ãŠã³ãã§èšå®ãããŠããããšã確èªããŠãã ããã Amazon MWAA Serverless ã«ã¢ã¯ã»ã¹ã§ãã AWS ã¢ã«ãŠã³ã AWS CLI v2 (ææ°ããŒãžã§ã³) ã®ã€ã³ã¹ããŒã«ãšèšå®ãã€ã³ã¹ããŒã«ãŸãã¯æŽæ°ã®æ¹æ³ã¯ AWS CLI ã®ææ°ããŒãžã§ã³ã®ã€ã³ã¹ããŒã«ãŸãã¯æŽæ° ãåç
§ããŠãã ããã DAG å®çŸ©ãšã³ãŒããã³ãã«ãä¿åãã Amazon S3 ãã±ãã MWAA Serverless ãåŒãåãããã IAM ããŒã« (å®è¡ããŒã«ã®èšå®ã¯åŸè¿°ããŸã) ãŠã©ãŒã¯ã¹ã«ãŒ: ãµãŒããŒã¬ã¹ã® CSV â JSON ãã€ãã©ã€ã³ãæ§ç¯ãã â»ä»¥éã® Amazon S3 ãã±ããå amzn-s3-demo-mwaa-data ã¯ãµã³ãã«ã§ãããå©çšã® Amazon S3 ãã±ããåã«å€æŽããŠãã ããã ãã®ãŠã©ãŒã¯ã¹ã«ãŒã§ã¯ãCSV ãã¡ã€ã«ã JSON 圢åŒã«å€æãããã€ãã©ã€ã³ãæ§ç¯ããŸããJSON ãæ±ãäžæµã® API ãåæã·ã¹ãã ã«åãããããããããŒã¿å€æã§ãã倿ããžãã¯ã«ã¯ PythonOperator ããåºåã®æ€èšŒã«ã¯ BashOperator ã䜿ããŸãããã€ãã©ã€ã³ã®åŠçå
å®¹ã¯æ¬¡ã®ãšããã§ãã Amazon S3 ãã±ãããã CSV ãã¡ã€ã«ãèªã¿èŸŒã åã®åãæšè«ããªãã JSON 圢åŒã«å€æãã JSON ãã¡ã€ã«ã Amazon S3 ãã±ããã«æžãæ»ã 倿å
ãšåºåã§ã¬ã³ãŒãä»¶æ°ãäžèŽããããšãæ€èšŒãã ã¹ããã 1: å®è¡ããŒã«ãäœæãã ã¯ãŒã¯ãããŒãå®è¡æã«åŒãåãã IAM ããŒã«ãäœæããŸããä¿¡é Œããªã·ãŒã§ã¯ airflow-serverless.amazonaws.com ãµãŒãã¹ãããŒã«ãåŒãåããããããã«ããå¿
èŠããããŸãã cat > trust-policy.json << 'EOF' { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "airflow-serverless.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } EOF ããŒã«ãäœæããS3 ãã±ãããžã®æå°æš©éã¢ã¯ã»ã¹ãèš±å¯ããã€ã³ã©ã€ã³ããªã·ãŒãã¢ã¿ããããŸãã aws iam create-role \ --role-name MWAAServerlessExecutionRole \ --assume-role-policy-document file://trust-policy.json aws iam put-role-policy \ --role-name MWAAServerlessExecutionRole \ --policy-name MWAAServerlessAccessPolicy \ --policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-mwaa-data", "arn:aws:s3:::amzn-s3-demo-mwaa-data/*" ] }, { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents", "logs:DescribeLogStreams", "logs:GetLogEvents" ], "Resource": "arn:aws:logs:*:*:log-group:/aws/mwaa-serverless/*" } ] }' ã¹ããã 2: Python ã¢ãžã¥ãŒã«ãäœæãã 倿ããžãã¯ãèšè¿°ãã csv_to_json.py ãšãããã¡ã€ã«ãäœæããŸãã # csv_to_json.py import csv import json import boto3 import io def convert(**kwargs): """Read a CSV from S3 and write it back as JSON lines.""" bucket = "amzn-s3-demo-mwaa-data" source_key = "raw/sales_data.csv" output_key = "processed/sales_data.json" s3 = boto3.client("s3") # Read source file response = s3.get_object(Bucket=bucket, Key=source_key) content = response["Body"].read().decode("utf-8") # Parse CSV reader = csv.DictReader(io.StringIO(content)) rows = list(reader) # Type inference - convert numeric fields for row in rows: for key, value in row.items(): try: row[key] = float(value) except (ValueError, TypeError): pass # Write as JSON lines output = "\n".join(json.dumps(row) for row in rows) + "\n" s3.put_object(Bucket=bucket, Key=output_key, Body=output.encode("utf-8")) print(f"Converted {len(rows)} rows to JSON lines") print(f"Output: s3://amzn-s3-demo-mwaa-data/{output_key}") return {"rows": len(rows), "output_key": output_key} ãã®é¢æ°ã¯ boto3 (MWAA Serverless ã®å®è¡ç°å¢ã«ããªã€ã³ã¹ããŒã«æžã¿) ãš Python æšæºã©ã€ãã©ãªã® csv ããã³ json ã¢ãžã¥ãŒã«ã䜿ããŸããCSV ãèªã¿èŸŒãã§æ°å€åãæšè«ããJSON Lines ãã¡ã€ã«ã S3 ãã±ããã«æžãæ»ããŸãã ã¹ããã 3: æ€èšŒã¹ã¯ãªãããäœæãã verify_output.sh ãšãããã¡ã€ã«ãäœæããŸãããã®ã¹ã¯ãªããã¯ã倿å
CSV ãšåºå JSON ãã¡ã€ã«ã®ã¬ã³ãŒãä»¶æ°ãæ¯èŒããŠãã€ãã©ã€ã³ã®åºåãæ€èšŒããŸããä»¶æ°ãäžèŽããªãå Žåãã¿ã¹ã¯ã¯ 0 以å€ã®çµäºã³ãŒãã§å€±æããã¯ãŒã¯ãããŒã®å®è¡ã倱æããŸãã #!/bin/bash echo "=== Data Validation ===" # Count source records (skip CSV header) SOURCE_COUNT=$(python3 -m awscli s3 cp s3://amzn-s3-demo-mwaa-data/raw/sales_data.csv - | tail -n +2 | wc -l) echo "Source CSV records: $SOURCE_COUNT" # Count output records OUTPUT_COUNT=$(python3 -m awscli s3 cp s3://amzn-s3-demo-mwaa-data/processed/sales_data.json - | wc -l) echo "Output JSON records: $OUTPUT_COUNT" # Validate counts match if [ "$SOURCE_COUNT" -ne "$OUTPUT_COUNT" ]; then echo "FAILED: Record count mismatch (source=$SOURCE_COUNT, output=$OUTPUT_COUNT)" exit 1 fi echo "PASSED: Record counts match ($OUTPUT_COUNT records)" echo "Timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ)" æ€èšŒã¹ã¯ãªãã㯠AWS CLI ãå®è¡ããŸããAWS CLI ã¯ã³ãŒãããã±ãŒãžã«äŸåé¢ä¿ãšããŠãã³ãã«ãããŠããŸããs3 cp ã¯ãã¡ã€ã«ã®å
容ããã£ã¹ã¯ã«æžãåºããã« stdout ãžã¹ããªãŒãã³ã°ããããã wc -l ã tail ãšãã£ãæšæºçãªã·ã§ã«ããŒã«ã§åŠçã§ããŸããå®è¡ããŒã«ã®èªèšŒæ
å ±ã¯å®è¡ç°å¢ã§èªåçã«å©çšã§ããã®ã§ã远å ã®èšå®ãªãã« CLI ãã S3 ã«ã¢ã¯ã»ã¹ã§ããŸãã ã¹ããã 4: ã³ãŒããããã±ãŒãžåã㊠Amazon S3 ã«ã¢ããããŒããã æ€èšŒã¹ã¯ãªããã AWS CLI ã䜿ããããPython ã¢ãžã¥ãŒã«ãšã·ã§ã«ã¹ã¯ãªããã«å ããŠãAWS CLI ãäŸåé¢ä¿ãšã㊠ZIP ã¢ãŒã«ã€ãã«ãã³ãã«ããŸãã BUCKET="amzn-s3-demo-mwaa-data" REGION="us-east-1" # Install awscli into a package directory pip install awscli \ --target my_package/ \ --platform manylinux2014_x86_64 \ --python-version 3.12 \ --only-binary=:all: # Add your module cp csv_to_json.py my_package/ cp verify_output.sh my_package/ # Create the ZIP archive cd my_package && zip -r ../code_bundle.zip . && cd .. # Upload to S3 aws s3 cp code_bundle.zip s3://$BUCKET/code/code_bundle.zip --region $REGION ãã¹ãçšã®ãµã³ãã« CSV ãã¡ã€ã«ãã¢ããããŒãããŸãã cat > sales_data.csv << 'EOF' date,region,product,units,revenue 2026-07-01,us-east,widget-a,150,4500.00 2026-07-01,eu-west,widget-b,89,2670.00 2026-07-02,us-east,widget-a,203,6090.00 2026-07-02,ap-south,widget-c,67,1340.00 2026-07-03,us-east,widget-b,178,5340.00 EOF aws s3 cp sales_data.csv s3://$BUCKET/raw/sales_data.csv --region $REGION ã¹ããã 5: DAG ãå®çŸ©ãã (YAML) MWAA Serverless 㯠DAG å®çŸ©ã«å®£èšç㪠YAML 圢åŒã䜿ããŸãã conversion_dag.yaml ãšãããã¡ã€ã«ãäœæããŸãã csv_to_json_pipeline: start_date: "2026-01-01" schedule: null tasks: convert_to_json: operator: airflow.operators.python.PythonOperator python_callable: csv_to_json.convert verify_output: operator: airflow.operators.bash.BashOperator bash_command: "verify_output.sh" dependencies: - convert_to_json ãã® DAG 㯠2 ã€ã®ã¿ã¹ã¯ãå®çŸ©ããŠããŸãã convert_to_json â Python ã¢ãžã¥ãŒã«ã® convert 颿°ãå®è¡ããCSV ã JSON Lines ã«å€æããŸãã verify_output â ã·ã§ã«ã¹ã¯ãªãããå®è¡ãã倿å
ãšåºåã®ã¬ã³ãŒãä»¶æ°ãæ¯èŒããŠãã€ãã©ã€ã³ã®åºåãæ€èšŒããŸããäžèŽããªãå Žåã¯ã¿ã¹ã¯ã倱æãããŸãã DAG å®çŸ©ã S3 ã«ã¢ããããŒãããŸãããªããã·ã§ã«ã¹ã¯ãªããã䜿ããã«ã€ã³ã©ã€ã³ã® Bash ã³ãã³ããçŽæ¥å®è¡ããããšãã§ããŸãã aws s3 cp conversion_dag.yaml s3://$BUCKET/dags/conversion_dag.yaml --region $REGION ã¹ããã 6: ã¯ãŒã¯ãããŒãäœæãã DAG å®çŸ©ãšã³ãŒããã³ãã«ãåç
§ã㊠MWAA Serverless ã¯ãŒã¯ãããŒãäœæããŸãã ROLE_ARN="arn:aws:iam::<your-account-id>:role/MWAAServerlessExecutionRole" aws mwaa-serverless create-workflow \ --name csv-to-json-workflow \ --definition-s3-location Bucket="$BUCKET",ObjectKey="dags/conversion_dag.yaml" \ --code '{"S3Location": {"Bucket":"'"$BUCKET"'","ObjectKey":"code/code_bundle.zip"}}' \ --role-arn $ROLE_ARN \ --region $REGION ã¬ã¹ãã³ã¹ã«ã¯ãå®è¡ãããªã¬ãŒããéã«äœ¿ã WorkflowArn ãå«ãŸããŸãã { "WorkflowArn": "arn:aws:airflow-serverless:us-east-1:123456789012:workflow/csv-to-json-workflow-abc123", "CreatedAt": "2026-07-15T10:30:00.000000+00:00", "WorkflowVersion": "a1b2c3d4e5f6" } ã¹ããã 7: ã¯ãŒã¯ãããŒãå®è¡ãã ã¯ãŒã¯ãããŒã®å®è¡ãããªã¬ãŒããŸãã WORKFLOW_ARN="arn:aws:airflow-serverless:us-east-1:123456789012:workflow/csv-to-json-workflow-abc123" aws mwaa-serverless start-workflow-run \ --workflow-arn $WORKFLOW_ARN \ --region $REGION ã¬ã¹ãã³ã¹ã§å®è¡ãéå§ãããããšã確èªã§ããŸãã { "RunId": "6OZV9ABF9enHKXk", "Status": "STARTING" } ã¹ããã 8: å®è¡ãç£èŠãã å®è¡ã®ã¹ããŒã¿ã¹ã確èªããŸãã RUN_ID="6OZV9ABF9enHKXk" aws mwaa-serverless get-workflow-run \ --workflow-arn $WORKFLOW_ARN \ --run-id $RUN_ID \ --region $REGION å®è¡ãæåãããšæ¬¡ã®ããã«è¿ããŸãã { "RunDetail": { "Duration": 45, "RunState": "SUCCESS", "TaskInstances": ["ex_abc123_convert_to_json_1", "ex_abc123_verify_output_1"] }, "RunId": "6OZV9ABF9enHKXk", "RunType": "ON_DEMAND", "WorkflowArn": "arn:aws:airflow-serverless:us-east-1:123456789012:workflow/csv-to-json-workflow-abc123", "WorkflowVersion": "a1b2c3d4e5f6" } ã¹ããã 9: åºåãæ€èšŒãã JSON ãã¡ã€ã«ã S3 ãã±ããã«æžã蟌ãŸããããšã確èªããŸãã # List the output file aws s3 ls s3://$BUCKET/processed/sales_data.json --region $REGION 次ã®ããã« JSON ãã¡ã€ã«ã衚瀺ãããŸãã 2026-07-15 10:32:45 1847 sales_data.json ã¿ã¹ã¯åäœã®åºå㯠Amazon CloudWatch Logs ã§ã確èªã§ããŸããã¯ãŒã¯ãããŒã®ãã°ã°ã«ãŒããéãã convert_to_json ã¿ã¹ã¯ã®ãã°ã¹ããªãŒã ãæ¢ããŠãã ããã Converted 5 rows to JSON lines Output: s3://amzn-s3-demo-mwaa-data/processed/sales_data.json èæ
®äºé
ãšå¶é PythonOperator ãš BashOperator ã䜿ãã¯ãŒã¯ããŒãã MWAA Serverless ã§èšç»ããéã¯ã次ã®ç¹ã«æ³šæããŠãã ããã ã³ãŒããã³ãã«ã®ãµã€ãº â ZIP ã¢ãŒã«ã€ã㯠1 ãã³ãã«ããã 250 MB æªæºã«ããå¿
èŠããããŸãã ãããã¯ãŒã¯ã¢ã¯ã»ã¹ â Python ã¿ã¹ã¯ãš Bash ã¿ã¹ã¯ã¯ã€ã³ã¿ãŒãããã«ã¢ã¯ã»ã¹ã§ããŸãããã©ã³ã¿ã€ã ã®åäœã«å¿
èŠãªéããã AWS ãµãŒãã¹ (Amazon S3ãAmazon ECRãAmazon CloudWatch) ã«ã¯å°éã§ããŸãããã»ãã® AWS ãµãŒãã¹ãå€éšãšã³ããã€ã³ãã¯åŒã³åºããŸãããã¯ãŒã¯ãããŒã§å€éš API ã®åŒã³åºããå¿
èŠãªå Žåã¯ãäºåã«ããŒã¿ãåŠçã㊠Amazon S3 ãã±ããã«ä¿åãããã®ããã§ã¯ãŒã¯ãããŒãå®è¡ããŠãã ããã ã©ã³ã¿ã€ã ã®äŸåé¢ä¿ â boto3 ãš Python æšæºã©ã€ãã©ãªã¯ããªã€ã³ã¹ããŒã«æžã¿ã§ããpandas ã requests ãªã©ã®è¿œå ããã±ãŒãžã¯ã Amazon MWAA Serverless ã®ããã±ãŒãžã³ã°ã¬ã€ãã©ã€ã³ ã«åŸã£ãŠ ZIP ã¢ãŒã«ã€ãã«ãã³ãã«ããŠãã ããã å®è¡ã¿ã€ã ã¢ãŠã â ã¿ã¹ã¯ã¯ã¯ãŒã¯ãããŒã«èšå®ãããã¿ã€ã ã¢ãŠãå¶éã«åŸããŸãã Python ã®ããŒãžã§ã³ â çŸåšãµããŒããããŠãã Python ã©ã³ã¿ã€ã ã®ããŒãžã§ã³ã¯ Amazon MWAA Serverless ã®ããã¥ã¡ã³ã ã§ç¢ºèªããŠãã ããã DAG ã®åœ¢åŒ â MWAA Serverless ã¯åŸæ¥ã® Python ã® DAG ãã¡ã€ã«ã§ã¯ãªããYAML ããŒã¹ã® DAG å®çŸ©ã䜿ããŸããMWAA Provisioned ããç§»è¡ããå Žåã¯ãDAG ã YAML 圢åŒã«å€æããå¿
èŠããããŸãã ãµããŒããããªããªãã¬ãŒã¿ãŒ â Airflow ã³ãã¥ããã£ã®ãªãã¬ãŒã¿ãŒãã«ã¹ã¿ã ãã©ã°ã€ã³ã®äžéšã¯ Serverless ã©ã³ã¿ã€ã ã§ã¯å©çšã§ããŸãããäºææ§ã®äžèŠ§ã¯ ããã¥ã¡ã³ã ãåç
§ããŠãã ããã ã¯ãªãŒã³ã¢ãã ç¶ç¶çãªèª²éãé¿ãããããæ¬èšäºã®ãŠã©ãŒã¯ã¹ã«ãŒã§äœæãããªãœãŒã¹ãåé€ããŸããã¯ãŒã¯ãããŒãS3 ãªããžã§ã¯ããIAM ããŒã«ã¯æ¬¡ã®ã³ãã³ãã§åé€ã§ããŸãã æ³š: $WORKFLOW_ARN ã¯ã¹ããã 7 ã§å®çŸ©ããŠããŸãã # Delete the workflow aws mwaa-serverless delete-workflow \ --workflow-arn $WORKFLOW_ARN \ --region $REGION 泚: $BUCKET ã¯ã¹ããã 4 ã§ãšã¯ã¹ããŒãããŠããŸããå¿
èŠã«å¿ããŠãã±ãããåé€ããŠãã ããã # Remove S3 objects aws s3 rm s3://$BUCKET/code/code_bundle.zip aws s3 rm s3://$BUCKET/dags/conversion_dag.yaml aws s3 rm s3://$BUCKET/raw/sales_data.csv aws s3 rm s3://$BUCKET/processed/sales_data.json # Delete the IAM role aws iam delete-role-policy \ --role-name MWAAServerlessExecutionRole \ --policy-name MWAAServerlessAccessPolicy aws iam delete-role --role-name MWAAServerlessExecutionRole ãŸãšã PythonOperator ãš BashOperator ããã€ãã£ãã«ãµããŒããããããšã§ãå€ãã®ããŒã¿ãšã³ãžãã¢ãªã³ã°ããŒã ãæ¥åžžçã«äœ¿ã£ãŠããã«ã¹ã¿ã ã³ãŒãã®å®è¡ãã¿ãŒã³ããMWAA Serverless ã§çŽæ¥äœ¿ããŸããããŒã¿å€æã圢åŒå€æãæ€èšŒãã·ã§ã«ã¹ã¯ãªããããã³ã³ãã¥ãŒãã£ã³ã°ãªãœãŒã¹ã®è¿œå ããããžã§ãã³ã°ãã³ã³ããã®ç®¡çãªãã«ãµãŒããŒã¬ã¹ã©ã³ã¿ã€ã ã§å®è¡ã§ããŸãã MWAA Provisioned ãã»ã«ããããŒãžãã®ã€ã³ãã©ã¹ãã©ã¯ãã£ã§ Airflow ã¯ãŒã¯ããŒããå®è¡ããŠããå Žåãæ¢åã® PythonOperator ãš BashOperator ã®ããžãã¯ã¯ã»ãšãã©å€æŽããã«äœ¿ããŸããPython ã® DAG ãã¡ã€ã«ã YAML 圢åŒã«å€æããã³ãŒãããã³ãã«ãšããŠããã±ãŒãžåããã°ãMWAA Serverless ã§å®è¡ã§ããŸãã ãŸã㯠Amazon MWAA Serverless ã®ããã¥ã¡ã³ã ãåç
§ããæ¬èšäºã®ãŠã©ãŒã¯ã¹ã«ãŒãèªåã®ããŒã¿ã§è©ŠããŠãã ãããæéã®è©³çŽ°ã¯ Amazon MWAA ã®æéããŒãž ãåç
§ããŠãã ããããã£ãŒãããã¯ããåŸ
ã¡ããŠããŸãã èè
ã«ã€ã㊠Pradeep Kumar Nalluri AWS ã®ãœãããŠã§ã¢éçºãšã³ãžãã¢ã§ãã¹ã±ãŒã©ãã«ãªã¢ããªã±ãŒã·ã§ã³ã®èšèšãšéçºãå°éãšããŠããŸããäŒæ¥ã¯ãã¬ãçªçµãæ ç»ã芳ãŠéãããŠããŸãã Karthik Seshadri AWS ã®ã·ãã¢ãœãããŠã§ã¢éçºãšã³ãžãã¢ã§ãããã°ããŒã¿æè¡ã®ãªãŒã±ã¹ãã¬ãŒã·ã§ã³ãå°éãšããŠããŸãããµãŒããŒã¬ã¹æè¡ãããŒã¿ãšã³ãžãã¢ãªã³ã°ãã¹ã±ãŒã©ãã«ãªãµãŒãã¹ã®æ§ç¯ã«æ
ç±ã泚ãã§ããŸããä»äºä»¥å€ã§ã¯ãæ
è¡ãããŸããŸãªã¹ããŒããæ¥œããã§ããŸãã Aritra Ghosh Amazon Web Services (AWS) ã®ã·ãã¢ãããã¯ããããŒãžã£ãŒã§ãAmazon Managed Workflows for Apache Airflow (Amazon MWAA) ãš Amazon SageMaker Unified Studio ã®è£œåéçºãçããŠããŸããä»äºä»¥å€ã§ã¯ãã¹ã«ãã·ã¥ãšãžã éããæ¥œããã§ããŸãã Sriram Ramarathnam AWS Analytics ã§ AWS GlueãAWS Data PipelineãManaged Serverless Airflow ãæ
åœãããœãããŠã§ã¢éçºãããŒãžã£ãŒã§ããããŒã ã§ã¯ããµãŒããŒã¬ã¹ãšããããžã§ã³ãäž¡æ¹ã®ã³ã³ãã¥ãŒãã£ã³ã°æäŸåœ¢æ
ã«ãŸããããªãŒã±ã¹ãã¬ãŒã·ã§ã³é åã®é£ãã課é¡ã«åãçµãã§ããŸãã