ã¯ããã« ããã«ã¡ã¯ïŒData PlatformããŒã ã§ããŒã¿ãšã³ãžãã¢ãšããŠåããŠãã @shota.imazeki ã§ãã åŒããŒã ã§ã¯ãåŸæ¥ã®åæåºç€ã段éçã«å·æ°ããåãçµã¿ãé²ããŠããããã®ç¬¬äžæ©ãšããŠãECSäžã§åãããŠããAirflowãAWSäžã®ãããŒãžããµãŒãã¹ã§ãã Amazon Managed Workflows for Apache AirflowïŒä»¥äžãMWAAïŒã«ç§»è¡ããŸããã ããšããšã¯ã€ã³ãã©ç®¡çã®æéãæžããããšãç®çã§ããããçµæãšããŠãããåŠçæéã倧å¹
ã«ççž®ããããšããæå€ãªå¹æãåŸãããŸããã ãã®èšäºã§ã¯ãECSäžã®AirflowããMWAAãžã®ç§»è¡ã«è³ã£ãèæ¯ãã工倫ãããã€ã³ããåŸãããæ¹å广ãªã©ã玹ä»ããŠãããŸãã ç§»è¡ã«è³ã£ãèæ¯ ãããŸã§BASEã§ã¯ECSäžã§Apache Airflow v1ãéçšããŠããŸããããéçšè² è·ãé«ããã€ã³ãã©åšãã®ç®¡çã«ã¯å¥ããŒã ã®æ¯æŽããé¡ãããããšããããŸããããŸãECSçšã«çšŒåããŠããRDSã®ã¹ãã¬ãŒãžå®¹éãéŒè¿«ãã€ã€ãã£ãããšã倧ããªèª²é¡ã®äžã€ã§ããã ãŸããAirflow v1ã§ã¯ãµããŒããããŠããªããªãã¬ãŒã¿ãŒãæ©èœããããã¯ãŒã¯ãããŒã®èšèšã»å®è£
ã«å¶éã課ãããŠããŸãããåŸæ¥ã®åæåºç€ãå·æ°ããŠããã«ãããããããã®å¶çŽã¯å€§ããªéå£ã«ãªããšå€æããåºç€ã®äžæ žãæ
ããªãŒã±ã¹ãã¬ãŒã·ã§ã³ããŒã«ã§ããAirflowããçæããããšã«ããŸããã ãã®çµæãAirflowã®ã¡ãžã£ãŒããŒãžã§ã³ã¢ããããŒãã«ããããŠãã€ã³ãã©ç®¡çã®è² è·ã軜æžã§ãããããŒãžããµãŒãã¹ã§ããMWAAãžã®ç§»è¡ã決æããŸããã ç°å¢æ§ç¯ãšç§»è¡æºå æåã«MWAAã®å
¬åŒããã¥ã¡ã³ããåèã«ããªããç°å¢æ§ç¯ãè¡ããŸãããäºåã«å¿
èŠãªãã®ãšããŠã¯ãããŒãžã§ãã³ã°ãæå¹åãããS3ãã±ããã®ã¿ã§ããã以å€ã®VPCãã»ãã¥ãªãã£ã°ã«ãŒããIAMããŒã«ã«ã€ããŠã¯MWAAã®ãããžã¡ã³ãã³ã³ãœãŒã«äžã§äœæããŸããã åè: https://docs.aws.amazon.com/ja_jp/mwaa/latest/userguide/get-started.html ãã®ç°å¢æ§ç¯ã®æ®µéã§å·¥å€«ããããšãèºããç¹ããã£ããããæ¬¡ã«ãããã«ã€ããŠè§ŠããŠãããŸãã 1. SSHéµãAPIããŒã¯ã³ãªã©ã®æ©å¯æ
å ±ãSecrets Managerã§ç®¡çãã ECSã§éçšããŠããéã¯ãAirflowäžã®ç°å¢å€æ°ãAirflow Variable, Connectionã«SSHéµãAPIããŒã¯ã³ãªã©ãèšå®ããŠç®¡çããŠãŸããããã ããã©ãã«ã©ã®æ
å ±ã眮ããã®åºæºãææ§ã«ãªã£ãŠããã管çãç
©éåããŠãããããç§»è¡ã«åãããŠç®¡çæ¹æ³ãèŠçŽããŸããã MWAAã§ã¯ S3ã«ãã¡ã€ã«ãšããŠé
眮ããŠåç
§ããæ¹æ³ ããããŸããã以äžã®èгç¹ããSecrets Managerã«çµ±äžããŸããã ã»ãã¥ãªãã£ç®¡çã®èŠ³ç¹ æ
å ±æŽæ°æã®äœæ¥è² è·è»œæžãšå®å
šæ§åäž MWAAãžã®ç§»è¡ã«äŒŽããæ©å¯æ§ã®é«ãæ
å ±ã¯å
šãŠSecrets Managerã«ä¿åããAirflowããçŽæ¥åç
§ããèšèš ã«åãæ¿ããŸããã MWAAã®Airflowèšå®ãªãã·ã§ã³ MWAAç°å¢æ§ç¯æã«ãAirflowèšå®ãªãã·ã§ã³ã§[ã«ã¹ã¿ã èšå®ã远å ]ãéžæãã以äžã®ããŒãšå€ã®ãã¢ã远å ããŸãã secrets . backend : airflow . providers . amazon . aws . secrets .secrets_manager. SecretsManagerBackend secrets .backend_kwargs: { " connections_prefix " : " airflow/connections ", " variables_prefix " : " airflow/variables " } MWAAããSecrets Managerãžã®ã¢ã¯ã»ã¹èšå® MWAAããSecrets Managerã«å®å
šã«ã¢ã¯ã»ã¹ã§ãããããIAMããŒã«ãšããªã·ãŒãèšå®ããŸãããã»ãã¥ãªãã£ãèæ
®ããŠã airflow/* ãšããååã® Secretsã®ã¿ã¢ã¯ã»ã¹ã§ããããã«ã¹ã³ãŒããçµã£ãŠããŸãã åè: https://docs.aws.amazon.com/ja_jp/mwaa/latest/userguide/mwaa-create-role.html#mwaa-create-role-attach-json-policy { "Version" : "2012-10-17" , "Statement" : [ { "Effect" : "Allow" , "Action" : [ "secretsmanager:GetResourcePolicy" , "secretsmanager:GetSecretValue" , "secretsmanager:DescribeSecret" , "secretsmanager:ListSecretVersionIds" ], "Resource" : "arn:aws:secretsmanager:{ãªãŒãžã§ã³}:{ã¢ã«ãŠã³ãID}:secret:airflow/*" }, { "Effect" : "Allow" , "Action" : "secretsmanager:ListSecrets" , "Resource" : "*" } ] } Secrets ManageråŽã®èšå® Secrets Managerã«ãŠã以äžã®ããã«ååãä»ããŠæ
å ±ãä¿åããŸããè€æ°ã®ãã©ã¡ãŒã¿ããŸãšãããå Žåã¯JSON圢åŒã«ããŠãããšãAirflowåŽã§åå²ããŠååŸããå¿
èŠããªã䟿å©ã§ããã airflow/connections/SSH_CONNECTION { "conn_id" : "SSH_CONNECTION" , "conn_type" : "ssh" , "host" : "192.0.2.0" , "login" : "centos" , "port" : 22 , "extra" : { "private_key" : "-----BEGIN RSA PRIVATE KEY----- \n <ç§å¯éµã®å
容ã¯äŒããŠããŸã> \n -----END RSA PRIVATE KEY-----" } } airflow/connections/GCP_CONNECTION { " conn_type ": " google_cloud_platform ", " extra ": { " project ": " sample-project ", " keyfile_dict ": { " type ": " service_account ", " project_id ": " sample-project ", " private_key_id ": " abc123def456 ", " private_key ": " -----BEGIN PRIVATE KEY----- \n <ç§å¯éµã®å
容ã¯äŒããŠããŸã> \n -----END PRIVATE KEY----- \n ", " client_email ": " sample-sa@example.com ", " client_id ": " 1234567890 ", " auth_uri ": " https://accounts.google.com/o/oauth2/auth ", " token_uri ": " https://oauth2.googleapis.com/token ", " auth_provider_x509_cert_url ": " https://www.googleapis.com/oauth2/v1/certs ", " client_x509_cert_url ": " https://www.googleapis.com/robot/v1/metadata/x509/sample-sa%40sample-project.iam.gserviceaccount.com " } } } airflow/variables/API_TOKEN abcdefg1234567890 DAGããã®å©çšäŸ ãã®åŸãDAGãã¡ã€ã«äžã§ä»¥äžã®ããã«ã³ãŒããæžãããšã§SSHéµãAPI_TOKENãå©çšããããšãå¯èœã«ãªããŸãã from airflow.providers.ssh.operators.ssh import SSHOperator from airflow.providers.google.cloud.operators.bigquery import BigQueryExecuteQueryOperator from airflow.models import Variable # Secrets Manager ã«ä¿åããã SSH Connection ãå©çš ssh_task = SSHOperator( task_id= 'ssh_task' , ssh_conn_id= 'SSH_CONNECTION' , command= "echo 'Hello MWAA!'" , dag=dag, ) bq_task = BigQueryExecuteQueryOperator( task_id= 'bq_task' , sql= "select 'Hello MWAA!'" , use_legacy_sql= False , location= 'asia-northeast1' , gcp_conn_id= 'GCP_CONNECTION' , dag=dag, ) # Secrets Manager ã«ä¿åããã API ããŒã¯ã³ãååŸ token = Variable.get( 'API_TOKEN' ) ããã«ãããåŸæ¥ã®ç°å¢å€æ°ãAirflow ConnectionãS3ã«ãããã¡ã€ã«ç®¡çãšæ¯èŒããŠãããæç¢ºãã€å®å
šã«æ©å¯æ
å ±ãäžå
管çã§ããããã«ãªããŸããã 2. ãã©ã«ãæ§æã®æŽç MWAAã§ã¯DAGãã¡ã€ã«ä»¥å€ã«ãã以äžã®ãããªå
±éãªãœãŒã¹ãå©çšãããããdagsãã©ã«ãé
äžã«çšéããšã«æŽçããŸããã dags/ âââ sample_dag1.py # DAGãã¡ã€ã« âââ sample_dag2.py âââ plugins/ # ã«ã¹ã¿ã ãªãã¬ãŒã¿ãŒãããã¯ãªã© â âââ custom_operator.py âââ sql/ # ã¯ãšãªãã¡ã€ã« â âââ sample_query.sql âââ common/ # å
±é颿°ããŠãŒãã£ãªã㣠âââ utils.py MWAAã®S3ãã±ããã§ã¯ãdagsãã©ã«ãçŽäžãPythonã¢ãžã¥ãŒã«ãã¹ãšããŠèªèããããããDAGãã¡ã€ã«ãšåãéå±€é
äž( dags/ )ã«å
±éãªãœãŒã¹ããã¹ãŠãŸãšããŠããŸãã ãã®ãããDAGãã¡ã€ã«å
ã§ã¯ä»¥äžã®ããã«ã·ã³ãã«ã«ã€ã³ããŒãã§ããŸãã from common.utils import some_function from plugins.custom_operator import CustomOperator SQLãã¡ã€ã«ã®å©ç𿹿³ Airflowã§å©çšããã¯ãšãªã¯ä»¥åãã .sql ãã¡ã€ã«ã§ç®¡çããŠããããã®éçšãMWAAã§ãç¶ç¶ããŠããŸããDAGã®template_searchpathã«sqlãã©ã«ããèšå®ããããšã§BigQueryExecuteQueryOperatorã«ãã¡ã€ã«åãæå®ããã°ã¯ãšãªãå®è¡ã§ããŸãã from airflow.providers.google.cloud.operators.bigquery import BigQueryExecuteQueryOperator # DAGãªããžã§ã¯ãã®äœæïŒè©³çްã¯çç¥ãSQLãã¡ã€ã«ã®æ€çŽ¢ãã¹ã®ã¿èšèŒïŒ dag = DAG( dag_id= 'base_dwh' , template_searchpath=[ '/usr/local/airflow/dags/sql' ], ) bq_task = BigQueryExecuteQueryOperator( task_id= 'bq_task' , sql= 'sample_query.sql' , # sqlãã£ã¬ã¯ããªé
äžã®ãã¡ã€ã« use_legacy_sql= False , location= 'asia-northeast1' , gcp_conn_id= 'GCP_CONNECTION' , dag=dag, ) ãã¡ããJinjaãã³ãã¬ãŒãã«ã察å¿ããŠãããDAGå®è¡æã«åçã«å€ãåã蟌ãããšãå¯èœã§ãã以äžã¯ãAirflowã®çµã¿èŸŒã¿ãã¯ã( {{ ds }} )ãšãDAGåŽãã params ã䜿ã£ãŠæž¡ããå€ã®äž¡æ¹ã SQLãã¡ã€ã«å
ã§å©çšããäŸã§ãã -- sample_query.sql SELECT column1, column2 FROM `sample_dataset.sample_table` WHERE DATE (column_date) = ' {{ ds }} ' AND category = ' {{ params.category }} ' DAGåŽã§ã¯ã params ã«å€ãæž¡ãããšã§ãSQLãã¡ã€ã«å
ã®Jinjaãã³ãã¬ãŒããå±éãããŸãã from airflow.providers.google.cloud.operators.bigquery import BigQueryExecuteQueryOperator bq_task = BigQueryExecuteQueryOperator( task_id= 'bq_task' , sql= 'sample_query.sql' , use_legacy_sql= False , location= 'asia-northeast1' , gcp_conn_id= 'GCP_CONNECTION' , params={ 'category' : 'category1' }, # params ã§ category ãæå®ããäŸ dag=dag, ) 3. GitHub Actions ã§ MWAA ãžã®èªåããã〠ECSæä»£ãCircleCIã䜿ã£ãŠCI/CDã«ãããããã€ãè¡ã£ãŠããŸããããMWAAã§ã¯S3ã«ãã¡ã€ã«ãé
眮ããæ¹åŒã«ãªããããç§»è¡ã«ããããŠGitHub Actionsã«åãæ¿ãããããã€æ¹æ³ãèŠçŽããŸããã æ§ç¯ããå
容 GitHub Actionsã«ãã£ãŠä»¥äžãèªååããŸããã mainãã©ã³ããžã®ããŒãžæã«ãDAGãã¡ã€ã«ã»pluginsã»å
±éåŠçã»SQLãã¡ã€ã«ãªã©ã«å€æŽãããã°ãS3ã«ã¢ããããŒãããã 以äžã¯ãã®GitHub Actionsã®ãµã³ãã«ã§ãã name : Deploy MWAA DAGs on : push : branches : - main paths : - 'dags/**' jobs : deploy : runs-on : ubuntu-latest steps : - name : Checkout repository uses : actions/checkout@v4 - name : Configure AWS credentials uses : aws-actions/configure-aws-credentials@v2 with : aws-region : ap-northeast-1 # 䜿çšããŠãã AWS ãªãŒãžã§ã³ aws-access-key-id : ${{ secrets.AWS_ACCESS_KEY_ID }} aws-secret-access-key : ${{ secrets.AWS_SECRET_ACCESS_KEY }} - name : Sync DAGs to S3 run : aws s3 sync dags/ s3://sample-airflow-dag-bucket/dags/ --delete - name : List uploaded DAGs run : aws s3 ls s3://sample-airflow-dag-bucket/dags/ ãªãã requirements.txt ãæŽæ°ãããå Žåã¯MWAAç°å¢ã®æŽæ°ãå¿
èŠã§ããçŸåšã¯æåã§å¯Ÿå¿ããŠãããã®ã®ãä»åŸã¯GitHub Actionsã«ããèªååæ ãæ€èšããŠããŸãã ãã ãç°å¢æŽæ°äžã¯äžæçã«MWAAã忢ããããããããåŠçãžã®åœ±é¿ããªãã¿ã€ãã³ã°ã§å®è¡ããå¿
èŠãããããã®ãããã¯æ
éã«æ€èšããŠãããããšèããŠããŸãã 4. requirements.txt ã§çºçããåé¡ãšå¯ŸåŠ MWAAã§ãå€éšAPIãå©çšããããã«ãrequirements.txtã«ã©ã€ãã©ãªãèšèŒããŠã€ã³ã¹ããŒã«ããŠããŸããECSæä»£ããå©çšããŠããgoogle-ads, facebook-businessãªã©ã®åºåç³»APIçšã©ã€ãã©ãªãåŒãç¶ã䜿çšããŠããŸããã çºçããåé¡ MWAAã§requirements.txtãæŽæ°ããéãæ¬¡ã®ãããªåé¡ãçºçããŸããã äžéšã®ã©ã€ãã©ãª(google-ads, facebook-businessãªã©)ã®ã€ã³ã¹ããŒã«æã«ãšã©ãŒãçºç MWAAã¯pip installã1ã€ã§ã倱æããå Žåãå
šãŠã®ã€ã³ã¹ããŒã«ãäžæããã(All or Nothing)仿§ ãã®ãããå
šãŠã®DAGã§å¿
é ãªapache-airflow-providers-*ç³»ã©ã€ãã©ãªãã€ã³ã¹ããŒã«ããããDAGãåäœããªããªã åå Airflowå
¬åŒãæäŸããŠããconstraintsãã¡ã€ã«ãš requirements.txt ã«èšèŒããããŒãžã§ã³æå®ãè¡çªããŠããŸãããããšãã°ãMWAA v2.8.1(Python 3.10)ã®å Žåã以äžã®constraintsãã¡ã€ã«ãé©çšãããŸãã https://raw.githubusercontent.com/apache/airflow/constraints-2.8.1/constraints-3.10.txt ãã®constraintsãã¡ã€ã«ã«å«ãŸããããŒãžã§ã³ãšç°ãªãããŒãžã§ã³(google-ads==22.0.0 ãªã©)ãæç€ºçã«æå®ããŠããŸããšãããŒãžã§ã³è¡çªã«ãã£ãŠã€ã³ã¹ããŒã«ãšã©ãŒ â å
šãŠã®ã€ã³ã¹ããŒã«ã倱æãšããããšã«ãªããŸãã åè: https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#constraints-files 察å¿ç ããŒã«ã«ç°å¢ã§Airflowã®constraintsãé©çšããç¶æ
ã§ pip install ãæ€èšŒããããã«ããŸããã # ä»®æ³ç°å¢ãäœæ rm -rf venv python -m venv venv source venv/bin/activate # pip / setuptools / wheel ãææ°å pip install --upgrade pip setuptools wheel # constraints ãé©çšããŠã€ã³ã¹ããŒã« pip install -r requirements.txt \ -c https://raw.githubusercontent.com/apache/airflow/constraints-2.10.3/constraints-3.10.txt ç¹ã«ä»¥äžã«æ³šæããŠããŸãã MWAAã®ããŒãžã§ã³ããšã«å¯Ÿå¿ããconstraintsãã¡ã€ã«ãé©çšããããš ã€ã³ã¹ããŒã«ãããã©ã€ãã©ãªãconstraintsãã¡ã€ã«ã§å¶çŽãããŠããã確èªããããš å¿
ãããŒã«ã«ã§ã€ã³ã¹ããŒã«ãæåããããšã確èªããŠãããMWAAã«åæ ããããš MWAAãžã®ç§»è¡äœæ¥ ãããã€ãSecrets管çãªã©ã®ä»çµã¿ãæŽã£ãã®ã§ãå®éã®ç§»è¡äœæ¥èªäœã¯æ¯èŒçã·ã³ãã«ã§ããã ç§»è¡ã®é²ãæ¹ DAGããšã«ç¬ç«ããŠãããããå
šãŠãäžåºŠã«åãæ¿ããã®ã§ã¯ãªã1ã€ãã€é ã«ç§»è¡ã»åäœç¢ºèª TriggerDagRunOperatorãçšããDAGã®ã¿ããŸãšããŠåãæ¿ã 圱é¿ç¯å²ã®å°ããDAGããé ã«ç§»è¡ããããšã§ãªã¹ã¯ãæå¶ v1ã®æã¯å®è£
äžã®éœåããBigQueryã®ã¯ãšãªå®è¡ãSlackéç¥ãªã©ã§èªäœããŠããOperator / Hookã以äžã®èгç¹ããAirflowå
¬åŒã®Providerã«çœ®ãæã ä¿å®ã»ã¡ã³ããã³ã¹ã³ã¹ãã®åæž AirflowããŒãžã§ã³ã¢ããæã®äºææ§ç¢ºä¿ Providerããšã«çްããªæ¹åã»ãã°ä¿®æ£ãåã蟌ãŸããŠãã ãã®ä» MWAAã®åäœç°å¢(PythonããŒãžã§ã³ãAirflowããŒãžã§ã³)ã«åãããŠãã³ãŒãã®åŸ®ä¿®æ£ãè¡ããŸããã æ©å¯æ
å ±ã®ç®¡çãDAGé
çœ®å Žæã®å€æŽãäºåã«è¡ã£ãŠããããããã®éšåã®ã³ãŒãã®åŸ®ä¿®æ£ã®ã¿ã§DAGæ¬äœã®ç§»è¡äœæ¥èªäœã¯ã¹ã ãŒãºã«å®äºããŸããã ç§»è¡ã«ãã£ãŠåŸãããæå€ãªå¹æ MWAAãžã®ç§»è¡ãé²ããäž»ç®çã¯ãã€ã³ãã©ç®¡çã®æéåæžãã§ããããæãã¬æ§èœæ¹åãšãã广ãåŸãããŸããã åŠçæéã®æ¹å ããšããšæ¥æ¬¡ããããšããŠå®è¡ããŠããããŒã¿é£æºåŠçã§ã¯ã1æ¥ãããçŽ14æéã®åŠçæéãããã£ãŠããŸãããMWAAã«ç§»è¡ããçµæãçŽ9æéã§å®äºããããã«ãªãã4ã5æéã»ã©ççž®ãããŸããã 詳现ã«èŠãŠãããšãECSåŽã§ã¯åã¿ã¹ã¯ã®éå§ééã«ãçŽ30ç§ã®é
å»¶ããããæ¥æ¬¡ãããå
šäœã§ã¯çŽ500ã¿ã¹ã¯ à 30ç§ = çŽ4.2æéã®åŸ
æ©æéãçºçããŠããŸãããMWAAã§ã¯ãã®åŸ
æ©æéãã»ãŒãªããªã£ãŠãããããã«ãã£ãŠåŠçæéãæ¹åããããšèããŠããŸãã åå ã®èå¯ ããã«èª¿æ»ãããšãããECSã®æ¹ã§ã¯ExecuterãSequentialExecutor(鿬¡å®è¡)ã«ãªã£ãŠããŸããããMWAAã§ã¯ããã©ã«ãã§CeleryExecutor(䞊åå®è¡)ã䜿çšãããŠããããã§ããã ECSæä»£ã¯ã·ã³ãã«ããåªå
ã㊠SequentialExecutor ã䜿çšããŠãããã¿ã¹ã¯ã¯1ã€ãã€å®è¡ããŠããŸãããäžæ¹ãMWAAã§ã¯CeleryExecutor(ãŸãã¯KubernetesExecutor)ãããã©ã«ãã§ãã¿ã¹ã¯ãäŸåé¢ä¿ã«å¿ããŠè€æ°åæã«å®è¡ãããæ§æãšãªã£ãŠããŸããããã«ãããç¬ç«ããŠåäœå¯èœãªã¿ã¹ã¯ã¯èªåçã«äžŠåå®è¡ãããã¿ã¹ã¯éã®åŸ
æ©æéãè§£æ¶ããããããå
šäœã®åŠçæéã倧å¹
ã«ççž®ããããšèããŠããŸãã ãªããECSæä»£ã§ãExecutorã®èšå®ã倿Žããã°äžŠåå®è¡ã«ããæ¹åã¯å¯èœã ã£ããããããŸããããã ãããããã調æŽãã¡ã³ããã³ã¹ãèªåã§è¡ãããšãªãããããŒãžãç°å¢ãããã©ã«ãã§æé©ãªæ§æã«ããŠãããã®ããMWAAãéžãã 倧ããªã¡ãªããã ãšæããŸããã 圱é¿ç¯å² ã¿ã¹ã¯ã䞊åå®è¡ãããããšã«ãã圱é¿ããªããã以äžã®èгç¹ã§ç¢ºèªããŸããã å
ããŒã¿ããŒã¹ãžã®åœ±é¿ã«ã€ã㊠MWAAãžã®ç§»è¡ã«ãã£ãŠåŠçã®äžŠååãé²ã¿ãŸããããETLããŒã«ãšããŠã¡ã€ã³ã§å©çšããŠããEmbulkèªäœã¯é次çã«å®è¡ãããããã«ãªã£ãŠãããããå
ããŒã¿ããŒã¹ãžã®è² è·ã䞊åå®è¡ã§æ¥å¢ããããšã¯ãããŸããã§ãããåçŽã«åŸ
æ©æéãæžã£ãã®ã¿ã§ãç§»è¡åŸãå
DBãžã®è² è·å¢å ãå¿é
ããããšãªããå®å®ããŠéçšã§ããŠããŸãã 䞊åå®è¡ã«ããã¿ã¹ã¯éã®åœ±é¿ åãããŒãã«ãåããã¡ã€ã«ãžã®åºåãªã©ãDAGã®ã¿ã¹ã¯å
ã§è¡ãªã£ãŠããããã¿ã¹ã¯ã®äŸåé¢ä¿(task1 >> task2)ãDAGå
ã§æç€ºããŠããããã«åæå®è¡ããªãããŠãåé¡ã«ã¯ãªããŸããã§ãããä»åŸãExecutorã«äŸåããªã(䞊åå®è¡ãããŠãåé¡ãªã)å
ç¢ãªDAGèšèšãç¶ç¶ããŠãããŸãã ä»åŸã®æ¹åç¹ 1. DAGsãã©ã«ãçŽäžã®ãã¡ã€ã«æŽç çŸåšã dags ãã©ã«ãçŽäžã«DAGãã¡ã€ã«ã ãã§ãªã plugins/ , sql/ , common/ ãšãã£ãè€æ°ã®ãã©ã«ããæ··åšããŠããç¶æ
ã§ãã å°æ¥çã«ã¯ãDAGãã¡ã€ã«ãšå
±éãªãœãŒã¹ãããåãããããæŽçãã管çã»ã¡ã³ããã³ã¹ããããæ§æã«æ¹åããããšèããŠããŸãã 2. Terraformã«ããã€ã³ãã©æ§æç®¡ç MWAAç°å¢ãSecrets ManagerãS3ãã±ãããªã©ã®AWSãªãœãŒã¹ãTerraformã§ç®¡çããããšã§ãç°å¢æ§ç¯ã倿Žã®åçŸæ§ãé«ããéçšè² è·ã®è»œæžãå³ãããã§ãã çŸç¶ã¯æåã§èšå®ããŠããéšåãå€ããããIaCåã«ããæšæºåã»èªååãæãŸããŸãã ãããã« Airflowã®ç§»è¡ã«ãã£ãŠãéçšè² è·ã®è»œæžã ãã§ãªããæãã¬æ§èœæ¹åãåŸãããšãã§ããŸããã ä»åŸãåºç€å
šäœã®æ¹åãé²ããªãããããå®å®ããããŒã¿é£æºåºç€ãç®æããŠãããŸãã æåŸãšãªããŸãããåŒç€Ÿã§ã¯ããŒã¿ãšã³ãžãã¢ãåéããŠããŸããäžèšã§è¿°ã¹ã課é¡ä»¥å€ã«ãBASEã®åæåºç€ã«ã¯å€ãã®èª²é¡ããã£ãŠããšãŠãããããã®ããä»äºããªãšæã£ãŠãããŸãããèå³ã®ããæ¹ã¯æ°è»œã«ãå¿åãã ããïŒ open.talentio.com