ã¯ããã« ããã«ã¡ã¯ãXIæ¬éšãšã³ãžãã¢ãªã³ã°ãã¯ãã ãžãŒ ã»ã³ã¿ãŒã®åŸ³å±±åºå£«ã§ãã ãã®èšäºã§ã¯ãããŒã¿åºç€ïŒããŒã¿åæåºç€ã«ãããããŒã¿ãã€ãã©ã€ã³ã®AIé§åéçºã®ææ³ãææ¡ããŸãã ã ODCS (Open Data Contract Standard) ããã©ãŒãããã®"Data Contract"ã仿§æžãšããŠçæAIãžäžããŠããŒã¿ãã€ãã©ã€ã³ã®ã³ãŒããçæãããææ³ã§ãã Data Contractã¯ãããŒã¿ã®ä»æ§ïŒããŒãã«æ§é ãã«ã©ã å®çŸ©ãããŒã¿åãªã©ïŒãšå質èŠä»¶ã YAML 圢åŒã§å®çŸ©ããæšæºãã©ãŒãããã§ãã åœèšäºã®äž»ãªå
容ã¯ãData ContractãããŒã«ã«éçºç°å¢ãªã©ã®åºç€çãªèª¬æããåœä»¶ã®AIé§åéçºæ¹æ³ãã³ãŒãçæã«æåããæ€èšŒçµæã«ã€ããŠã§ãã æ³å®èªè
: ããŒã¿ãšã³ãžãã¢ãããŒã¿ãããžã¡ã³ãæ¥åæ
åœè
åœèšäºã§åŸãããç¥èŠ: ããŒã¿ãã€ãã©ã€ã³ã®AIé§åéçºã®åæ åæç¥è: ããŒã¿ãã€ãã©ã€ã³ïŒETL/ ELT åŠçïŒã®åºç€ç¥è äžè¬çã«åŽåãšæéã®ãããããŒã¿ãããžã¡ã³ãã»ãµã€ã¯ã«ãAIé§åéçºã«ãã£ãŠå éã§ããã°ãšèããŠãããŒã¿ãšã³ãžãã¢ãªã³ã°ã®AIé§åéçºææ³ãæ¢æ±ãããã®èšäºã«ãŸãšããŸããã ããŒã¿ãã€ãã©ã€ã³ã®AIé§åéçºå®çŸã®èª²é¡ çæAIã«äœãäŒããã¹ãã ã¢ããªã±ãŒã·ã§ã³ã®AIé§åéçºã«ãããŠã¯ã ãã¡ã€ã³ ã ãŠãŒã¹ã±ãŒã¹ ãæ©èœãå¶çŽãªã©ãçæAIã«èª¬æããããšã§ã³ãŒãçæããã¢ãããŒããèŠèãããŸããã§ã¯ã ããŒã¿ãšã³ãžãã¢ãªã³ã°ã«ãããŠã¯äœã説æããã°è¯ãã®ãïŒã©ã®ããã«æç€ºãåºãã°å
æ¬çã«éçºããŠãããã®ãïŒ ãããæåã«çŽé¢ãã課é¡ã§ããã èªç¶èšèª ã®ããã³ããã§ç®çã® SQL ã¯ãšãªã«ã€ããŠèª¬æããæ¹æ³ãèããŸããããè€éãªããžãã¯ã®è¡šçŸãå°é£ã§ããã现ããèŠä»¶ãäŒããããããã³ããã®åçŸæ§ã»åå©çšæ§ã«ãçåããããŸããã ã¢ãããŒã: ããŒã¿ ã¢ãŒããã¯ã㣠ãšããŒã¿ã¢ãã«ãäŒãã ãã®èª²é¡ã«å¯ŸããŠãããŒã¿ãšã³ãžãã¢ãªã³ã°ã®èгç¹ãããããŒã¿ ã¢ãŒããã¯ã㣠ããšãããŒã¿ã¢ãã«ãã®2ã€ã®è»žã§æ
å ±ãæŽçããããŒã¿åºç€ïŒããŒã¿åæåºç€ã®å
šäœåãšéçºæ¹æ³ãçæAIã«äŒããã¢ãããŒããæ¡çšããŸããã ããŒã¿ ã¢ãŒããã¯ã㣠ã®èª¬æ çæAIãå
šäœåãçè§£ããããã®æ
å ±ïŒ ããŒã¿ã¬ã€ã€ãŒæ§é ãšåã¬ã€ã€ãŒã®è²¬å: raw, staging, core, martãªã© ããŒã¿ã¹ãã¢ã®æ§æ: ããŒã¿ã¬ã€ã¯, ããŒã¿ãŠã§ã¢ããŠã¹, ããŒã¿ããŒããªã© åããŒã¿ã¹ãã¢ã®ããŒã¿ãã©ã«ãæ§æ: "raw/ã·ã¹ãã å/YYYYMMDD"ãªã© ããŒã¿åŠçã¢ãããŒã: ELT ãªã© ã€ã³ãã©ã¹ãã©ã¯ãã£æ§æ: åããŒã¿ã¹ãã¢ã®ç©çå®è£
ã§å©çšã® SaaS ãªã© ããŒã¿ãã€ãã©ã€ã³æ§ææè¡: dagster , dbt core ãªã© ãããã®å
容ã ãããžã§ã¯ãããã¥ã¡ã³ãïŒDESIGN.mdçïŒ ãšããŠæŽåããŸããã ããŒã¿ã¢ãã«ã®èª¬æ çæAIãããŒã¿ãã€ãã©ã€ã³ã§å®çŸãã¹ãå¯Ÿè±¡ã®æ
å ±ïŒ ããŒã¿ã®æ§æ: ããŒãã«, ã«ã©ã , ãªã¬ãŒã·ã§ã³ã·ãããªã© ããŒã¿ä»æ§: ããŒã¿å, äž»ããŒçã®å¶çŽ, 倿ããžãã¯ãªã© ããŒã¿åè³ªä»æ§: æºããã¹ãåºæº, ãã¹ãã«ãŒã«ãªã© ãããã Data ContractïŒODCS圢åŒïŒ ãšããŠããã¥ã¡ã³ãæŽåããŸããã ãã®2ã€ãçµã¿åãããããšã§ãçæAIã¯ããŒã¿ãã€ãã©ã€ã³ã®å®è¡ç°å¢å
šäœã®èšèšæå³ãçè§£ããããŒã¿åºç€ïŒããŒã¿åæåºç€ã«é©ããå
·äœçãªå®è£
ãè¡ããããã«ãªããŸããã ããŒã¿ãã€ãã©ã€ã³äœæäŸé Œ çæAIãžã®éçºäŸé Œã¯ãããŒã¿ãã€ãã©ã€ã³ã®èŠä»¶ã ADR (Architecture Decision Record) ã«èšè¿°ãããã® ADR ã§å®çŸ©ãããããŒã¿ãã€ãã©ã€ã³ãé¢é£ããData Contractã«ããšã¥ããŠéçºããããã«çæAIãžäŸé Œããæ¹æ³ãæ¡çšããŸããã ADR ã«å°ãªããšã以äžã®å
å®¹ãæžãããã«ããŸããã ããŒã¿ãã€ãã©ã€ã³ã®äœæå¯Ÿè±¡ã®ããŒã¿å ããŒã¿æäŸã®èæ¯ãç®ç, ããŒã¿å©çšã® ãŠãŒã¹ã±ãŒã¹ ADR ã®èšèŒå
容ã®åèã€ã¡ãŒãž ã¿ã€ãã«: P001 ECããžãã¹ããŒã¿ãã€ãã©ã€ã³æ§é ã¹ããŒã¿ã¹: Accepted ã³ã³ããã¹ã: Eã³ããŒã¹ããžãã¹ãã€ãã©ã€ã³ã®ããŒã¿çµ±å察象ã®ããŒã¿ã決ããå¿
èŠãããã æ±ºå®äºé
: Eã³ããŒã¹ããžãã¹ãã€ãã©ã€ã³ã¯ãweb_salesãšweb_returnsã察象ãšããæ§æã«ããã æ€èšŒå
容ã®è©³çް å®éã®æ€èšŒç°å¢ãèŠä»¶ãäœæå¯Ÿè±¡ã®ããŒã¿ãªã©ã説æããŸãã æ€èšŒã§ã¯ TPC-DS (*1)ã®ããŒã¿ã䜿çšããçæAIã«ã¯ Claude CodeïŒSonnet 4.5, AnthropicïŒ ã䜿çšããŸããã web_salesããŒãã«ã®æŽæ°çšã®ãœãŒã¹ããŒã¿ã倿ããŠweb_salesãžçµ±åããããŒã¿ãã€ãã©ã€ã³ã® SQL ã¯ãšãªãçæããŸãã â»1 TPC -DS 㯠Transaction Processing Performance Council ( TPC ) ã®åæšã§ãã æ¬èšäºã¯ TPC ãã³ãããŒã¯ çµæã®å
¬è¡šãç®çãšãããã®ã§ã¯ãããŸããã ããŒã«ã«éçºç°å¢ ããŒã¿ãã€ãã©ã€ã³ã®ãããã€å
㯠AWS ãAzureã Google Cloudãªã©ã® SaaS ã§ãããAIé§åéçºãå¹çåããããã«ãããã€å
ãæš¡ããããŒã«ã«å®çµåã®éçºç°å¢ãæŽåããŸããã ããŒã¿ã¬ã€ã¯å
ŒããŒã¿ãŠã§ã¢ããŠã¹: DuckDB ããŒã¿ ã¢ããªã³ã° : dbt core ãªãŒã±ã¹ãã¬ãŒã¿: dagster ããŒã¿å質ãã¹ã: Soda Core DuckDBã§æ§æ : SaaS ãžã®å€éšéä¿¡ãæé€ããçæAIã®çç£æ§ãæå€§åã SaaS ã§å·šå€§ã«ãªããã¡ãªããŒã¿ã¬ã€ã¯ãããŒã¿ãŠã§ã¢ããŠã¹ãã³ã³ãã¯ããªDuckDBã§ä»£çšããŸããã ã³ãŒãããŒã¹ã®ããŒã«éžå® : çæAIãã³ãŒããçŽæ¥åç
§ã»å®è£
ã§ãããããå
šãŠã³ãŒãããŒã¹ã®ããŒã«ãæ¡çšã GUI ã®ETLããŒã«ã¯ãçæAIãšã®é£æºã§ç¬èª SDK / API çµç±ãå¿
èŠãšãªãçç£æ§ã®äœäžãæžå¿µããããã䜿çšããŸããã§ããã ããŒã«ã«ããã€ã³ãã©ã¹ãã©ã¯ãã£ã®æœè±¡å : ãããã€å
ã® SaaS ãšã¯ç©ççãªç°å¢æ§æãç°ãªããŸãããdbt coreãªã©ã®ãœãããŠã§ã¢ãæœè±¡åãç°å¢å·®åãåžåããŠãããŸãã ããŒã¿ ã¢ãŒããã¯ã㣠æŠèŠ ããŒã¿ãã€ãã©ã€ã³ã®ãããã€å
ã®ããŒã¿åæåºç€ã®ããŒã¿ ã¢ãŒããã¯ã㣠ãå³ã«ãããã®ã§ããè«çæ§æã®äž»èŠéšåã®ã¿ãæç²ããç°¡æœãªå
å®¹ãæ²èŒããŠãããŸãã ããŒã¿ã¬ã€ã€ãŒ: "raw", "staging", "core", "mart" ã®4å±€ ããŒã¿ã¹ãã¢: ããŒã¿ã¬ã€ã¯ãšããŒã¿ãŠã§ã¢ããŠã¹ ããŒã¿åŠçã¢ãããŒã: ELT ããŒã¿ãã€ãã©ã€ã³æ§ææè¡: Dagster, dbt core, Soda Core ã€ã³ãã©ã¹ãã©ã¯ãã£: AWS ãšAzure, Google Cloudã®3倧 ã¯ã©ãŠã ãµãŒãã¹ã§ãããã宿œã詳现ã¯å²æ ããŒã¿ã¢ãã« web sales â»äžéšã®ã«ã©ã ã¯å²æ ãã£ã¡ã³ã·ã§ãã« ã¢ããªã³ã° ãããã¹ã¿ãŒ ã¹ããŒã æ§é ãšãªã£ãŠãããè€æ°ã® ãµãã²ãŒãã㌠ãæã£ãŠããŸããããŒã¿æ§é ã¯ææ¡ãããããã®ã®ETL/ ELT åŠçã®èгç¹ã§ã¯10å以äžã®ããŒãã«çµåãäžéããŒãã«ãä»ããããŒãã«çµåãäžéšã«ã©ã ã®æŒç®åŠçãå¿
èŠãªè€éãªæ§é ã«ãªã£ãŠããŸãã äœæå¯Ÿè±¡ã®ããŒã¿ãã€ãã©ã€ã³ rawã¬ã€ã€ãŒ çããŒã¿ã®ããŒã¿ãã¡ã€ã«ãå€éšããŒãã« (External table) ãšããŠããŒã¿ãŠã§ã¢ããŠã¹ã«æ¢ã«ããŒãã«åãããŠããããã®ããŒãã«ãåç
§ããã°è¯ããããåŠçã®å®è£
ã¯äžèŠã stagingã¬ã€ã€ãŒ rawã¬ã€ã€ãŒã®ããŒã¿ãããšã«ãã¹ã¿ãŒããŒã¿ãšã®çµåã«ããããŒã¿ååŸãæŒç®åŠçãªã©ã® ããžãã¹ããžã㯠ãdbtã® SQL ã¯ãšãªã¢ãã«ãšããŠå®è£
SELECT察象ã®ã«ã©ã æ°: 35å ããŒãã«çµåæ°: 12å coreã¬ã€ã€ãŒ stagingã¬ã€ã€ãŒã®ããŒã¿ãããšã«å¢ååŠçã§ããŒã¿æŽæ°ããåŠçãdbtã® SQL ã¯ãšãªã¢ãã«ãšããŠå®è£
SELECT察象ã®ã«ã©ã æ°: 38å ããŒãã«çµåæ°: 0å å®è£
ã«ãŒã« SELECTæã§ã«ã©ã ã®åãã£ã¹ããæèš SELECTæã§ã«ã©ã ã®åç§°ãASæã§æå® CTE (Common Table Expression) ã§ãœãŒã¹ããŒã¿å®çŸ© coreã¬ã€ã€ãŒã§ã¯å¢ååŠçãå®è£
çæAIãžã®äŸé Œ åè¿°ã®ã ã¢ãããŒã: ããŒã¿ã¢ãŒããã¯ãã£ãšããŒã¿ã¢ãã«ãäŒãã ãã«èšèŒã®ããã¥ã¡ã³ãã«å ããŠãéçº ã¬ã€ãã©ã€ã³ ãæŽåãéçºäŸé Œãè¡ããŸããã çæAIãžæäŸããæ
å ± ãããžã§ã¯ãããã¥ã¡ã³ã Data Contract éçº ã¬ã€ãã©ã€ã³ (éçºèŠçŽã ãã©ãã«ã·ã¥ãŒãã£ã³ã° ãªã©) ADR (Architecture Decision Record) ããã³ãã å®éã®ããã³ãã 以äžã®ãã¡ã€ã«ãèªã¿èŸŒãã§ãåœãããžã§ã¯ãã«ã€ããŠçè§£ããŠãã ããã - ./docs/DESIGN.md - ./docs/DATAMODEL_GENERATION_GUIDELINES.md ADRã§å®çŸ©ãããããŒã¿ãã€ãã©ã€ã³ãé¢é£ãã`Data Contract`ã«ããšã¥ããŠäœæããŸãã ä»åã¯ãADRã®"./docs/adr/pipelines/P001-ecommerce-business-pipeline-structure.md"ã§æ±ºå®ããããŒã¿ãã€ãã©ã€ã³ãäœæããŠãã ããã å®éã®Data Contractã®ãµã³ãã« "ws_item_sk"ãšããäž»èŠãªã«ã©ã ã®Data Contractãäžéšæç²ãããã®ã§ãããã詳ããå
容ãšè§£èª¬ã¯ 詳现解説 ã«ãŠåŸè¿°ããŸãã - name : ws_item_sk businessName : ãŠã§ã販売ååãµãã²ãŒãã㌠logicalType : string physicalType : STRING primaryKey : true primaryKeyPosition : 1 transformSourceObjects : - core_tpcds.item.i_item_sk transformLogic : SELECT i_item_sk FROM s_web_order JOIN s_web_order_lineitem ON (word_order_id = wlin_order_id) LEFT JOIN item ON (wlin_item_id = i_item_id AND i_rec_end_date IS NULL ) relationships : - type : foreignKey to : - core_tpcds.item.i_item_sk quality : - type : library rule : duplicateCount name : "äž»ããŒéè€ãã§ãã¯ïŒws_item_skïŒ" dimension : uniqueness mustBe : 0 severity : error businessImpact : "éè€ã¬ã³ãŒãã«ãããã¡ã¯ãããŒãã«ã®ããŒã¿æŽåæ§ã厩ãã売äžéèšã«èª€å·®ãçãã" çæãããã³ãŒã stagingã¬ã€ã€ãŒ 12ããŒãã«ãçµåãã35ã«ã©ã ãçæãã SQL ã¯ãšãªã§ããData Contractã® transformLogic ã§å®çŸ©ãã倿ããžãã¯ãæ£ç¢ºã«åæ ãããŠããŸãã with s_web_order as ( select * from {{ source( ' raw_tpcds ' , ' s_web_order_1 ' ) }} ), s_web_order_lineitem as ( select * from {{ source( ' raw_tpcds ' , ' s_web_order_lineitem_1 ' ) }} ) SELECT d1.d_date_sk::STRING AS ws_sold_date_sk, t_time_sk::STRING AS ws_sold_time_sk, d2.d_date_sk::STRING AS ws_ship_date_sk, i_item_sk::STRING AS ws_item_sk, c1.c_customer_sk::STRING AS ws_bill_customer_sk, c1.c_current_cdemo_sk::STRING AS ws_bill_cdemo_sk, c1.c_current_hdemo_sk::STRING AS ws_bill_hdemo_sk, c1.c_current_addr_sk::STRING AS ws_bill_addr_sk, c2.c_customer_sk::STRING AS ws_ship_customer_sk, c2.c_current_cdemo_sk::STRING AS ws_ship_cdemo_sk, c2.c_current_hdemo_sk::STRING AS ws_ship_hdemo_sk, c2.c_current_addr_sk::STRING AS ws_ship_addr_sk, wp_web_page_sk::STRING AS ws_web_page_sk, web_site_sk::STRING AS ws_web_site_sk, sm_ship_mode_sk::STRING AS ws_ship_mode_sk, w_warehouse_sk::STRING AS ws_warehouse_sk, p_promo_sk::STRING AS ws_promo_sk, word_order_id::STRING AS ws_order_number, wlin_quantity:: INTEGER AS ws_quantity, i_wholesale_cost:: DECIMAL ( 7 , 2 ) AS ws_wholesale_cost, i_current_price:: DECIMAL ( 7 , 2 ) AS ws_list_price, wlin_sales_price:: DECIMAL ( 7 , 2 ) AS ws_sales_price, ((i_current_price:: DECIMAL ( 7 , 2 ) - wlin_sales_price:: DECIMAL ( 7 , 2 )) * wlin_quantity:: INTEGER ):: DECIMAL ( 7 , 2 ) AS ws_ext_discount_amt, (wlin_sales_price:: DECIMAL ( 7 , 2 ) * wlin_quantity:: INTEGER ):: DECIMAL ( 7 , 2 ) AS ws_ext_sales_price, (i_wholesale_cost:: DECIMAL ( 7 , 2 ) * wlin_quantity:: INTEGER ):: DECIMAL ( 7 , 2 ) AS ws_ext_wholesale_cost, (i_current_price:: DECIMAL ( 7 , 2 ) * wlin_quantity:: INTEGER ):: DECIMAL ( 7 , 2 ) AS ws_ext_list_price, (i_current_price:: DECIMAL ( 7 , 2 ) * web_tax_percentage:: DECIMAL ( 7 , 2 )):: DECIMAL ( 7 , 2 ) AS ws_ext_tax, wlin_coupon_amt:: DECIMAL ( 7 , 2 ) AS ws_coupon_amt, (wlin_ship_cost:: DECIMAL ( 7 , 2 ) * wlin_quantity:: INTEGER ):: DECIMAL ( 7 , 2 ) AS ws_ext_ship_cost, (wlin_sales_price:: DECIMAL ( 7 , 2 ) * wlin_quantity:: INTEGER - wlin_coupon_amt:: DECIMAL ( 7 , 2 )):: DECIMAL ( 7 , 2 ) AS ws_net_paid, (((wlin_sales_price:: DECIMAL ( 7 , 2 ) * wlin_quantity:: INTEGER ) - wlin_coupon_amt:: DECIMAL ( 7 , 2 )) * ( 1 + web_tax_percentage:: DECIMAL ( 7 , 2 ))):: DECIMAL ( 7 , 2 ) AS ws_net_paid_inc_tax, (((wlin_sales_price:: DECIMAL ( 7 , 2 ) * wlin_quantity:: INTEGER ) - wlin_coupon_amt:: DECIMAL ( 7 , 2 )) - (wlin_quantity:: INTEGER * i_wholesale_cost:: DECIMAL ( 7 , 2 ))):: DECIMAL ( 7 , 2 ) AS ws_net_paid_inc_ship, ((wlin_sales_price:: DECIMAL ( 7 , 2 ) * wlin_quantity:: INTEGER ) - wlin_coupon_amt:: DECIMAL ( 7 , 2 ) + (wlin_ship_cost:: DECIMAL ( 7 , 2 ) * wlin_quantity:: INTEGER ) + i_current_price:: DECIMAL ( 7 , 2 ) * web_tax_percentage:: DECIMAL ( 7 , 2 )):: DECIMAL ( 7 , 2 ) AS ws_net_paid_inc_ship_tax, (((wlin_sales_price:: DECIMAL ( 7 , 2 ) * wlin_quantity:: INTEGER ) - wlin_coupon_amt:: DECIMAL ( 7 , 2 )) - (i_wholesale_cost:: DECIMAL ( 7 , 2 ) * wlin_quantity:: INTEGER )):: DECIMAL ( 7 , 2 ) AS ws_net_profit, s_web_order.processing_date::STRING AS processing_date, current_timestamp :: TIMESTAMP AS ingested_at FROM s_web_order LEFT JOIN {{ ref( ' date_dim ' ) }} d1 ON ( cast (word_order_date AS date ) = d1.d_date) LEFT JOIN {{ ref( ' time_dim ' ) }} ON (word_order_time = t_time) LEFT JOIN {{ ref( ' customer ' ) }} c1 ON (word_bill_customer_id = c1.c_customer_id) LEFT JOIN {{ ref( ' customer ' ) }} c2 ON (word_ship_customer_id = c2.c_customer_id) LEFT JOIN {{ ref( ' web_site ' ) }} ON (word_web_site_id = web_site_id AND web_rec_end_date IS NULL ) LEFT JOIN {{ ref( ' ship_mode ' ) }} ON (word_ship_mode_id = sm_ship_mode_id) JOIN s_web_order_lineitem ON (word_order_id = wlin_order_id) LEFT JOIN {{ ref( ' date_dim ' ) }} d2 ON ( cast (wlin_ship_date AS date ) = d2.d_date) LEFT JOIN {{ ref( ' item ' ) }} ON (wlin_item_id = i_item_id AND i_rec_end_date IS NULL ) LEFT JOIN {{ ref( ' web_page ' ) }} ON (wlin_web_page_id = wp_web_page_id AND wp_rec_end_date IS NULL ) LEFT JOIN {{ ref( ' warehouse ' ) }} ON (wlin_warehouse_id = w_warehouse_id) LEFT JOIN {{ ref( ' promotion ' ) }} ON (wlin_promotion_id = p_promo_id) coreã¬ã€ã€ãŒ stagingã¬ã€ã€ãŒã®ããŒã¿ãå¢ååŠçã§coreã¬ã€ã€ãŒã«åã蟌ã SQL ã¯ãšãªã§ããdbtã®incrementalã¢ãã«ã䜿çšããŠããŸãã {{ config( materialized= ' incremental ' , unique_key=[ ' ws_item_sk ' , ' ws_order_number ' ], on_schema_change= ' fail ' , incremental_strategy= ' merge ' if target. type != ' duckdb ' else ' delete+insert ' ) }} -- Incrementalã¢ãã«ïŒstagingå±€ããããŒã¿ãååŸ select ws_sold_date_sk::STRING as ws_sold_date_sk, ws_sold_time_sk::STRING as ws_sold_time_sk, ws_ship_date_sk::STRING as ws_ship_date_sk, ws_item_sk::STRING as ws_item_sk, ws_bill_customer_sk::STRING as ws_bill_customer_sk, ws_bill_cdemo_sk::STRING as ws_bill_cdemo_sk, ws_bill_hdemo_sk::STRING as ws_bill_hdemo_sk, ws_bill_addr_sk::STRING as ws_bill_addr_sk, ws_ship_customer_sk::STRING as ws_ship_customer_sk, ws_ship_cdemo_sk::STRING as ws_ship_cdemo_sk, ws_ship_hdemo_sk::STRING as ws_ship_hdemo_sk, ws_ship_addr_sk::STRING as ws_ship_addr_sk, ws_web_page_sk::STRING as ws_web_page_sk, ws_web_site_sk::STRING as ws_web_site_sk, ws_ship_mode_sk::STRING as ws_ship_mode_sk, ws_warehouse_sk::STRING as ws_warehouse_sk, ws_promo_sk::STRING as ws_promo_sk, ws_order_number::STRING as ws_order_number, ws_quantity:: INTEGER as ws_quantity, ws_wholesale_cost:: DECIMAL ( 7 , 2 ) as ws_wholesale_cost, ws_list_price:: DECIMAL ( 7 , 2 ) as ws_list_price, ws_sales_price:: DECIMAL ( 7 , 2 ) as ws_sales_price, ws_ext_discount_amt:: DECIMAL ( 7 , 2 ) as ws_ext_discount_amt, ws_ext_sales_price:: DECIMAL ( 7 , 2 ) as ws_ext_sales_price, ws_ext_wholesale_cost:: DECIMAL ( 7 , 2 ) as ws_ext_wholesale_cost, ws_ext_list_price:: DECIMAL ( 7 , 2 ) as ws_ext_list_price, ws_ext_tax:: DECIMAL ( 7 , 2 ) as ws_ext_tax, ws_coupon_amt:: DECIMAL ( 7 , 2 ) as ws_coupon_amt, ws_ext_ship_cost:: DECIMAL ( 7 , 2 ) as ws_ext_ship_cost, ws_net_paid:: DECIMAL ( 7 , 2 ) as ws_net_paid, ws_net_paid_inc_tax:: DECIMAL ( 7 , 2 ) as ws_net_paid_inc_tax, ws_net_paid_inc_ship:: DECIMAL ( 7 , 2 ) as ws_net_paid_inc_ship, ws_net_paid_inc_ship_tax:: DECIMAL ( 7 , 2 ) as ws_net_paid_inc_ship_tax, ws_net_profit:: DECIMAL ( 7 , 2 ) as ws_net_profit, processing_date::STRING as processing_date, ingested_at:: TIMESTAMP as ingested_at, current_timestamp :: TIMESTAMP as inserted_at, current_timestamp :: TIMESTAMP as updated_at from {{ ref( ' web_sales_view ' ) }} {% if is_incremental() %} -- incrementalã®å Žåãprocessing_dateã§ãã£ã«ã¿ãªã³ã° where processing_date = ' {{ var("processing_date", "20240101") }} ' {% endif %} 詳现解説 Data Contractãšã¯ïŒ Data Contractã¯ãããŒã¿ã®æäŸè
ãšå©çšè
ã®éã§ããŒã¿ä»æ§ãããŒã¿å質èŠä»¶ãåæããããã®æšæºãã©ãŒãããã§ãã ãã©ãŒãããã®æ¢å®ã®é
ç®ã䜿ã£ãŠããŒã¿ä»æ§ãããŒã¿å質èŠä»¶ãå®çŸ©ããããšãã§ããŸãã æ¢å®ã®é
ç®ããµããŒãããŠãããã®ã¯ãããŒãã« ã¹ããŒã ãåçš®ã®è«çåãããŒã¿ã®ç²åºŠãªã©ã® ã¡ã¿ããŒã¿ ã«å ããŠããµã³ãã«ããŒã¿ãããŒã¿å質ã«ãŒã«ãªã©å€å²ã«æž¡ããŸãã åœæ€èšŒã§ã¯ã ODCS (Open Data Contract Standard) ããšãã YAML 圢åŒã®ãã©ãŒãããã䜿çšããŸããODCS㯠Linux Foundationåäžã®Bitolãããžã§ã¯ãããµããŒããã代衚çãªä»æ§ã§ãã åœèšäºã§ã¯ãData ContractãçæAIãžäžããã³ã³ããã¹ããšããŠæŽ»çšããŸããData Contractã®äœæèªäœã¯èšèšããã»ã¹ã§çæAIã«è¡ãããããšãå¯èœã§ãïŒå¥èšäºã§èª¬æäºå®ïŒã ä»å䜿çšã®Data Contractã®äž»èŠé
ç® äž»ã«äœ¿çšããé
ç®ãšåé
ç®ã®ç°¡åãªèª¬æãèšèŒããŸãã name: ã«ã©ã ç©çå businessName: ã«ã©ã è«çå logicalType: è«çããŒã¿å physicalType: ç©çããŒã¿å transformSourceObjects: ã«ã©ã ã®ããŒã¿ãäœæããéã«å¿
èŠãªãœãŒã¹ããŒã¿ã«é¢ããæ
å ± transformLogic: ã«ã©ã ã®ããŒã¿ãäœæããéã«å¿
èŠãªããŒã¿å€æããžã㯠relationships: ã«ã©ã ã®ãªã¬ãŒã·ã§ã³ã·ããæ
å ± quality: ã«ã©ã ã®ããŒã¿å質èŠä»¶ å®éã®Data Contract 以äžã代衚çãª2ã€ã®ã«ã©ã ã®Data Contractãæç²ããŸãã äž»ããŒå
Œå€éšããŒã®äŸ: ws_item_sk ãã¡ãã¯åè¿°ã®Data Contractã®è©³è¿°ã§ãã è€åäž»ããŒã®äžéšã§ããå€éšããŒã§ãããã«ã©ã ã§ãã3ããŒãã«ã®çµåãšã3çš®é¡ã®ããŒã¿å質ãã¹ããå®çŸ©ãããŠããŸãã - name : ws_item_sk businessName : ãŠã§ã販売ååãµãã²ãŒãã㌠logicalType : string physicalType : STRING primaryKey : true primaryKeyPosition : 1 transformSourceObjects : - core_tpcds.item.i_item_sk transformLogic : SELECT i_item_sk FROM s_web_order JOIN s_web_order_lineitem ON (word_order_id = wlin_order_id) LEFT JOIN item ON (wlin_item_id = i_item_id AND i_rec_end_date IS NULL ) relationships : - type : foreignKey to : - core_tpcds.item.i_item_sk customProperties : - target_contract_id : "core_tpcds.item" - target_contract_path : "./core_tpcds/item.yaml" - target_property : "i_item_sk" quality : - type : library rule : duplicateCount name : "äž»ããŒéè€ãã§ãã¯ïŒws_item_skïŒ" dimension : uniqueness mustBe : 0 severity : error businessImpact : "éè€ã¬ã³ãŒãã«ãããã¡ã¯ãããŒãã«ã®ããŒã¿æŽåæ§ã厩ãã売äžéèšã«èª€å·®ãçãã" - type : library rule : nullCount name : "äž»ããŒNULLãã§ãã¯ïŒws_item_skïŒ" dimension : completeness mustBe : 0 severity : error businessImpact : "NULLå€ãååšãããšååå¥å£²äžåæãäžå¯èœã«ãªã" - type : sql name : "ååãã¹ã¿åç
§æŽåæ§ãã§ãã¯" dimension : consistency query : | SELECT COUNT(*) FROM ${object} ws LEFT JOIN item i ON ws.ws_item_sk = i.i_item_sk WHERE ws.ws_item_sk IS NOT NULL AND i.i_item_sk IS NULL mustBe : 0 severity : error businessImpact : "åç
§æŽåæ§ã厩ãããšååãã¹ã¿ãšã®çµåã§æ¬ æãçºçããååæ
å ±ãååŸã§ããªã売äžããŒã¿ãçãã" è€éãªèšç®ããžãã¯ã®äŸ: ws_net_profit çŽå©çã¯è€æ°ã®ãœãŒã¹ããèšç®ãããèšç®æŽåæ§ãšå¹³åå€ã®åŠ¥åœæ§ã®äž¡æ¹ããã§ãã¯ãããŸãã - name : ws_net_profit businessName : ãŠã§ãçŽå©ç logicalType : number physicalType : DECIMAL(7, 2) transformSourceObjects : - raw.s_web_order_lineitem.wlin_sales_price - raw.s_web_order_lineitem.wlin_quantity - raw.s_web_order_lineitem.wlin_coupon_amt - core_tpcds.item.i_wholesale_cost transformLogic : SELECT (((wlin_sales_price::DECIMAL(7,2) * wlin_quantity::INTEGER) - wlin_coupon_amt::DECIMAL(7,2)) - (i_wholesale_cost::DECIMAL(7,2) * wlin_quantity::INTEGER))::DECIMAL(7,2) FROM s_web_order JOIN s_web_order_lineitem ON (word_order_id = wlin_order_id) LEFT JOIN item ON (wlin_item_id = i_item_id AND i_rec_end_date IS NULL ) quality : - type : sql name : "çŽå©çèšç®æŽåæ§ãã§ãã¯" dimension : accuracy query : | SELECT COUNT(*) FROM ${object} WHERE ws_net_profit IS NOT NULL AND ws_net_paid IS NOT NULL AND ws_ext_wholesale_cost IS NOT NULL AND ABS(ws_net_profit - (ws_net_paid - ws_ext_wholesale_cost)) > 0.01 mustBe : 0 severity : error businessImpact : "èšç®åŒãäžæ£ç¢ºãªå Žåãå©çåæã®ä¿¡é Œæ§ãæãªã" - type : sql name : "çŽå©çå¹³åå€ãã§ãã¯" dimension : accuracy query : | SELECT AVG(ws_net_profit) FROM ${object} WHERE ws_net_profit IS NOT NULL mustBeBetween : [ -100, 1000 ] severity : warning businessImpact : "å¹³åå©çãç°åžžå€ã®å ŽåãäŸ¡æ Œèšå®ãå䟡ããŒã¿ã«åé¡ãããå¯èœæ§" transformSourceObjectsãštransformLogicã®éèŠæ§ åœæ€èšŒã§ç¹ã«è©Šè¡é¯èª€ããã®ãããã®2ã€ã®ããããã£ã®èšè¿°ã§ããã åœåã®ä»®èª¬ : åããããã£ã§ããŒã¿ä»æ§ãèšè¿°ããrelationshipsããããã£ã§ããŒã¿éã®é¢ä¿æ§ãèšè¿°ããã°ãçæAIãJOINåŠçãã«ã©ã æ¯ã®èšç®ããžãã¯ã倿ããžãã¯ã顿šã§ããã®ã§ã¯ãªãã å®éã®èª²é¡ : ããŒã¿ä»æ§ããªã¬ãŒã·ã§ã³ã·ããïŒããŒã¿ã¢ãã«äžã®é¢ä¿æ§ïŒãšETL/ ELT åŠçïŒããŒã¿å€æããžãã¯ïŒã®éã«ã¯éããããã äŸãã°ã ws_item_sk ã«ã©ã 㯠item ããŒãã«ãšå€éšããŒé¢ä¿ã«ãããŸãããå®éã®ããŒã¿ååŸã«ã¯ s_web_order â s_web_order_lineitem â item ãšãã3ããŒãã«ã®çµåãå¿
èŠã§ãã ããã¯çæAIã«åãã£ãŠãããããã§åãã£ãŠããããŸããã§ããã s_web_order ãš item ã®2ã€ã®ããŒãã«ãç¡çã«çµåããããšããŠããŸããŸããããã£ããã§èª¬æããŠæ¹åããŠãåŸã§å¿ããŠåé¡ãåçºããå¿ããªãããã«èšé²ããŠããã£ãŠãåå¥ã®ã±ãŒã¹ã®å®è£
æ¹æ³ãšããŠèšé²ãããŠä»ã±ãŒã¹ãžã®å¿çšãæžå¿µãããŸããã ä»ã«ãçæAI㯠ãµãã²ãŒãã㌠ã©ããã§ããŒãã«çµåããããäž»ããŒã§çµåããããã«æç€ºããŠæ¹åããŠãä»åºŠã¯èå¿ã® ãµãã²ãŒãã㌠ãååŸããŠãããªãã£ããããŸããã ãŸããã«ã©ã ã®æŒç®åŠçã¯ééã£ãå
容ãå®è£
ããããå
šãå®è£
ãããªããã§ããã ãã®ããã以äžã®2ã€ã®ããããã£ã䜿ãããã«ããŸããïŒ transformSourceObjects : ããŒã¿ã®åºæïŒãœãŒã¹ããŒãã«ã»ã«ã©ã ïŒãæç€º transformLogic : å
·äœçãªå€æããžãã¯ïŒJOINæ¡ä»¶ããã£ã«ã¿ãèšç®åŒãªã©ïŒãèšè¿° ããã«ãããçæAIã¯ãäœãåç
§ããŠãã©ã倿ãããããæ£ç¢ºã«çè§£ããæå³éãã® SQL ãçæã§ããããã«ãªããŸããã transformSourceObjectsãštransformLogicã®äœææ¹æ³ transformSourceObjectsããããã£ãštransformLogicããããã£ã®äœæèªäœãããŒã¿ã¢ãã«ã®èšèšéçšã§çæAIãšå
±ã«è¡ããŸãã ãããããŒã¿ãœãŒã¹åŽã®ã·ã¹ãã ã§æ¥åç¥è (äŸ. çŽå©çã®èšç®åŒ) ãäœãããã¥ã¡ã³ããªã©ã«æŽçãããŠããã°ããã®å
容ãããšã«çæAIãäž¡ããããã£ãå®çŸ©ã§ããå¯èœæ§ããããŸãã ãŸããäŸãã°ãã£ã¡ã³ã·ã§ãã« ã¢ããªã³ã° ã§ããã°ãããŒã¿ç²åºŠ (Grain) ã®å®çŸ©ãConformance Matrixãªã©ãæ©æ¢°å€èªå¯èœãªãã¡ã€ã«åœ¢åŒã§çæAIãšå
±ã«äœæãããããææç©ããœãŒã¹ãšããŠçæAIãäž¡ããããã£ãäœæã§ããå¯èœæ§ããããŸãã relationshipsããããã£ã¯ ODCS v3.1.0 ã«ãªãªãŒã¹äºå® ODCSã®çŸåšã®ææ°å
¬åŒããŒãžã§ã³ã¯v3.0.2ã§ããããã®ããŒãžã§ã³ã«ã¯ relationshipsããããã£ãååšããŸãã ã ODCS v3.1.0ã§relationshipsããããã£ãå®è£
ãããäºå®ã§ãåœæ€èšŒã§ã¯ODCSã® v3.1.0 ãå
è¡çã«äœ¿çšããŠããŸãã v3.1.0ã§ã®è¿œå äºå®: ODCS v3.1.0 RFC0013 ãã®ä»ã®ããããã£ã®äœææ¹æ³ nameããããã£ãphysicalTypeããããã£ãªã©ã®åºç€çãªããããã£ã®äœæã¯ãããŒã¿ãœãŒã¹ã®DBã® ã¹ããŒã æ
å ±ãçããŒã¿ã®ããŒã¿ãã¡ã€ã«ããœãŒã¹ãšããŠçæAIã«äžããããšã§åèªåçæãããã¯èªåçæãå¯èœã§ãã Excel ãªã©ã®åºæãœãããŠã§ã¢ã®ãã¡ã€ã«åœ¢åŒã®å Žåã¯ãæ©æ¢°å€èªå¯èœãªãªãŒãã³ãªãã¡ã€ã«åœ¢åŒãžã®å€æãå¿
èŠã§ãã ãããã« æ€èšŒã®æ¯ãè¿ã æ¬èšäºã§ã¯ãData ContractïŒODCS圢åŒïŒã掻çšããããŒã¿ãã€ãã©ã€ã³ã®AIé§åéçºææ³ãæ€èšŒããŸããã ææãšããŠåŸãããããš ïŒ 12ããŒãã«çµåã35ã«ã©ã ã®è€é㪠SQL ã¯ãšãªãçæAIãæ£ç¢ºã«çæ Data Contractã«å®çŸ©ããåãã£ã¹ããèšç®ããžãã¯ãJOINæ¡ä»¶ãæå³éãã«å®è£
å¢ååŠçãdbtã®incrementalèšå®ãé©åã«çæ æåèŠå ïŒ ããŒã¿ ã¢ãŒããã¯ã㣠ãšããŒã¿ã¢ãã«ã®2軞ã§ã³ã³ããã¹ããæŽç ããŒã¿é
ç®ãããŒã¿åãå¶çŽãªã©ã® ã¹ããŒã æ
å ±ã«å ããŠã transformSourceObjects ãš transformLogic ã§ããŒã¿ãªããŒãžãæç€º ããŒã«ã«å®çµåã®éçºç°å¢ã§çæAIã®è©Šè¡é¯èª€ãé«éå ææ ããŒã¿ãšã³ãžãã¢ãªã³ã°ã®AIé§åéçºã«åããããã¥ã¡ã³ãæŽåæ¹æ³ã®1ã€ãèŠåºãããšãã§ããŠè¯ãã£ããšæããŸãã ããŒã¿åºç€ïŒããŒã¿åæåºç€ãæŠå¿µçã«åè§£ããæ¢åã®å°éçšèªã«åœãŠã¯ããŠæ§é åããŠèª¬æããããšã§çæAIãäœç³»çãã€è©³çްã«çè§£ããããã«ãªãããšã宿ã§ããŸããã ãŸããç¹å®ã®è£œåã»ãµãŒãã¹ã«äŸåããã«é²ããããŸããã®ã§ãåå©çšæ§ãæ¯èŒçã«é«ããšèããŠãããä»åŸã®å¿çšãšããŠé¢é£ã®è£œåã»ãµãŒãã¹ãšé£æºãDataOpsãAIOpsãžæè¯ã§ããã°ãšæåŸ
ããŠããŸãã ããŒã¿ãããžã¡ã³ãã®ããåºãç¯å²ãžã®æŽ»çšã®å¯èœæ§ æ¬èšäºã§ã¯ããŒã¿ãã€ãã©ã€ã³ã®ã³ãŒãçæã«çŠç¹ãåœãŠãŸããããData Contractèªäœã®çæãããŒã¿å質ãã¹ãã®èªåçæãªã©ãããåºãç¯å²ãžã®æŽ»çšãèããããŸããå®éã«Data Contractã® quality ããããã£ãå¿çšããããŒã¿å質ãã¹ãã³ãŒãã®çæãæ€èšŒããŠããããã®å
容ã¯å¥éèšäºåã®äºå®ã§ãã ãŸããããŒã¿ã®æå³çãªå®çŸ©ãããŒã¿çµ±åã«é¢ãã ãããã³ã° æ
å ±ãªã©ãæŽåããŠData Contractãæ¡åŒµããããšã§ãäŸãã°ããŒã¿ãããžã¡ã³ãå
šè¬ã«ãããæ©æ¢°å€èªå¯èœãªããã¥ã¡ã³ããšããŠã®æŽ»çšãAI䌎走åã®åæããŒã¿ã¢ãã«éçºãžã®å¿çšãæ€èšããŠããŸãã ããŒã¿ãšã³ãžãã¢ãšããŠãããŒã¿ãããžã¡ã³ãã«ãããData ContractãšçæAIã®æŽ»çšå¯èœæ§ã«ã€ããŠã¯ãä»åŸãç¶ç¶çã«æ€èšŒãé²ããŠããäºå®ã§ãã ç§ãã¡ã¯äžç·ã«åããŠããã仲éãåéããŠããŸãïŒ é»éç·ç ãã£ãªã¢æ¡çšãµã€ã é»éç·ç æ°åæ¡çšãµã€ã å·çïŒ @shikarashika ã¬ãã¥ãŒïŒ @yamada.y ïŒ Shodo ã§å·çãããŸãã ïŒ