ã¿ãªãããããã«ã¡ã¯ãçŸåšã¡ã€ã³ãã¬ãŒã ãå©çšãããŠããæ¹ã®äžã§ãã¡ã€ã³ãã¬ãŒã ã·ã¹ãã äžã®æ¥åããŒã¿ã掻çšããããšã§ãããžãã¹æææ±ºå®ã®é«åºŠåãæ°ããªããžãã¹åµåºãè¡ããããžãã¹äŸ¡å€ãåäžãããæ¹æ³ãæ€èšãããŠããæ¹ã¯ããã£ããããŸãã§ããããããã¡ãã®æ€èšããããŠããæ¹ã¯ãå€ãã®ã±ãŒã¹ã«ãããŠä»¥äžã®ãããªèª²é¡ã«çŽé¢ãããŠããã®ã§ã¯ãªãã§ããããã ã¡ã€ã³ãã¬ãŒã ã®åŠçèœåãéçã«è¿ã¥ããŠãããçŸè¡åŠçã«åœ±é¿ããªãããã«ããŒã¿åæãããŒã¿å å·¥ãè¡ãããšã¯é£ããã ã¡ã€ã³ãã¬ãŒã ããããŒã¿ããšã¯ã¹ããŒããããããããŒã¿æ§é ãåå€æã«æéãšæéããããããŒã¿é®®åºŠãé«ããããªãã æ¬ããã°ã§ã¯ããããã®èª²é¡ã«å¯Ÿããäžã€ã®è§£æ±ºçã§ãã Precisely ãçšãããã¢ãªã¢ã«ã¿ã€ã ã®ããŒã¿åæãã玹ä»ããŸãããã®æ¹æ³ã䜿ãããšã§ããŒã¿æ§é ã圢åŒã倿ããªãããã¡ã€ã³ãã¬ãŒã äžã®æ§ã
ãªããŒã¿ãœãŒã¹ãã AWS ç°å¢äžã®ããŒã¿ã¹ãã¢ãžããã¢ãªã¢ã«ã¿ã€ã ã®åæãè¡ãããšã§ãå€éšã·ã¹ãã äžã§é®®åºŠã®é«ãããŒã¿ãçšããŠããŒã¿æŽ»çšãè¡ãããšãå¯èœã§ãã Precisely ã䜿çšãã AWS Mainframe Modernization Data Replication ãšã¯ AWS Mainframe Modernization Data Replication ã¯ãPrecisely 瀟㮠Change Data Capture (以äžãCDC) ãã¯ãããžãŒã掻çšãããã¡ã€ã³ãã¬ãŒã ã®ããŒã¿ã AWS ã¯ã©ãŠããžåæããããã®ãœãªã¥ãŒã·ã§ã³ã§ããã¡ã€ã³ãã¬ãŒã äžã®æ§ã
ãªããŒã¿ãœãŒã¹ãã AWS ç°å¢äžã®ããŒã¿ã¹ãã¢ãžããã¢ãªã¢ã«ã¿ã€ã ã®ã¬ããªã±ãŒã·ã§ã³ãå®çŸããŸãã ãã®ãœãªã¥ãŒã·ã§ã³ã®ç¹åŸŽãšããŠã¯ã以äžã®ãããªãã®ããããŸãã CDC ã«ãããã¢ãªã¢ã«ã¿ã€ã ã®ã¬ããªã±ãŒã·ã§ã³ã®å®çŸ ã¡ã€ã³ãã¬ãŒã ã®ããŒã¿ãœãŒã¹ãžã®è² è·ã®é
æ
® Db2ãIMSãVSAM ãªã©ã¡ã€ã³ãã¬ãŒã äžã®æ§ã
ãªããŒã¿ãœãŒã¹ãžã®å¯Ÿå¿ Amazon Managed Streaming for Apache Kafka (以äžãMSK) ãšçµ±åããããšã«ãããAWS ãµãŒãã¹ãçŽæ¥ã¿ãŒã²ãããšããã¬ããªã±ãŒã·ã§ã³ã®å®çŸ ãã®ãœãªã¥ãŒã·ã§ã³ã¯ã以äžã®ãããªæ§ã
ãªããŒã¿å©æŽ»çšã®ãŠãŒã¹ã±ãŒã¹ã«é©çšå¯èœã§ãã AWS ã¯ã©ãŠãäžã®ããŒã¿ã¬ã€ã¯ãããŒã¿ãŠã§ã¢ããŠã¹ã«åæããã¡ã€ã³ãã¬ãŒã ããŒã¿ã䜿ã£ããBI ã«ããå¯èŠåã AI/ML ã«ããæ°ããªããŒã¿ã®æŽ»çš AWS ã¯ã©ãŠãäžã®ããŒã¿ã¹ãã¢ã«ãã¢ãªã¢ã«ã¿ã€ã ã§åæãããã¡ã€ã³ãã¬ãŒã ããŒã¿ã«å¯Ÿãããæ°èŠããžãã¹ã¢ããªã±ãŒã·ã§ã³ããã®ããŒã¿åç
§ ã¡ã€ã³ãã¬ãŒã ã®ããŠã³ã¿ã€ã ãæå°åããªãããã¡ã€ã³ãã¬ãŒã ã¢ããªã±ãŒã·ã§ã³ããã³ããŒã¿ã AWS ã¯ã©ãŠãã«ç§»è¡ãã ã¢ãŒããã¯ãã£æŠèŠ å®éã«ã¡ã€ã³ãã¬ãŒã ããAWS ã¯ã©ãŠãäžã«ããŒã¿ãåæããéã®ãæšå¥šããåºæ¬æ§æã¯ä»¥äžã®ããã«ãªããŸãã ãã®ã¢ãŒããã¯ãã£ã¯ãAmazon EC2 ã€ã³ã¹ã¿ã³ã¹ã§ãããã€ããã Precisely Apply Engine ãå¢ç®ãšããŠãã¡ã€ã³ãã¬ãŒã åŽãš AWS ã¯ã©ãŠãåŽã«åããããŸããã¡ã€ã³ãã¬ãŒã åŽããéä¿¡ãããããŒã¿ã¯ãApply Engine ã§åŠçã»å€æãããåŸç¶ã® AWS ãµãŒãã¹ã«éãããŸããã¡ã€ã³ãã¬ãŒã åŽã«ã¯ãã°åéçšã®ãšãŒãžã§ã³ããã€ã³ã¹ããŒã«ãããŠãããããŒã¿ãäžå®éèç©ããäžã§ã®ãããéä¿¡ãŸãã¯ãã°ã¹ããªãŒã ã®ãã¢ãªã¢ã«ã¿ã€ã éä¿¡ãå¯èœã§ããAWSã¯ã©ãŠãåŽã«ã¯ Apply Engine ãæ§æããããšã§ããã¹ãããéä¿¡ãããããŒã¿ãåŸç¶ã®ãµãŒãã¹ã§å©çšãããã圢åŒã«å å·¥ããªãã飿ºããããšãå¯èœã§ããMSK ãšçµã¿åãããããšã§ãAmazon RedshiftãAmazon S3ãAmazon RDSãAmazon AuroraãAWS Lambda ãªã©ãæ§ã
ãªãµãŒãã¹ã«å¯ŸããŠãã¢ãªã¢ã«ã¿ã€ã ãªããŒã¿é£æºãå¯èœã§ããæ¬ããã°äžã§ã¯ãS3 ãš Redshift ã«é£æºããããã®æ§ææ¹æ³ãæ±ããŸãã ããŒã¿ã®æŽ»çšã·ããªãªäŸ å€ãã®ã¡ã€ã³ãã¬ãŒã äžã«ã¯ã顧客æ
å ±ãååŒå±¥æŽãåšåº«ããŒã¿ã売äžããŒã¿ãªã©ãäŒæ¥ã®åºå¹¹æ¥åã§æ±ãããéèŠãªæ
å ±ãå«ãŸããŠããããšæããŸãããããã®ããŒã¿ã掻çšããäŸãšããŠã以äžã®ãããªã·ããªãªãæ³å®ããªããä»åã®ç°å¢æ§ç¯æ¹æ³ã玹ä»ããŠãããŸããã·ããªãªäŸ1ãšããŠã¯ãããŒã¿ã®é·æä¿åãæ©æ¢°åŠç¿ã§ã®æŽ»çšãäž»ãªç®çãšã㊠S3 ã«ããŒã¿ãæ ŒçŽããæ§æãã玹ä»ããŸããã·ããªãªäŸ2ãšããŠã¯ããã峿æ§ãéèŠãããã¢ãªã¢ã«ã¿ã€ã åæãããã·ã¥ããŒã衚瀺ãå¯èœãšãã Redshift ã«ããŒã¿ãæ ŒçŽããæ§æãã玹ä»ããŸãã ã·ããªãªäŸ1ïŒææ¬¡ã®ç£æ»å®æœã¿ã€ãã³ã°ã§ååŒããŒã¿ã顧客è¡åããŒã¿ãå奿œåºããŠäžæ£ååŒãæ€åºããŠããéèæ©é¢ãååšããããã®ç¶æ³ã«å¯ŸããŠãPrecisely ãçšããŠåœè©²ããŒã¿ããã¢ãªã¢ã«ã¿ã€ã ã§æœåºã S3 ã«èç©ãããã倿Žãè¡ãããšã§ãæ¥ã
ã®ææ°ããŒã¿ãçšããŠäžæ£ååŒæ€åºãè¡ããè¿
éã«èªåã¢ã©ãŒããçºä¿¡ããããšãå¯èœã«ãªãã ã·ããªãªäŸ2ïŒè£œé ã©ã€ã³ã®åå·¥çšã§çæãããå質ããŒã¿ã皌åããŒã¿ãã¡ã€ã³ãã¬ãŒã ã«èç©ããæ¥æ¬¡ã¬ããŒãã§å質åŸåãææ¡ããŠããè£œé æ¥äŒæ¥ãååšããããã®ç¶æ³ã«å¯ŸããŠãPrecisely ãçšããŠã補é å·¥çšã§ã®å質ããŒã¿ã»çšŒåããŒã¿çæãšåæã« Redshift ã«æ ŒçŽãããã倿Žãè¡ãããšã§ãå質ç°åžžãèšåæ
éã®å
åãæ°å以å
ã«æ€ç¥ããçç£ã©ã€ã³åæ¢åã®äºé²ä¿å
šãå®çŸã§ããããã«ãªãã 詳现ãªèšå®æ¹æ³ Precisely Connect Apply Engine ã®æºå Precisely ã® Apply Engine ãå©çšããããã®æãç°¡åãªæ¹æ³ã¯ AWS ããŒã±ãããã¬ã€ã¹ã«æºåãããã Precisely Connect ã®ã¢ãã©ã€ãšã³ãžã³ãæ§æããããã® EC2 ã® AMI ãå©çšããããšã§ãããã® AMI ã«ã¯ããã€ã³ã¹ããŒã«æžã¿ã®ãœãããŠã§ã¢ã¹ã¿ãã¯ããäœæã»åæåæžã¿ã®ã¬ããªã±ãŒã·ã§ã³ã€ã³ã¹ã¿ã³ã¹ããAmazon CloudWatch ãšçµ±åæžã¿ã®ããã»ã¹ãšããŒã±ãããã¬ã€ã¹ãšçµ±åæžã¿ã®ã¬ããªã±ãŒã·ã§ã³ãµãŒãã¹ããå°å
¥æžã¿ã®ãããEC2 ã€ã³ã¹ã¿ã³ã¹èµ·ååŸã«ããã«å©çšãéå§ããããšãã§ããŸããããŒã±ãããã¬ã€ã¹ãããAWS Mainframe Modernization â Data Replication for IBM z/OSããæ€çŽ¢ãããµãã¹ã¯ã©ã€ããè¡ãããšã§ã該åœã® AMI ãå©çšãã EC2 ã€ã³ã¹ã¿ã³ã¹ãèµ·åã§ããããã«ãªããŸãã äžèš AMI ãçšã㊠EC2 ã€ã³ã¹ã¿ã³ã¹ãèµ·åããåŸã«ã¯ãã€ã³ã¹ã¿ã³ã¹ã«ãã°ã€ã³ãè¡ããã¬ããªã±ãŒã·ã§ã³ãè¡ãããã®æ§æãè¡ããŸããæ§æã®ããã®è©³çްæé ã¯ã Precisely Connect ã®æ§ææé ã®ããã¥ã¡ã³ã ãåç
§ããŠãã ããã æ¬ããã°ã§ã¯ãApply Engine 以éã®åŠçã®ç¢ºèªã«çŠç¹ãåœãŠããããã¡ã€ã³ãã¬ãŒã ç°å¢ã¯äœ¿çšããã«ãEC2 ã€ã³ã¹ã¿ã³ã¹äžã«é
眮ãããµã³ãã«ãã¡ã€ã«ãããŒã¿ãœãŒã¹ãšããŠãåŸç¶ã®ã¿ãŒã²ããã«å¯ŸããŠã¬ããªã±ãŒã·ã§ã³ãè¡ãã¢ãã©ã€ã¹ã¯ãªãããã玹ä»ããŸãã [å£²äžæçŽ°ã®ãµã³ãã« (CSV)] ããŒã¿ãœãŒã¹ãšããŠäœ¿çšãã CSV ã®ååã®æå³ã¯ store_id: åºèID, product_id: ååID, amount: æ°é, unit_price: å䟡, total_price: åèšéé¡, timestamp: å£²äžæ¥æ ã§ãããå®éã®ãã¡ã€ã«ã¯ä»¥äžãšãªããŸãã 3,1008,6,1000,credit,2024-04-01 09:39:00,6000 3,1005,2,200,mobile,2024-04-01 14:00:00,400 2,1004,3,2000,credit,2024-04-03 11:19:00,6000 5,1001,9,200,mobile,2024-04-03 12:17:00,1800 4,1009,8,500,cash,2024-04-03 17:29:00,4000 4,1007,10,100,debit,2024-04-03 20:24:00,1000 1,1004,4,100,mobile,2024-04-04 13:22:00,400 2,1006,1,200,mobile,2024-04-05 14:02:00,200 4,1002,8,1500,mobile,2024-04-05 15:55:00,12000 4,1001,1,1000,cash,2024-04-05 17:35:00,1000 ... [ãµã³ãã«ã®ã¢ãã©ã€ã¹ã¯ãªãã] å®éã«äœ¿çšããã¢ãã©ã€ã¹ã¯ãªããã¯ä»¥äžã®éãã§ãã JOBNAME sales_test_job; REPORT ./SALES_TEST_JOB.rpt; BEGIN GROUP SOURCE; DESCRIPTION SQLDDL /+ CREATE TABLE SALES ( STORE_ID INTEGER, PRODUCT_ID INTEGER, AMOUNT DECIMAL(10), UNIT_PRICE DECIMAL(10), TOTAL_PRICE DECIMAL(10), SALES_DATETIME TIMESTAMP ) +/ AS SALES; END GROUP; BEGIN GROUP TARGET; DESCRIPTION SQLDDL /+ CREATE TABLE T_SALES ( STORE_ID INTEGER, PRODUCT_ID INTEGER, AMOUNT DECIMAL(10), UNIT_PRICE DECIMAL(10), TOTAL_PRICE DECIMAL(10), SALES_DATETIME TIMESTAMP ) +/ AS T_SALES; END GROUP; DATASTORE ./input.csv OF DELIMITED COLDEL('\x2C') RECDEL('\x0A') ASCII AS FILE_IN DESCRIBED BY GROUP SOURCE; DATASTORE kafka:///<topicå> OF DELIMITED COLDEL('\x2C') RECDEL('\x0A') ASCII AS FILE_OUT DESCRIBED BY GROUP TARGET; PROCESS INTO FILE_OUT SELECT { REPLICATE(FILE_OUT) } FROM FILE_IN; MSK ã®æºå 次ã«ãApply Engine ããéä¿¡ãããããŒã¿ãåŠçãã MSK éšåãæ§ç¯ããŸããæ§ç¯ã®æåã®ã¹ãããã¯ããããžã¡ã³ãã³ã³ãœãŒã«ã§ãMSK ã®ã³ã³ãœãŒã«ç»é¢ã«ç§»åããŠãMSK ã¯ã©ã¹ã¿ãŒãäœæããããšããå§ãŸããŸããApply Engine ãš MSK ã¯ã©ã¹ã¿ãŒãå®å
šã«éä¿¡ããããã«ãä»åã¯åã VPC å
ã«äž¡è
ãååšããæ§æãæ¡çšããŸããã¯ã©ã¹ã¿ãŒã¿ã€ã㯠Provisioned ãéžã³ãApply Engine ã® VPC ãžé
眮ããŠãããŸããApache Kafka ããŒãžã§ã³ã¯èŠä»¶ã«ãã£ãŠå€ãããŸãããä»åã¯ç¹æ®èŠä»¶ã¯æ³å®ããªãããææ°çãéžã³ãŸãã ãããŒã«ãŒã¯ Standard ãéžæãããããŒã«ãŒãµã€ãºã¯ãåŸç¶åŠçãšã㊠VPC ãã©ã€ããŒãæ¥ç¶ãå¿
èŠãª Amazon Data Firehose ãå©çšãããããVPC ãã©ã€ããŒãæ¥ç¶ãå©çšå¯èœãª m5.large ã€ã³ã¹ã¿ã³ã¹ãå©çšããŸãããããŒã«ãŒã®ãŸãŒã³æ°ã¯é
眮å
ã® VPC ãšåãããŠæ°ãéžæããŸããä»å㯠2 ãå©çšããŸãããŸãŒã³ãããã®ãããŒã«ãŒæ°ã¯ã1ã€ãéžæããŸããèŠä»¶ã«å¿ããŠäžã€ã®ãŸãŒã³ (AZ) ã«è€æ°ã®ãããŒã«ãŒãé
眮ããããšãå¯èœã§ããä»åã¯ãã¯ã©ã¹ã¿ãŒã®ã¹ãã¬ãŒãžã¯ããã©ã«ãå€ãå©çšããŸãã ç¶ããŠããããã¯ãŒã¯èšå®ãè¡ããŸãããŸãã¯å¯Ÿè±¡ãšã㊠Apply Engine ãå±éããã VPC ãéžæããŸãããããŠãæåã®ãŸãŒã³ããšããŠãæå®ãã AZ ããããã«å¯Ÿå¿ãããµãããããéžæããŸããã»ãã¥ãªãã£ã°ã«ãŒãã«ã€ããŠã¯ãApply Engine ãã¹ããš EC2 ã®äž¡æ¹ã«åäžãªã»ãã¥ãªãã£ã°ã«ãŒãã䜿çšããããKafka ãå©çšããããŒããéæŸããæ¥ç¶ã»çéãå¯èœãªã»ãã¥ãªãã£ã°ã«ãŒããæºåããŠæå®ããŠãã ããã é¢é£ãããªãœãŒã¹ã MSK ã«ã¢ã¯ã»ã¹ã§ããããã«ãå¿
èŠãªæš©éèšå®ãèš±å¯ããŸããä»åã¯ä»¥äžã®èšå®ã䜿çšããŸãã Apply Engine ãšã®æ¥ç¶ã¯ãŠãŒã¶ãŒåãšãã¹ã¯ãŒããçšãããããSASL/SCRAM èªèšŒãæå¹ã«ããã MSK ãã Data Firehose ãžæ¥ç¶ãããããIAM ããŒã«ããŒã¹ã®èªèšŒãæå¹ã«ããã æ¬¡ã«ã¢ãã¿ãªã³ã°ã«é¢ããèšå®ãããŠãã¯ã©ã¹ã¿ãŒããããã€ããŸããã¯ã©ã¹ã¿ãŒã®ç«ã¡äžã㯠æ°ååçšåºŠãããå ŽåããããŸããç«ã¡äžããå®äºããŠãããæ¥ç¶èš±å¯ã®å€æŽããã³ Apply Engine ããããŒã¿ã飿ºããããã«ãMSK ã®ãããã¯ãäœæãèšå®ããŸãã MSK ã§ãããã¯ãäœæããæ¹æ³ã¯ããã€ããããŸãããä»å㯠Amazon MSK ã®Developer Guide ã«åŸã£ãŠãã¯ã©ã€ã¢ã³ããšããŠå©çšããŠãã EC2 ã€ã³ã¹ã¿ã³ã¹ãã Kafka ãæäœããæ¹æ³ãæ¡çšããŸãã詳现ã®èšå®æé ã¯å Developer Guide ã® ãã¡ã ãåèããŠãã ãããå¿
èŠãªäœæ¥ã¹ãããæŠèŠã¯ä»¥äžãšãªããŸããããã§äœæãããããã¯æ
å ±ã¯ãåŸç¶ã®éåä¿¡èšå®ã§äœ¿çšããŸãã MSK ã¯ã©ã¹ã¿ãŒãæäœããæš©éã®ãã IAM ããŒã«ãäœæããã äžèšæäœæš©éãæã€ããŒã«ãä»äžããã EC2 ã€ã³ã¹ã¿ã³ã¹ãç«ã¡äžãããMSK ã¯ã©ã¹ã¿ãŒãšçéã§ããããã«ãã»ãã¥ãªãã£ã°ã«ãŒããªã©ãèšå®ãããïŒApply Engine ããã¹ãããã EC2 ã§ä»£çšå¯èœïŒ EC2 ã«ãã°ã€ã³ããŠãMSK ã¯ã©ã¹ã¿ãŒã§å±éããã Kafka ã®ããŒãžã§ã³ã«å¯Ÿå¿ãã Kafka ããã±ãŒãžãããŠã³ããŒãããŠèšå®ããã çéã確èªããŠãããMSK ã¯ã©ã¹ã¿ãŒã® BootstrapServer ã«å¯ŸããŠãApply Engine ã®åºå飿ºçšã®ãããã¯ãäœæããã æåŸã«ã以äžå³ã®ããã«MSK ã¯ã©ã¹ã¿ãŒã®ãããã¯ãŒã¯èšå®ã§ãã«ã VPC æ¥ç¶ãæå¹ã«ããŠãã¯ã©ã¹ã¿ãŒããªã·ãŒã§ã Data Firehose ããã®æ¥ç¶ãèš±å¯ããŸãã Apply Engine ãã MSK ãžã®æ¥ç¶ MSK ã¯ã©ã¹ã¿ãŒã®èšå®ã®åŸã«ã¯ãå®éã« Apply Engine ãã MSK ã®éã§ããŒã¿éåä¿¡ã®ç¢ºèªãè¡ããŸããå
·äœçã«ã¯ãApply Engine ã®èšå®æã«å©çšããããŒã¿å€æã¹ã¯ãªãããæžãæããŠå®éã®éä¿¡ãè¡ããŸããèšå®å€æŽã®ãã€ã³ãã¯ãéä¿¡å
ãæå®ãã TARGET DATASTORE ã®ã»ã¯ã·ã§ã³ã§ãDATASTORE ã®éšåããããã¯åïŒ kafka:///<topicå> ïŒã«æå®ããããšã§ããå®éã®ã¹ã¯ãªããäŸã¯ä»¥äžãšãªããŸãããªããä»åã¯ããŒã¿ã®ãã©ãŒããã倿ã¯è¡ãããçŽæ¥ MSK ãžéä¿¡ããŸãã DATASTORE kafka:///<topicå> OF DELIMITED COLDEL('\x2C') RECDEL('\x0A') ASCII AS FILE_OUT DESCRIBED BY GROUP TARGET; 次ã«ãMSK åŽã«ãŠ Apply Engine ã®èªèšŒèšå®ãè¡ããŸããApply Engine ãå©çšãããŠãŒã¶ãŒåãšãã¹ã¯ãŒãã以äžå³ã®ããã« Secrets Manager ã«ä¿ç®¡ããŸããæå·åã«ã¯ã«ã¹ã¿ã ããŒã«ããæå·åãå©çšããŸããã«ã¹ã¿ã ããŒã¯ AWS Key Management Service (KMS) ãçšããŠäœæã§ããŸããAWS ãããŒãžãããŒãå©çšãããš MSK ã§ã¯é¢é£ä»ãã§ããŸããã®ã§ã泚æãã ãããå
·äœçã«ã¯ã·ãŒã¯ã¬ãããäœæããŠããã以äžå³ã®ããã«ãMSK ã¯ã©ã¹ã¿ãŒã®ã»ãã¥ãªãã£èšå®ã§é¢é£ä»ããè¡ããŸãããªããã·ãŒã¯ã¬ãããäœæãããšããåœåã«ãŒã«ãšããŠãAmazonMSK_ã®ãã¬ãã£ãã¯ã¹ãå¿
é ã§ãã æåŸã«ãApply Engine ããã¹ããããã€ã³ã¹ã¿ã³ã¹ã§ãApply Engine ã®èšå®ã倿ŽããŸããApply Engine ã®ã¯ãŒãã³ã°ãã£ã¬ã¯ããªäžã§ãããŒã¿éä¿¡çšã®èšå®ãã¡ã€ã« ( sqdata_kafka_producer.conf ) ããäœæããç°å¢ã«åãããŠä»¥äžã®ããã«æžãæããŸãã builtin.features=SASL_SCRAM security.protocol=SASL_SSL sasl.mechanism=SCRAM-SHA-512 sasl.username=<ã·ãŒã¯ã¬ããã§ä¿ç®¡ããã username> sasl.password=<ã·ãŒã¯ã¬ããã§ä¿ç®¡ããã password> metadata.broker.list=<MSK ã¯ã©ã¹ã¿ãŒã® broker URL> ãã®åŸ Apply Engine ã§ããŒã¿å€æã¹ã¯ãªããã以äžã®ã³ãã³ãã§ã³ã³ãã€ã«ããŠå®è¡ããã°ãMSK ãããã¯ã«å¯ŸããŠããŒã¿éä¿¡ãè¡ãããŸãã sqdparse <JOB_SCRIPT>.sqd <JOB_SCRIPT>.prc sqdata <JOB_SCRIPT>.prc Data Firehose ãéãã S3 ãžã®æ¥ç¶ïŒã·ããªãªäŸ1ïŒ ãããŸã§ã®äœæ¥ã«ãããã¡ã€ã³ãã¬ãŒã ã®ããŒã¿ã MSK ãããã¯ã«ãã¢ãªã¢ã«ã¿ã€ã ã§éä¿¡ãããããã«ãªããŸãããåŸæ®µã®æ§æãšããŠãŸãã¯ãã·ããªãªäŸ1ãå®çŸããããã«ãMSK ã§åä¿¡ãããããŒã¿ããData Firehose ãéã㊠S3 ããŒã¿ã¬ã€ã¯ã«é£æºããæ§ææ¹æ³ãã玹ä»ããŸããMSK ããçŽæ¥ S3 ãµãŒãã¹ãžããŒã¿ã飿ºããããšãã§ããŸãããä»å㯠S3 ã«æ ŒçŽããããŒã¿ãæè»ã«å å·¥ã§ããããšãéèŠã㊠Data Firehose ã«æ¥ç¶ããäžã§ S3 ã«é£æºããæ§æãå©çšããŸããData Firehose ãå©çšããããšã§èªåçãªããŒã¿ã®å§çž®ãæå·åãããŒãã£ã·ã§ã³åå²ãå®çŸããããšãå¯èœã§ãã IAM ããŒã«ãšããªã·ãŒã®äœæ Data Firehose ã MSK ããã€ãã³ããååŸããããã«ããŸã㯠MSK ã¯ã©ã¹ã¿ãŒããããã¯ãã°ã«ãŒããžã¢ã¯ã»ã¹ã§ãã IAM ããŒã«ãšããªã·ãŒãäœæããŸããåæã«ãéä¿¡å
ã§ãã S3 ãã±ãããžã®ã¢ã¯ã»ã¹æš©éãä»äžããŸããIAM ããŒã«äŸã¯ä»¥äžãšãªããŸãã { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "kafka-cluster:Connect", "kafka-cluster:AlterCluster", "kafka-cluster:DescribeCluster" ], "Resource": "arn:aws:kafka:region:012345678901:cluster/cluster-name/*" }, { "Effect": "Allow", "Action": [ "kafka-cluster:*Topic*", "kafka-cluster:WriteData", "kafka-cluster:ReadData" ], "Resource": "arn:aws:kafka:region:012345678901:topic/cluster-name/*" }, { "Effect": "Allow", "Action": [ "kafka-cluster:AlterGroup", "kafka-cluster:DescribeGroup" ], "Resource": "arn:aws:kafka:region:012345678901:group/cluster-name/*" }, { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:GetBucketLocation", "s3:AbortMultipartUpload" ], "Resource": [ "arn:aws:s3:::your-bucket-name", "arn:aws:s3:::your-bucket-name/*" ] } ] } MSK ã¯ã©ã¹ã¿ãŒããªã·ãŒã®èšå® MSK åŽã®ã¯ã©ã¹ã¿ãŒããªã·ãŒã§ã Data Firehose ããã®ã¢ã¯ã»ã¹ãèš±å¯ããŠããå¿
èŠããããŸããå
·äœçãªèšå®æ¹æ³ã¯ã Amazon MSK ã® Firehose çµ±å ã Amazon MSK ã®ãœãŒã¹èšå®ãæ§æãã ãåç
§ãã ããã Data Firehose ã®äœæ 次ã«ãããŒã¿ãœãŒã¹ã MSK ã¯ã©ã¹ã¿ãŒãéä¿¡å
ã S3 ãã±ãããšãã Data Firehose ã®äœæãè¡ããŸããæé ã¯ä»¥äžã®éãã§ãã Amazon Data Firehose ã³ã³ãœãŒã«ã§ãCreate delivery streamããéžæ Source ãšããŠãAmazon MSKããéžæ MSK ã¯ã©ã¹ã¿ãŒããããã¯åãéå§äœçœ®ãæå® Destination ãšããŠãAmazon S3ããéžæ S3 ãã±ããåãšãã¬ãã£ãã¯ã¹ãèšå® äœæãã IAM ããŒã«ãæå® èšå®ã®å®äºåŸã«ç¶æ
ãã¢ã¯ãã£ãã«ãªãã°ãApply Engine ãã MSK ãš Data Firehose ãçµç±ã㊠S3 ãžã®ããŒã¿éä¿¡ãå¯èœã«ãªããŸããApply Engine ã§ããŒã¿éä¿¡ã宿œããã°ãéä¿¡å
ã® S3 ãã±ããã§æ°ãªããžã§ã¯ããäœæãããããšã確èªå¯èœã§ãã Redshift ãžã®çŽæ¥æ¥ç¶ïŒã·ããªãªäŸ2ïŒ æ¬¡ã«ãã·ããªãªäŸ2ãå®çŸããããã«ãMSK ã§åä¿¡ãããããŒã¿ã Amazon Redshift ã«çŽæ¥é£æº (ã¹ããªãŒãã³ã°çµ±å) ããæ§ææ¹æ³ãã玹ä»ããŸãããã®å ŽåãRedshift ã®ãããªã¢ã©ã€ãºããã¥ãŒã䜿çšããŠãã¢ãªã¢ã«ã¿ã€ã ããŒã¿åã蟌ã¿ãå®çŸããŸãã IAM ããŒã«ãšããªã·ãŒã®äœæ ãŸãæºåãšããŠãRedshift çšã® MSK ã¯ã©ã¹ã¿ãŒããããã¯ãã°ã«ãŒãã«ã¢ã¯ã»ã¹ããŠããŒã¿ãååŸããæš©éãæã€ IAM ããªã·ãŒãä»äžããã IAM ããŒã«ãäœæããŸããIAM ããŒã«äŸã¯ä»¥äžãšãªããŸãã { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "kafka-cluster:Connect", "kafka-cluster:AlterCluster", "kafka-cluster:DescribeCluster" ], "Resource": "arn:aws:kafka:region:012345678901:cluster/cluster-name/*" }, { "Effect": "Allow", "Action": [ "kafka-cluster:*Topic*", "kafka-cluster:WriteData", "kafka-cluster:ReadData" ], "Resource": "arn:aws:kafka:region:012345678901:topic/cluster-name/*" }, { "Effect": "Allow", "Action": [ "kafka-cluster:AlterGroup", "kafka-cluster:DescribeGroup" ], "Resource": "arn:aws:kafka:region:012345678901:group/cluster-name/*" } ] } Redshift ã¯ã©ã¹ã¿ãŒã®äœæ 次ã«äœæããããŒã«ãé©å¿æžã¿ã®ãæ¡åŒµããã VPC ã«ãŒãã£ã³ã°ãæå¹ã«ãã Redshift ã¯ã©ã¹ã¿ãŒãäœæããŠãå©çšå¯èœãªç¶æ
ã«ããŸããRedshift ã¯ã©ã¹ã¿ãŒã®äœæã®è©³çްã«ã€ããŠã¯ Amazon Redshift 管çã¬ã€ã ãåç
§ããŠãã ããã ã¹ããŒããšãããªã¢ã©ã€ãºããã¥ãŒã®äœæ Redshift ãå©çšå¯èœãªç¶æ
ã«ãªã£ãŠãããéä¿¡ãããããŒã¿ã®åãç¿ãšãªãããŒã¿ããŒã¹ãäœæããŸããã¯ãšãªãšãã£ã¿ãŒãªã©ã§äœæããããŒã¿ããŒã¹ãžã¢ã¯ã»ã¹ããŠãããŒã¿ãåãå
¥ããå€éšã¹ããŒããäŸãã°ä»¥äžã® SQL ã§äœæããŸãã CREATE EXTERNAL SCHEMA msk_schema FROM MSK IAM_ROLE 'arn:aws:iam::012345678901:role/MSK_access_role' AUTHENTICATION IAM URI 'b-1.<clustername>.123456.c2.kafka.<region>.amazonaws.com:9098,b-2.<clustername>.123z8u.c2.kafka.<region>.amazonaws.com:9098' 次ã«ãåä¿¡ããããŒã¿ãéçŽãããããªã¢ã©ã€ãºããã¥ãŒãäœæããŸããAUTO REFRESH ãæå¹ã«ããããšã§ãMSK ããã®æ°ããããŒã¿ãèªåçã«åæ ãããŸãããããŸã§ã§ããã° MSK ãš Redshift ã®éã§ããŒã¿é£æºãè¡ãããšãå¯èœã§ãã CREATE MATERIALIZED VIEW realtime_data_view AUTO REFRESH YES AS SELECT * FROM msk_schema."<topic_name>"; ããŒã¿å€æãšããžãã¹åæçšãã¥ãŒã®äœæ å®éã«ããŒã¿é£æºãè¡ããšãäœæãããã¥ãŒã®äžã§ã¯ãkafka_value ã®ã«ã©ã ã«åä¿¡ããã¬ã³ãŒãã®ããŒã¿ããã®ãŸãŸä¿ç®¡ãããŠããããšããããããšæããŸãããã®ãŸãŸã§ã¯ããžãã¹åæã«å¿
èŠãªããŒã¿åç
§ãããã«ãããããåä¿¡ããããŒã¿ãæŽåœ¢åŠçãããããªã¢ã©ã€ãºããã¥ãŒãæ¹ããŠäœæããŸããä»åã®ãµã³ãã«ããŒã¿ã§å£²äžããŒã¿ãéèšããããã«ã以äžã®ãããªå€æãè¡ããŸãã CREATE MATERIALIZED VIEW sales_data_view DISTKEY(store_id) SORTKEY(product_id) AS SELECT NULLIF(TRIM(SPLIT_PART(kafka_value::VARCHAR, ',', 1)), '')::INT AS store_id, NULLIF(TRIM(SPLIT_PART(kafka_value::VARCHAR, ',', 2)), '')::INT AS product_id, NULLIF(TRIM(SPLIT_PART(kafka_value::VARCHAR, ',', 3)), '')::INT AS amount, NULLIF(TRIM(SPLIT_PART(kafka_value::VARCHAR, ',', 4)), '')::INT AS unit_price, NULLIF(TRIM(SPLIT_PART(kafka_value::VARCHAR, ',', 5)), '')::INT AS total_price, TRIM(SPLIT_PART(kafka_value::VARCHAR, ',', 6)) AS timestamp FROM msk_schema."<topic_name>" WHERE kafka_value IS NOT NULL AND kafka_value::VARCHAR != '' å®éã«ãã¥ãŒãšããŠç¢ºèªã§ããããŒã¿å
容ã¯ä»¥äžã®éãã§ãã ãã¢ãªã¢ã«ã¿ã€ã åæã®å®è¡ ãã®äžã§ãäœæãããã¥ãŒã䜿çšããŠããã¢ãªã¢ã«ã¿ã€ã ã§ã®å£²äžåæãç°åžžæ€åºã¯ãšãªãå®è¡ã§ããŸãã -- çŽè¿1æéã®åºèå¥å£²äžéèš SELECT store_id, SUM(total_price) as hourly_sales FROM sales_data_view WHERE timestamp >= DATEADD(hour, -1, GETDATE()) #åºæºæ¥ã'2024-04-01 00:00:00'ã®ããã«ãæå®ã§ãã GROUP BY store_id ORDER BY hourly_sales DESC; ãã®æ§æã«ãããã¡ã€ã³ãã¬ãŒã ã§ã®ååŒçºçããæ°å以å
ã« Redshift ã§ã®åæçµæãååŸã§ããAmazon Quick Sight ãªã©ã® BI ããŒã«ãšé£æºããŠãã¢ãªã¢ã«ã¿ã€ã ããã·ã¥ããŒããæ§ç¯ããããšãå¯èœã§ãã æ§æã®äœ¿ãåã ããã§ã代衚çãª2çš®é¡ã®æ§æã玹ä»ããŸããããããããã«é©ãããŠãŒã¹ã±ãŒã¹ã¯æ¬¡ã®ããã«ããŸãã Data Firehose ãçµç±ãã S3 ãžã®é£æºæ§æ é·æä¿åãæ©æ¢°åŠç¿ã倧容éããŒã¿åŠçãå¿
èŠãªå Žå Redshift ãžã®çŽæ¥é£æºæ§æ ãã¢ãªã¢ã«ã¿ã€ã åæãããã·ã¥ããŒã衚瀺ãå³åº§ã®æææ±ºå®æ¯æŽãå¿
èŠãªå Žå çšéã«å¿ããŠäž¡æ¹ã®é£æºå
ãåæã«èšå®ããããšãå¯èœã§ãå
æ¬çãªããŒã¿æŽ»çšåºç€ãæ§ç¯ã§ããŸããMSK ãš Redshift ã«ã€ããŠãã詳现ã«ç¥ãããæ¹ã¯ã以äžã®ããã¥ã¡ã³ããåç
§ããŠãã ããã Amazon Managed Streaming for Apache Kafka ããããããŒã¬ã€ã: Amazon MSK ã® Amazon Redshift ã¹ããªãŒãã³ã°ããŒã¿ã€ã³ãžã§ã¹ã Amazon Redshift ããŒã¿ããŒã¹éçºè
ã¬ã€ã: ãããªã¢ã©ã€ãºããã¥ãŒãžã®ã¹ããªãŒãã³ã°åã蟌㿠Amazon Redshift ããŒã¿ããŒã¹éçºè
ã¬ã€ã: Amazon Kinesis Data Streams ããã¹ããªãŒãã³ã°åã蟌ã¿ãéå§ããæ¹æ³ ãããã« æ¬ããã°ã§ã¯ãPrecisely ãçšãã AWS Mainframe Modernization Data Replication ã«ãããã¡ã€ã³ãã¬ãŒã ããŒã¿ã AWS ã¯ã©ãŠããžãã¢ãªã¢ã«ã¿ã€ã ã§åæããæ¹æ³ãã玹ä»ããŸããããã®æ§æã«ãããåŸæ¥ã®æ¥æ¬¡ãããåŠçã§ã¯å®çŸã§ããªãã£ãè¿
éãªããŒã¿æŽ»çšãå¯èœã«ãªããS3 ããŒã¿ã¬ã€ã¯ã§ã®æ©æ¢°åŠç¿æŽ»çšãã Redshift ã§ã®ãã¢ãªã¢ã«ã¿ã€ã åæãŸã§ã倿§ãªãŠãŒã¹ã±ãŒã¹ã«å¯Ÿå¿ã§ããŸããã¡ã€ã³ãã¬ãŒã ãžã®åœ±é¿ãæå°éã«æããªãããã¯ã©ãŠãã®æè»æ§ãšã¹ã±ãŒã©ããªãã£ã掻çšããããŒã¿ããªãã³ãªæææ±ºå®ãå®çŸããããšã§ãããžãã¹äŸ¡å€ã®åäžã«è²¢ç®ããããšãå¯èœã§ãããã詳现ãªãçžè«ãå
·äœçãªå®è£
æ¯æŽã«ã€ããŠã¯ãAWS Professional Services ãŸã§ãæ°è»œã«ãåãåãããã ããã åèæ
å ± AWS marketplace â precisely software Precisely å
¬åŒããã¥ã¡ã³ã èè
ã«ã€ã㊠å®è€ äº®äž (ANDO Ryoichi) å®è€ 亮äžã¯ãAWS Japan ãããã§ã·ã§ãã«ãµãŒãã¹ããŒã ã®ããªããªã³ã³ãµã«ã¿ã³ãã§ããããŒã¿ãã©ãããã©ãŒã ã®æ§ç¯ãæ€èšãããŠããã客æ§ã®æ§æ³çå®ã®æ¯æŽããçæ AI ã®ããã®ããŒã¿ãã©ãããã©ãŒã ã®æ§æ³çå®ã»èšèšæ§ç¯ã®æ¯æŽãªã©ãè¡ã£ãŠããŸããããŒã¿åºç€ã®æ§ç¯ã婿޻çšã§ãå°ãã®ããšãããã°ããã²ãçžè«ãã ããã è³ é²å£ (Yunjian He) è³ é²å£ã¯ãAWS Japan ãããã§ã·ã§ãã«ãµãŒãã¹ããŒã ã®ããªããªã³ã³ãµã«ã¿ã³ãã§ããAWSèªå®è©Šéšå
šååŸã§ãŽãŒã«ãã³ãžã£ã±ãããä¿æããŠããŸãã é«ãšãã«ã®ãŒç©çåŠåéã§å士ïŒçåŠïŒã®åŠäœãååŸãå€§èŠæš¡ããŒã¿ã®åæã§æ©ãã§ããéå»ã®èªèº«ã®ãããªã客æ§ã«åœ¹ç«ã¡ãããšæã£ãŠAWSã«å
¥ç€ŸããŸãããæè¿ã¯çæAI婿޻çšã®ãã©ãããã©ãŒã æ§ç¯æ¯æŽãããžã§ã¯ãã«åç»ããããŒã¿ã«é¢ããåŽé¢ã§ã客æ§ã®æ¯æŽãè¡ã£ãŠããŸãã åå· è£ä» (KITAGAWA Yusuke) åå· è£ä»ã¯ãAWS Japan ãããã§ã·ã§ãã«ãµãŒãã¹ããŒã ã®ããªããªã³ã³ãµã«ã¿ã³ãã§ããäž»ã«ã¡ã€ã³ãã¬ãŒã ã®æŽ»çšãç§»è¡ã«é¢ããæŽ»åãæ¯æŽããŠããŸãã