çããããã«ã¡ã¯ãOpenShift Virtualization ãšã¹ãã¬ãŒãžãçæ¥ã«ããŠãã Red Hat ã®ãã€ãŒïŒå®éœå®®ïŒã§ãã ãNetApp ã³ã©ãç¹éãVirt å±ã®ã²ãšãããš : 第3åã¯æ§èœèšèšã§ãã ååã®èšäº ã§ã¯ ontap-nas ãš ontap-san ã§ VM ã®ããŒã¿ãã£ã¹ã¯ãäœæãããšãã®å®¹éæ¶è²»ã®éããæ¯èŒããŸããããä»åã¯ããã«èžã¿èŸŒãã§ãåããã£ã¹ã¯ã«åãã¯ãŒã¯ããŒããããããšã IO ããã©ãŒãã³ã¹ã«ã©ãã ãå·®ãåºãã®ãã fio ãã³ãããŒã¯ã§æž¬å®ããŸãã å ããŒã 第1å ã¹ãã¬ãŒãžåºæ¬èšèšãTrident ã®ãã©ã€ãéžå® 第2å 容éèšèš 第3åïŒæ¬èšäºïŒ æ§èœèšèš 第4å VM ç§»è¡ïŒShift ToolkitïŒ ç¬¬5å ãã«ãããã³ãèšèš 第6å ããŒã¿ä¿è· 第7å DR èšèš 第8å ç£èŠãšéçš VMware 管çè
ã®ã¿ãªãã㞠枬å®çµæã«ã€ããŠã®ãæã æ€èšŒç°å¢ æž¬å®ææ³ã®æ¹åç¹ æ§èœæ€èšŒ 1 : ãããã¯ãµã€ãº æ§èœæ€èšŒ 2 : 䞊å床ïŒiodepthïŒ nconnect=8 ã®å¹æãš SAN/NAS ã®æ§èœå·®ã®èŠå ã«é¢ããèå¯ æ§èœæ€èšŒ 3 : è€æ° VM 䞊å ãŸãšããšéžå®ã¬ã€ã ããŸã : ãã³ãããŒã¯äžã«èµ·ãããã©ãã« VMware 管çè
ã®ã¿ãªãããž VMware ç°å¢ã§ã¯ãNFS ããŒã¿ã¹ãã¢ãš VMFSïŒiSCSI / FCïŒããŒã¿ã¹ãã¢ã®æ§èœå·®ãæ°ã«ããŠãã管çè
ã¯å€ããšæããŸãããNFS ã¯æè»œã ãã©æ§èœã¯ VMFS ã«å£ãããVMFS ã¯éããã©ç®¡çãæéããšãã£ãè°è«ã¯ãVMware ã®äžçã§ã¯å®çªã®è©±é¡ã§ããã OpenShift Virtualization + ONTAP ã§ãåãçåããããŸããNFS ããŒã¹ã® ontap-nas ãš iSCSI ããŒã¹ã® ontap-san ã§ãVM ã®ãã£ã¹ã¯æ§èœã«ã©ããããå·®ãåºãã®ããVMware ã®ãšããšåããããªåŸåãªã®ãããããšã Kubernetes ãªãã§ã¯ã®éããããã®ããä»åã¯ãã®çåã« fio ãã³ãããŒã¯ã§çããŸãã æž¬å®çµæã«ã€ããŠã®ãæã æ¬èšäºã®æž¬å®çµæã¯ã ç¹å®ã® 1 æ§æïŒAFF C190 HA ãã¢ã10GbE x 4ãSSD 12 æ¬ïŒã§ã1 VM ãŸãã¯å°æ° VM ãã fio ãå®è¡ããçµæ ã§ããã¹ãã¬ãŒãžã®ã¢ãã«ããã£ã¹ã¯æ§æããããã¯ãŒã¯åž¯åãVM æ°ãå€ããã°çµæãå€ãããŸãã ontap-nas ãš ontap-san ã®çžå¯ŸçãªåŸåãæŽãããã®åèå€ãšããŠèªãã§ãã ããããNFS ã¯é
ãããiSCSI ãéãããšããäžè¬è«ãå°ããã®ã§ã¯ãããŸããã ãªããQoS ã«ããæ§èœç®¡çã«ã€ããŠã¯ç¬¬5åïŒãã«ãããã³ãèšèšïŒã§ãæ§èœã®ç£èŠæ¹æ³ã«ã€ããŠã¯ç¬¬8åïŒç£èŠãšéçšïŒã§è©³ããæ±ããŸãã æ€èšŒç°å¢ ãã®èšäºãäœãäžã§ããåããã NetApp æ©åšã OpenShift ã¯ã©ã¹ã¿ã«æ¥ç¶ããæ€èšŒç°å¢ãæ§ç¯ããŠåäœã確èªããŸããã以äžã«ã¯ç°å¢æ
å ±ãèšèŒããŸãã NetApp æ©åšããã³ Trident é
ç® å€ ã¹ãã¬ãŒãžã¢ãã« NetApp AFF C190 x 2ïŒHA ãã¢ïŒ ONTAP ããŒãžã§ã³ 9.18.1P2 Trident ããŒãžã§ã³ 26.02 ãã£ã¹ã¯ SSD x 12 æ¬ïŒã¢ãã«: X382_S1643960ATE / 894.3GBïŒ ããŒãã£ã·ã§ãã³ã° ADPïŒAdvanced Disk PartitioningïŒæå¹ ããŒã¿ã¢ã°ãªã²ãŒã RAID-DP x 2ïŒåããŒã 1 ã€ïŒãåã¢ã°ãªã²ãŒã㯠12 æ¬å
šãã£ã¹ã¯ã®ããŒã¿ããŒãã£ã·ã§ã³ïŒ10 data + 2 parityïŒã§æ§æ ã¢ã°ãªã²ãŒã容é çŽ 3.65TB / ããŒãïŒããŒãã£ã·ã§ã³ããã Usable 415.8GBïŒ SVM svm-openshift ïŒäž¡ããŒãã®ã¢ã°ãªã²ãŒãã䜿çšïŒ ãããã¯ãŒã¯ 10GbE x 4ïŒiSCSI / NFS å
±éïŒ AFF C190 ã¯ãšã³ããªãŒã¯ã©ã¹ã® All-Flash ã¢ãã«ã§ãã12 æ¬ã®ç©ç SSD 㯠ADP ã«ãã root ããŒãã£ã·ã§ã³ãšããŒã¿ããŒãã£ã·ã§ã³ã«åå²ãããåããŒãã® root ã¢ã°ãªã²ãŒãïŒ aggr0 ïŒãšããŒã¿ã¢ã°ãªã²ãŒãã®äž¡æ¹ã«ããŒãã£ã·ã§ã³ãæäŸããŠããŸããSVM svm-openshift ã¯äž¡ããŒãã®ããŒã¿ã¢ã°ãªã²ãŒãïŒ sr_c190rj_01_SSD_1 , sr_c190rj_02_SSD_1 ïŒã䜿çšããŠããŸãã å¹³ããèšãã°ãã©ã¡ãã®ããŒã¿ã¢ã°ãªã²ãŒãã§äœããã FlexVol ã§ããããŒã¿ã¯ 12 æ¬ãã¹ãŠã® SSD ã« stripe ãããŸããã€ãŸã ontap-san ãš ontap-nas ã®ã©ã¡ãã® FlexVol ãã©ã¡ãã®ããŒãã«èŒã£ãŠããŠããããã¯ãšã³ãã®ãã£ã¹ã¯ IO æ§èœã¯åæ¡ä»¶ãšãªããŸãã OpenShift ã¯ã©ã¹ã¿ é
ç® å€ OpenShift ããŒãžã§ã³ 4.22.0 OpenShift Virtualization ããŒãžã§ã³ 4.22.2 Control Plane Node 3 x ä»®æ³ãã·ã³ïŒ8vCPU, 24Gi RAMïŒ Compute Node 3 x ãã¢ã¡ã¿ã«ãã·ã³ïŒ8c/16t CPU, 64Gi RAMïŒ ãããã¯ãŒã¯ ãããã¯ãŒã¯æ¥ç¶ã¯ä»¥äžïŒAll 10GbE, MTU=9000ïŒ ãã£ãããããã¯ãŒã¯å³ Trident ã®èšå® ontap-san ã® TBCïŒæç²ïŒ: defaults : snapshotPolicy : default snapshotReserve : "10" spaceReserve : volume spaceAllocation : "true" ontap-nas ã® TBCïŒæç²ïŒ: defaults : snapshotPolicy : default snapshotReserve : "10" spaceReserve : volume provisioningType : thin nfsMountOptions : nconnect=8 ontap-nas ã® TBC ã« nfsMountOptions: nconnect=8 ã远å ããŠããŸããVMware ç°å¢ã§ã NFS ããŒã¿ã¹ãã¢ã®ããŠã³ããªãã·ã§ã³ãã¥ãŒãã³ã°ïŒTCP ãŠã£ã³ããŠãµã€ãºã®èª¿æŽãªã©ïŒã¯è¡ãããŠããŸãããã nconnect 㯠Linux ã«ãŒãã«ã® NFS ã¯ã©ã€ã¢ã³ãæ©èœã§ãããKubernetes / Trident ç°å¢ã§ TBC ã® nfsMountOptions ã« 1 è¡è¿œå ããã ãã§æå¹åã§ããã®ãç¹åŸŽã§ãã广ã®è©³çŽ°ã¯æ¬¡ã®ã»ã¯ã·ã§ã³ã§èª¬æããŸãã VM ã®èšå® rhel10-ontap-san rhel10-ontap-nas OS RHEL 10 RHEL 10 rootdisk (30Gi) ontap-san ontap-san datadisk (500Gi) ontap-san ontap-nas volumeMode Block Filesystem ãããã³ã« iSCSI NFSïŒNFSv4.2ïŒ NFS ããŠã³ããªãã·ã§ã³ã®è©³çް ontap-nas ã® NFS ããŠã³ããªãã·ã§ã³ã¯ã ontap-nas ã® PVC ãä»®æ³ãã£ã¹ã¯ãšããŠäœ¿çšããŠãã VM ã皌åãã Compute Node äžã§ä»¥äžã®ã³ãã³ãã«ãã確èªã§ããŸãã $ oc debug node/wk-02 -- cat /host/proc/mounts | grep nfs 192.168.34.xxx:/trident_pvc_... /host/var/lib/kubelet/pods/.../mount nfs4 rw,relatime,vers=4.2,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,nconnect=8,timeo=600,retrans=2,sec=sys,clientaddr=192.168.34.xxx,local_lock=none,addr=192.168.34.xxx 0 0 ä»åã®æ€èšŒã§äœ¿çšããã NFS ããŠã³ããªãã·ã§ã³ã®äž»èŠãªå€ã¯ä»¥äžã®ãšããã§ãã ãªãã·ã§ã³ å€ èª¬æ vers 4.2 NFS ãããã³ã«ããŒãžã§ã³ rsize 65536 (64k) 1 åã® NFS READ RPC ã§è»¢éããæå€§ãã€ãæ° wsize 65536 (64k) 1 åã® NFS WRITE RPC ã§è»¢éããæå€§ãã€ãæ° nconnect 8 NFS ããŠã³ããããã® TCP ã³ãã¯ã·ã§ã³æ° proto tcp ãã©ã³ã¹ããŒããããã³ã« sec sys ã»ãã¥ãªãã£ã¢ãŒãïŒAUTH_SYSïŒ hard (æå¹) ãµãŒããŒç¡å¿çæã«ç¡æéãªãã©ã€ Trident ã® TBC ã§ã¯ nfsMountOptions: nconnect=8 ã®ã¿ãæå®ããŠããããã以å€ã®ãªãã·ã§ã³ïŒ vers=4.2 , rsize=65536 , wsize=65536 , hard , sec=sys çïŒã¯ Compute Node ãš ONTAP ã®ããŽã·ãšãŒã·ã§ã³ã§æ±ºãŸã£ãããã©ã«ãå€ã§ãã ãã ããæå³ããŠç¹å®ã®ãªãã·ã§ã³å€ïŒäŸãã° nfsvers=4.2 ãªã©ïŒã䜿ãããå Žåã¯ãããŽã·ãšãŒã·ã§ã³ä»»ãã«ãã TBC ã® nfsMountOptions ã StorageClass ã® mountOptions ã§æç€ºçã«æå®ããŠããã®ã劥åœã§ããç¹ã« rsize / wsize 㯠NFS ã®æ§èœã«çŽæ¥åœ±é¿ãããã©ã¡ãŒã¿ã§ãããåŸè¿°ã®èå¯ã§ãè§ŠããŸãã æ³šæ : Trident ããŒãžã§ã³ã«ã€ã㊠æ¬èšäºïŒç¬¬1åã第3åïŒã®æ€èšŒç°å¢ã§ã¯ Trident 26.02 ã䜿çšããŠããŸãããTrident 26.02 㯠OpenShift 4.22 ãå
¬åŒã«ã¯ãµããŒãããŠããŸãããæ€èšŒã®ç¯å²ã§ã¯åäœã«åœ±é¿ã¯ãããŸããã§ããããç§ã®äžæ³šæããè©«ã³ç³ãäžããŸããå®éã«ç°å¢ãæ§ç¯ãããéã¯ãå¿
ã Trident ã®ãµããŒããããªã¯ã¹ ã確èªããã䜿ãã® OpenShift ããŒãžã§ã³ã«å¯Ÿå¿ãã Trident ããŒãžã§ã³ã䜿çšããŠãã ããã æž¬å®ææ³ã®æ¹åç¹ å®ã¯ãã®èšäºãåçš¿ã§ã¯ããã©ã«ãã® NFS èšå®ïŒ nconnect ãªã = TCP 1 æ¬ïŒã§æž¬å®ããŠãã»ãŒå
šãã¿ãŒã³ã§ ontap-san ãæå©ããšçµè«ã¥ããŠããŸããããããããã®æ¡ä»¶ã¯ NFS ã«ãšã£ãŠãã§ã¢ãšã¯èšããŸããã§ãããæž¬å®ææ³ãèŠçŽããŠå枬å®ãè¡ã£ããšãããç¹ã«é« iodepth ãè€æ° VM ã®çµæã«åœ±é¿ããããŸãããèŠçŽãã 2 ç¹ãšãNFSv4.2 ã«é¢ããè£è¶³ã説æããŸãã 1. nconnect=8 ã®æå¹å NFS ã¯ããã©ã«ãã§ 1 æ¬ã® TCP ã³ãã¯ã·ã§ã³ã§ãã¹ãŠã® IO ãéåä¿¡ããŸãã10GbE ã®åž¯åããã£ãŠãã1 æ¬ã® TCP ã³ãã¯ã·ã§ã³ã§ã¯ CPU ã®ã·ã³ã°ã«ã¹ã¬ããåŠçãããã«ããã¯ã«ãªãããããNFS ã®äžŠå IO æ§èœãå¶éãããŸãã äžæ¹ iSCSI ã¯ãmultipath ãæ§æããŠããªããŠã iSCSI initiator å
éšã§è€æ°ã® IO ã䞊åã«çºè¡ã§ããŸããã€ãŸãããã©ã«ãèšå®åå£«ã®æ¯èŒã¯ NFS ã«ãšã£ãŠäžå©ãšèšããŸãã NFS ã® nconnect ããŠã³ããªãã·ã§ã³ã¯ã1 ã€ã® NFS ããŠã³ãã«å¯ŸããŠè€æ°ã® TCP ã³ãã¯ã·ã§ã³ã匵ããªãã·ã§ã³ã§ããããã«ãã IO ãè€æ°ã³ãã¯ã·ã§ã³ã«åæ£ã§ããNFS ã®äžŠå IO æ§èœã倧å¹
ã«åäžããŸããTrident ã§ã¯ backend ã® nfsMountOptions ã« nconnect=8 ãšèšå®ããã ãã§æå¹ã«ãªããŸãã VMware ã®äžçã§ã¯ãESXi ã® NFS ã¯ã©ã€ã¢ã³ãå®è£
ãç¬èªã§ãããTCP ã³ãã¯ã·ã§ã³æ°ã¯ãã€ããŒãã€ã¶ãŒåŽã§å¶åŸ¡ãããŠããŸãããOpenShift Virtualization ã§ã¯ Linux ã«ãŒãã«ã® NFS ã¯ã©ã€ã¢ã³ãããã®ãŸãŸäœ¿ãããããã nconnect ã®ãã㪠Linux ã«ãŒãã«ã®ããŠã³ããªãã·ã§ã³ããã®ãŸãŸå¹ããŸãããã㯠Kubernetes / Trident ç°å¢ãªãã§ã¯ã®æè»ãã§ãã æ€èšŒ 2 ã§ã¯ nconnect=8 ã® NAS ãš nconnect ãªãïŒ=1ïŒã® NAS ã®äž¡æ¹ã枬å®ãã nconnect ã®å¹æãåãããŠæ€èšŒããŠããŸãã 2. random_generator=lfsr ã®äœ¿çš fio ã®ããã©ã«ãã®ä¹±æ°çæåšã¯ãåããããã¯ãè€æ°åã¢ã¯ã»ã¹ããå¯èœæ§ããããŸããã©ã³ãã IO ã®ãã³ãããŒã¯ã§åäžãããã¯ã«åã¢ã¯ã»ã¹ãããšãONTAP ã® read cacheïŒDRAMïŒã«ãããããŠå®éã®ãã£ã¹ã¯æ§èœãããé«ã IOPS ãåºãŠããŸããŸãã random_generator=lfsr ïŒLinear Feedback Shift RegisterïŒã䜿çšãããšããã¹ããã¡ã€ã«å
ã®åãããã¯ãæ£ç¢ºã« 1 åã ãã¢ã¯ã»ã¹ãããŸããããã«ãã ONTAP ã® read cache ãžã®åããããæå¶ããã¹ãã¬ãŒãžã®å®æ§èœãããæ£ç¢ºã«æž¬å®ã§ããŸãã 3. NFSv4.2 ã®äœ¿çš ä»åã®æ€èšŒã§ã¯ NFS ã®ããŒãžã§ã³ãšã㊠NFSv4.2 ã䜿ãããŠããŸãããã㯠Trident ã ONTAP åŽã§æç€ºçã«ããŒãžã§ã³æå®ãããã®ã§ã¯ãªããCompute Node ãš ONTAP ã®ããŽã·ãšãŒã·ã§ã³ã§èªåçã«æ±ºãŸã£ãããŒãžã§ã³ã§ãã ãªããOpenShift Virtualization èªäœã¯ã¹ãã¬ãŒãžãããã³ã«ã«å¶çŽãèšããŠããŸããã Red Hat ã®ããã¥ã¡ã³ã ã«ã¯ã "there are no restrictions on storage protocol (e.g. NFS, iSCSI, FC, etc.) specific to OpenShift Virtualization. The only requirement is that the RWX access mode is available to support live migration of VMs within the cluster." ãšæèšãããŠãããLive migration ã®èŠä»¶ã RWX ã¢ã¯ã»ã¹ã¢ãŒãã§ãã£ãŠ NFS ããŒãžã§ã³ã§ã¯ãããŸãããNFSv3 ã§ã NFSv4 ã§ã ontap-nas ã§ããã° RWX ãæäŸã§ããLive migration ã¯åäœããŸãã ãã ã NFSv3 ã§æ¥ç¶ããå Žåã¯ãã¹ãã¬ãŒãžåŽã NLMïŒNetwork Lock Manager / lockdïŒããµããŒãããŠããå¿
èŠããããŸããLive migration äžã« QEMU ããã£ã¹ã¯ã€ã¡ãŒãžãã¡ã€ã«ã®ããã¯ãååŸã»è§£æŸãããããNLM ãå©çšã§ããªããšç§»è¡ã«å€±æããŸããNFSv4 ã§ã¯ããã¯æ©æ§ããããã³ã«ã«çµã¿èŸŒãŸããŠãããã NLM ã¯äžèŠã§ãããªã ONTAP 㯠NFSv3 + NLM ããµããŒãããŠããã®ã§ãONTAP ãããã¯ãšã³ãã«äœ¿ãå Žåã¯ã©ã¡ãã®ããŒãžã§ã³ã§ãåé¡ãããŸããã æ§èœé¢ã§ã¯ãNFSv4 ã«ã¯ NFSv3 ã«ã¯ãªãå©ç¹ããããŸããCompound Operations ã«ããè€æ°ã®æäœïŒLOOKUP + OPEN + READ ãªã©ïŒã 1 ã€ã® RPC ã«ãŸãšããŠéä¿¡ã§ããã©ãŠã³ãããªããæ°ãåæžãããŸãããŸã NFSv4.1 以éã§ã¯ã»ãã·ã§ã³ãã©ã³ãã³ã°ïŒSession TrunkingïŒããµããŒãããã nconnect ã§è€æ°ã³ãã¯ã·ã§ã³ã匵ã£ãéã®ç®¡çãå¹ççã«è¡ãããŸããNFSv3 ã䜿ãç©æ¥µçãªçç±ããªããã°ãNFSv4 以äžã§ã®ããŠã³ãããå§ãããŸãã äžæ¹ãNFSv4 ã«ã¯ Grace Period ãšããæ³šæç¹ããããŸããONTAP ã³ã³ãããŒã©ã® HA ãã§ã€ã«ãªãŒããŒãçºçãããšãNFSv4 ãµãŒããŒã¯ã¯ã©ã€ã¢ã³ããããã¯ãåååŸããããã®ç¶äºæéïŒGrace PeriodãONTAP ããã©ã«ã 45 ç§ïŒã«å
¥ããŸãããã®é READ/WRITE 㯠NFS4ERR_GRACE ã§æåŠãããã¯ã©ã€ã¢ã³ãåŽã§ã¯ IO ãäžæåæ¢ããŸããNFSv3 ã«ã¯ãã®ä»çµã¿ããªãããããã§ã€ã«ãªãŒããŒæã® IO åé㯠NFSv3 ã®æ¹ãéããªããŸããVMware ç°å¢ã§ã HA ãã§ã€ã«ãªãŒããŒæã® NFS ããŒã¿ã¹ãã¢ã®åæ¥ç¶ã¯éçšäžã®æ³šæç¹ã§ããããNFSv4 ã® Grace Period ã¯ãããšæ§è³ªã®ç°ãªãé
å»¶èŠå ã§ãããã³ãããŒã¯çµæã«ã¯åœ±é¿ããŸããããéçšäžã¯ç¥ã£ãŠããã¹ããã€ã³ãã§ãã æ§èœæ€èšŒ 1 : ãããã¯ãµã€ãº ãŸãã¯ã·ã³ãã«ã«ããããã¯ãµã€ãºãå€ãããšãã®æ§èœå·®ãèŠãŠã¿ãŸãã7 çš®é¡ã®ãããã¯ãµã€ãºïŒ4kã256kïŒÃ 4 çš®é¡ã® IO ãã¿ãŒã³ïŒSequential / Random à Read / WriteïŒ= åèš 28 ãã¿ãŒã³ã iodepth=1 ã§å®è¡ããŸãããNAS 㯠nconnect=8 ã®æ¡ä»¶ã§ãã [global] time_based runtime=120 startdelay=5 ramp_time=10 direct=1 size=450G numjobs=1 iodepth=1 ioengine=libaio random_generator=lfsr filename=/myvol/testfile direct=1 ã§ OS ã®ããŒãžãã£ãã·ã¥ã®åœ±é¿ãæå¶ã random_generator=lfsr ã§ ONTAP read cache ãžã®åããããæå¶ åãã¿ãŒã³ 120 ç§é à 3 iterãiter2-3 ã®å¹³åãæ¡çš 1 VM ãã€å®è¡ïŒONTAP åŽã®è² è·ãçžäºã«åœ±é¿ããªãããã«ããããïŒ iodepth=1 ã«ã€ããŠã®æ³šæ : iodepth=1 ã¯åæã«æªåŠçã® IO ã 1 ã€ã ããšããæ¡ä»¶ã§ããper-IO ã®ã¬ã€ãã³ã·ã®éããæ¯èŒããã«ã¯é©ããŠããŸãããã¹ãã¬ãŒãžã®ããã³ã·ã£ã«ããã¹ãŠåŒãåºãæ¡ä»¶ã§ã¯ãããŸããããã®æ¡ä»¶ã§ã®çµæããã£ãŠã¹ãã¬ãŒãžã®è£œåæ§èœãéçæ§èœãèªãããšã¯ã§ããŸãããã¹ãã¬ãŒãžã®éçæ§èœã¯æ€èšŒ 2ïŒiodepth ãæ®µéçã«äžããæ€èšŒïŒã§æž¬å®ããŸãã çµæ IOPS æ¯èŒ : Random Read / Random WriteïŒiodepth=1ïŒ ã¹ã«ãŒãããæ¯èŒ : Sequential Read / Sequential WriteïŒiodepth=1ïŒ iodepth=1 ã®æ¡ä»¶ã§ã¯ãRead ã¯å
šãã¿ãŒã³ã§ SAN ã NAS ãäžåããŸãããç¹ã«ãããã¯ãµã€ãºãå°ããã»ã©å·®ã倧ãããRandom Read ã§ã¯ 4k ã§ SAN 2.5 åã256k ã§ 1.4 åã§ãïŒSequential Read: SAN 1.3ã1.9 åïŒãäžæ¹ Write 㯠SAN åªäœãªãããå·®ã¯å°ãããRandom Write ã§ SAN 1.0ã1.3 åãSequential Write ã§ SAN 0.95ã1.4 åãšã°ãã€ããããããããã¯ãµã€ãºã«ãã£ãŠã¯ NAS ãäžåãã±ãŒã¹ããããŸããã nconnect=8 ãèšå®ããŠã iodepth=1 ã§ã¯åæã«åŠçããã IO ãéãããããã nconnect ã®å¹æã¯ã»ãŒåºãŠããŸããã iodepth=1 ã§ã¯ IO ãã¹ã®çã SAN ãæå©ã§ãper-IO ãªãŒããŒãããã®åœ±é¿ã倧ããå°ãããã¯ã»ã©å·®ãé¡èã§ããã§ã¯ iodepth ãäžããŠãããšã©ããªãããæ¬¡ã®æ€èšŒ 2 ã§èŠãŠã¿ãŸãããã æ§èœæ€èšŒ 2 : 䞊å床ïŒiodepthïŒ æ€èšŒ 1 㯠iodepth=1 ïŒåæ IO ã 1 ã€ã ãïŒã®æ¡ä»¶ã§ããããããçŸå®ã®ç°å¢ã§ã¯è€æ°ã® VM ãã¢ããªãåæã« IO ãçºè¡ããã®ãæ®éã§ããã¹ãã¬ãŒãžã®ããã³ã·ã£ã«ãåŒãåºãããã«ãiodepth ã 1 â 128 ãŸã§æ®µéçã«äžããŠéçæ§èœãæ¢ããŸãã iodepth ãäžããŠãããš IOPS ã¯äŒžã³ãŸããããããŠåŠçèœåã®éçã«éã㊠Latency ãæ¥äžæããŸããæšªè»ž IOPSã»çžŠè»ž Latency ã§ãããããããšããããããã±ãŒã¹ãã£ãã¯åã®ã«ãŒãã«ãªããå®çšçãªåäœç¹ãšéçæ§èœãèŠããŠããŸãã ãã¹ãæ¡ä»¶ 4k Random Read / Random Write ã«çµããiodepth ã 1 â 128 ãŸã§æ®µéçã«äžããŠæž¬å®ããŸãããSANïŒiSCSIïŒãNAS nconnect=8 ãNAS nconnect=1 ïŒæ¯èŒçšããŒã¹ã©ã€ã³ïŒã® 3 æ§æãæž¬å®ããŠããŸãã [global] time_based runtime=120 startdelay=5 ramp_time=10 direct=1 bs=4k size=450G numjobs=1 ioengine=libaio random_generator=lfsr filename=/myvol/testfile iodepth: 1 / 2 / 4 / 8 / 16 / 32 / 64 / 128 å iodepth Ã åæ§æã§ 3 iter 宿œããiter2-3 ã®å¹³åãæ¡çšããŠããŸãïŒiter1 㯠ONTAP ãã£ãã·ã¥ãŠã©ãŒã ã¢ããã®åœ±é¿ãããããé€å€ïŒã ååºŠã®æ³šæ : 以äžã®çµæã¯ 1 VM ããã® fio å®è¡çµæã§ããã¯ã©ã¹ã¿å
šäœã®éçŽæ§èœããã«ãã¯ã©ã€ã¢ã³ãæ§èœã瀺ããã®ã§ã¯ãããŸãããè€æ° VM ã®çµæã¯æ€èšŒ 3 ã§èŠãŸãã çµæãšèå¯ Random Read iodepth SAN (iSCSI) NAS nc8 NAS nc1 SAN vs nc8 nc8 / nc1 1 4,394 1,804 1,796 SAN 2.4x 1.00x 2 10,275 4,072 4,152 SAN 2.5x 0.98x 4 21,302 8,190 8,081 SAN 2.6x 1.01x 8 38,704 15,606 15,152 SAN 2.5x 1.03x 16 65,668 25,724 24,247 SAN 2.6x 1.06x 32 106,533 58,893 36,535 SAN 1.8x 1.61x 64 136,682 108,739 50,807 SAN 1.3x 2.14x 128 135,911 121,113 48,758 SAN 1.1x 2.48x Random Read ã¯ããã®æ§æã§ã¯å
šãŠã® iodepth ã§ SAN ã NAS nconnect=8 ãäžåããŸããããã ãããã®å·®ã¯ iodepth ãäžããã«ã€ããŠçž®ãŸã£ãŠããŸããäœ iodepthïŒ1ã16ïŒã§ã¯ SAN ã 2.4ã2.6 åãšå€§ããå·®ãã€ããŠããŸããã iodepth=64 ã§ã¯ 1.3 åã iodepth=128 ã§ã¯ 1.1 åãšã»ãŒæ®æããŸãã æ³šç®ãã¹ã㯠nconnect=8 ã®å¹æã®åºæ¹ã§ããiodepth 1ã16 ã§ã¯ nconnect=8 ãš nconnect=1 ã®å·®ãã»ãŒãŒãïŒnc8/nc1 â 1.0xïŒã nconnect ã®å¹æãçŸããã®ã¯ iodepth=32 以éã§ã iodepth=128 ã§ã¯ nc8 ã nc1 ã® 2.48 åã«éããŠããŸããåæã«çºè¡ãã IO ãå°ãªããã¡ã¯è€æ° TCP ã³ãã¯ã·ã§ã³ãå©çšãã䞊ååã®å¹æãåºã«ãããšèããããŸãã Random Write iodepth SAN (iSCSI) NAS nc8 NAS nc1 SAN vs nc8 nc8 / nc1 1 2,240 1,970 1,986 SAN 1.1x 0.99x 2 4,628 3,874 3,545 SAN 1.2x 1.09x 4 9,513 7,569 6,765 SAN 1.3x 1.12x 8 16,387 13,556 12,301 SAN 1.2x 1.10x 16 26,607 23,655 20,756 SAN 1.1x 1.14x 32 36,975 34,686 30,471 SAN 1.1x 1.14x 64 42,132 41,759 36,840 SAN 1.0x 1.13x 128 42,771 42,824 37,253 åç 1.15x Random Write ãäœãäž iodepth ã§ã¯ SAN ãåªäœã§ãããRead ã»ã©ã®å€§å·®ã¯ãããŸãããSAN ã®åªäœã¯ iodepth=4 ã® 1.3 åãæå€§ã§ãiodepth ãäžããã«ã€ããŠå·®ãçž®ãŸãã iodepth=128 ã§ã¯ NAS nconnect=8 ã SAN ãšã»ãŒåçïŒ42,824 vs 42,771 IOPSïŒã«è¿œãã€ããŸãã nconnect=8 ã®å¹æã¯ Read ãšåæ§ã« iodepth äŸåã§ããäœ iodepth ã§ã¯ nc1 ãšã»ãŒåçã§ããã iodepth=16 以éã§ nc1 ã® 1.14ã1.15 åãšãæ§ãããªããäžè²«ãããã©ã¹å¹æãåºãŠããŸãã IOPS by iodepth 4k Random IO : IOPS by iodepthïŒSAN / NAS nconnect=8 / NAS nconnect=1ïŒ ã°ã©ãã«ãããšåŸåãããæç¢ºã«ãªããŸããRandom Read ã§ã¯ SAN ã®ç·ãçµå§ããããèµ°ããŸãããNAS nconnect=8 ã®ç·ã iodepth=32 ãããããæ¥åŸé
ã§ã¹ã±ãŒã«ããé« iodepth ã§ SAN ã«è¿«ã£ãŠããŸããNAS nconnect=1 㯠iodepth ãäžãã£ãŠã䌞ã³ãéãã nconnect ã®æç¡ã§å€§ããªå·®ãã€ããŸããRandom Write ã§ã¯ 3 æ§æãšã䌌ãè»è·¡ãæããŸããã iodepth=128 ã§ NAS nconnect=8 ã SAN ã«è¿œãã€ãã®ãèŠãŠåããŸãã ããã±ãŒã¹ãã£ãã¯ã«ãŒã IOPS vs Latency 暪軞ã IOPSã瞊軞ã Latency ãšããŠããããããããã±ãŒã¹ãã£ãã¯ã«ãŒãã§ãã Random Read ã§ã¯ãSAN ã®ã«ãŒããæãå³ã«äŒžã³ãŠããã137K IOPS ä»è¿ã§ã«ãŒããç«ã¡äžãã£ãŠããŸããNAS nconnect=8 㯠121K IOPS ãŸã§äŒžã³ãŠãã SAN ã®çŽ 89% ã«éããŠããŸãããLatency ã®äžæã SAN ããæ©ãå§ãŸã£ãŠããŸãã nconnect=1 㯠51K IOPS ä»è¿ã倩äºã§ãã Random Write ã§ã¯ãSAN ã 43K IOPS ã«å°éããŠã Latency ã®æ¥äžæã«è³ã£ãŠãããããŸã äœåãããããã§ããNAS nconnect=8 ãåçšåºŠã® 43K IOPS ã«éããŠãããWrite ã«é¢ããŠã¯ SAN ãš NAS ã®å€©äºãã»ãŒåããšããçµæã§ãã nconnect=8 ã®å¹æãš SAN/NAS ã®æ§èœå·®ã®èŠå ã«é¢ããèå¯ æ€èšŒ 2 ã®çµæãå·Šå³ããŠããã®ã¯ã次㮠2 ã€ã®èŠå ã ãšèããããŸãã IO ãã¹ã®é·ã : SAN ãæå©ã«ããèŠå TCP ã³ãã¯ã·ã§ã³ã®äžŠå床 : NAS ã远ãäžããèŠå iodepth ãäœããã¡ã¯èŠå 1 ãæ¯é
çã§ SAN ã倧差ãã€ããŸãããiodepth ãäžãããšèŠå 2 ãå¹ãå§ããå·®ãçž®ãŸã£ãŠãããŸããããããèŠãŠãããŸãããã èŠå 1 : IO ãã¹ã®é·ãïŒSAN æå©ïŒ ontap-san ïŒiSCSIïŒã® IO ãã¹ : VM â virtio-blk â iSCSI initiator â iSCSI target â LUN â WAFL ontap-nas ïŒNFSïŒã® IO ãã¹ : VM â virtio-blk â disk imageïŒããŒãäžã®ãã¡ã€ã«ã·ã¹ãã ïŒâ NFS client â NFS server â FlexVol å
ãã¡ã€ã« â WAFL ontap-nas ã®ãã¹ã«ã¯ãããŒãåŽã®ãã¡ã€ã«ã·ã¹ãã ããæãŸã£ãŠããŸããOpenShift Virtualization ã§ã¯ã ontap-nas ïŒ volumeMode: Filesystem ïŒã®å ŽåãVM ã®ãã£ã¹ã¯ã€ã¡ãŒãžã¯ NFS ããŠã³ãããããã¡ã€ã«ã·ã¹ãã äžã®ãã¡ã€ã«ãšããŠæ ŒçŽãããŸããVM ã IO ãçºè¡ãããã³ã«ãããŒãåŽã®ã«ãŒãã«ã§ã¡ã¿ããŒã¿åŠçïŒinode â extent map â NFS ãã¡ã€ã«ãã³ãã« + ãªãã»ããã®è§£æ±ºïŒãä»åšãããšèããããŸãã äžæ¹ ontap-san ïŒ volumeMode: Block ïŒã§ã¯ãLUN ãããŒããããããã¯ããã€ã¹ãšããŠçŽæ¥èŠãããããããŒãåŽã«ãã¡ã€ã«ã·ã¹ãã ã¬ã€ã€ãŒãååšããŸããããã® IO ãã¹ã®çãããSAN ã® per-IO ã¬ã€ãã³ã·ã®äœãã«ã€ãªãã£ãŠããèŠå ã®äžã€ãšèããããŸãã iodepth ãäœãïŒåæã«é£ã¶ IO ãå°ãªãïŒç¶æ
ã§ã¯ãIOPS ã¯ã»ãŒ per-IO ã¬ã€ãã³ã·ã§æ±ºãŸããŸããIO ãã¹ãçã SAN ã¯ããã§å€§ããæå©ã«ãªããRead ã§ 2.4ã2.6 åã®å·®ãã€ããŠããŸãã ãªããä»åã®æž¬å®ã ãã§ã¯ IO ãã¹å·®ãšä»ã®èŠå ïŒãããã³ã«ãªãŒããŒããããNFS RPC ã®åŠçã³ã¹ããªã©ïŒã®åå¥ã®å¯äžçãåãåããããšã¯ã§ããŸãããäžèšã¯ãããŸã§åŸåã説æããèå¯ã§ãã èŠå 2 : TCP ã³ãã¯ã·ã§ã³ã®äžŠå床ïŒNAS 远ãäžãïŒ nconnect=8 㯠NFS ããŠã³ãã« 8 æ¬ã® TCP ã³ãã¯ã·ã§ã³ã匵ããIO ãã©ãŠã³ãããã³ã§åæ£ããŸãããã ãããã®å¹æã¯ iodepth ã«åŒ·ãäŸåããŸãã iodepth 1ã16: Read 㯠nc8 â nc1ïŒnc8/nc1 â 1.0xïŒãWrite 㯠nc8 ãè¥å¹²æå©ïŒ1.0ã1.15xïŒ iodepth 32ã128: nc8 > nc1ïŒRead ã§æå€§ 2.5xãWrite ã§ 1.1ã1.2xïŒ iodepth=1 ã§ã¯åæã«æªåŠçã® IO 㯠1 ã€ã ããªã®ã§ãTCP ã³ãã¯ã·ã§ã³ã 8 æ¬ãã£ãŠãå®éã«äœ¿ãããã®ã¯ 1 æ¬ã§ããiodepth ãäžãã£ãŠåæã«è€æ°ã® IO ãé£ã¶ããã«ãªããšãããããè€æ°ã³ãã¯ã·ã§ã³ãžã®åæ£ãå¹ãå§ããŸãã iodepth ãååã«é«ããªããšãIOPS ã®åŸéèŠå ã per-IO ã¬ã€ãã³ã·ãã IO ã®åæåŠçèœåã«ç§»ããšèããããŸãããããªããšãIO ãã¹ã®é·ãã«ãããªãŒããŒãããã¯æ¶ããªããã®ã®ã8 æ¬ã® TCP ã³ãã¯ã·ã§ã³ã«ããäžŠåæ§ã§è£ããããã«ãªããNAS ã SAN ãšã®å·®ãè©°ããããããã«ãªããŸããRead ã§ã¯ iodepth=128 ã§ SAN ã® 89%ïŒ121K vs 136K IOPSïŒãŸã§è¿œãäžããWrite ã§ã¯åçïŒ43K IOPSïŒã«è¿œãã€ããŠããŸãã Read ã§ã®å¹æã Write ãã倧ããã®ã¯ãNFS ã® READ call ã䞊ååŠçã®å¶çŽãå°ãªãæäœã§ããã®ã«å¯Ÿãã WRITE call ã¯ãµãŒããŒåŽã§æŽåæ§åŠçïŒNVRAM ãžã®æžã蟌ã¿ã確èªå¿çã®åæãªã©ïŒãä»åšãããããã³ãã¯ã·ã§ã³ãå¢ãããã¶ãã ãã¹ã±ãŒã«ãã«ããããã§ã¯ãªãããšèããããŸãã iSCSI ã§ TCP ãå¢ããã«ã¯ iSCSI 㯠1 ã»ãã·ã§ã³ããã 1 æ¬ã® TCP ã³ãã¯ã·ã§ã³ã䜿ããŸããiSCSI ã®èŠæ ŒïŒRFC 7143ïŒã«ã¯ MC/SïŒMultiple Connections per SessionïŒãšããè€æ°ã³ãã¯ã·ã§ã³æ©èœããããŸãããLinux ã® iSCSI ã€ãã·ãšãŒã¿ïŒ open-iscsi ïŒã¯ããããµããŒãããŠããŸãããiSCSI ã§ TCP ã³ãã¯ã·ã§ã³ãå¢ããã«ã¯ iSCSI multipathïŒè€æ° NIC æ§æïŒãå¿
èŠã§ããNFS ã® nconnect ã®ããã« Trident ã® TBC ã« nfsMountOptions: nconnect=8 ãš 1 è¡è¿œå ããã ãã§ TCP ãå€éåã§ããæè»œãã¯ãNFS åŽã®ã¢ããã³ããŒãžãšèšããŸãã æ§èœæ€èšŒ 3 : è€æ° VM 䞊å ãããŸã§ã®ãã¹ã㯠1 VMïŒ1 fio ããã»ã¹ïŒã§ã®æž¬å®ã§ããããã VMware ç°å¢ã§ã¯ 1 ã€ã®ããŒã¿ã¹ãã¢ã«è€æ°ã® VM ãèŒãã®ãåœããåã§ãããåæ§ã«ãONTAP äžã§ãè€æ° VM ãåäžã® FlexVolïŒ ontap-san ã®å ŽåïŒãåäžã® NFS ããŠã³ãïŒ ontap-nas ã®å ŽåïŒã«åæã«ã¢ã¯ã»ã¹ããã®ãçŸå®ã§ããVM ãå¢ããããšãã« ONTAP ãã©ããŸã§ã¹ã±ãŒã«ããã®ããSAN ãš NAS ã§å·®ãåºãã®ããæ€èšŒããŸãã ãã¹ãæ¡ä»¶ 3 Compute Node ã« VM ãåçé
眮ããå
š VM ã§åæã« fio ãå®è¡ããŸããã 3 VM: åããŒãã« 1 VM ãã€ïŒ1/1/1 é
çœ®ïŒ 6 VM: åããŒãã« 2 VM ãã€ïŒ2/2/2 é
çœ®ïŒ fio ã®ãã©ã¡ãŒã¿ã¯æ€èšŒ 2 ãšåãã§ãããiodepth 㯠1 / 4 / 16 / 64 ã® 4 段éã«çµã£ãŠããŸããå VM ãããããç¬èªã® datadiskïŒ500GiïŒã«å¯Ÿã㊠fio ãå®è¡ããçµæã¯å
š VM ã®åç® IOPS ã§ãã VMware ã«äŸãããšãã1 ã€ã® ESXi ãã¹ãã« 1ã2 å°ã® VM ãèŒãã3 ãã¹ãã¯ã©ã¹ã¿å
šäœã§ã¹ãã¬ãŒãžã®ã¹ã±ãŒã©ããªãã£ãèŠãããšããæ§æã«è¿ããã¹ãã§ãã 3 VM ã®çµæ ãã¿ãŒã³ iodepth SAN (iSCSI) NAS nc8 åªäœ randread 1 7,460 5,584 SAN 1.3x randread 4 30,888 23,079 SAN 1.3x randread 16 83,059 71,570 SAN 1.2x randread 64 102,983 108,122 NAS 1.05x randwrite 1 6,649 5,395 SAN 1.2x randwrite 4 24,010 20,065 SAN 1.2x randwrite 16 43,230 50,608 NAS 1.2x randwrite 64 50,355 70,838 NAS 1.4x 3 VM ã§ã¯è峿·±ãçµæã«ãªããŸãããäœã iodepthïŒ1, 4ïŒã§ã¯ SAN ã 1.2ã1.3 åã§åªäœã§ããã iodepth ãé«ããªããš NAS ãé転 ããŠããŸãã æ€èšŒ 2 ã® 1 VM ã§ããé« iodepth ã§ã¯ NAS ã SAN ã«è¿«ãåŸåããããŸããïŒWrite iodepth=128 ã§åçïŒã3 VM ã§ã¯ãã®åŸåãããã«é²ãã§ããŸãã3 ããŒããããããã nconnect=8 ã® TCP ã³ãã¯ã·ã§ã³ã§ IO ãçºè¡ãããããONTAP åŽã§äžŠååŠçã§ãã IO ã®ç·éãå¢ãã nconnect ã®å¹æããã匷ãçŸããŠãããšèããããŸãã 6 VM ã®çµæ ãã¿ãŒã³ iodepth SAN (iSCSI) NAS nc8 åªäœ randread 1 14,869 10,707 SAN 1.4x randread 4 61,645 41,563 SAN 1.5x randread 16 162,501 92,993 SAN 1.7x randread 64 207,877 115,976 SAN 1.8x randwrite 1 13,986 10,238 SAN 1.4x randwrite 4 46,829 33,754 SAN 1.4x randwrite 16 85,962 65,814 SAN 1.3x randwrite 64 99,924 73,809 SAN 1.4x 6 VM ã«ãªããšæ§çžãäžå€ããŸãã å
šãã¿ãŒã³ã»å
š iodepth ã§ SAN ãåªäœ ã§ããã®å·®ã¯ 1.3ã1.8 åã3 VM ã§ NAS ãé転ããŠããé« iodepth é åã§ãã6 VM ã§ã¯ SAN ãæç¢ºã«äžåã£ãŠããŸãã SAN 㯠iodepth=64 ã§ randread 208K IOPSãrandwrite 100K IOPS ã«å°éãäžæ¹ NAS 㯠randread 116Kãrandwrite 74K ã«ãšã©ãŸããŸããã 3 VM â 6 VM ã¹ã±ãŒãªã³ã° è€æ° VM ã§ã®äžŠå IOïŒ3 VM / 6 VMïŒ 3 VM â 6 VM ã¹ã±ãŒã« ãã¿ãŒã³ iodepth SAN ã¹ã±ãŒã« NAS ã¹ã±ãŒã« randread 1 1.99x 1.92x randread 4 2.00x 1.80x randread 16 1.96x 1.30x randread 64 2.02x 1.07x randwrite 1 2.10x 1.90x randwrite 4 1.95x 1.68x randwrite 16 1.99x 1.30x randwrite 64 1.98x 1.04x ãããä»åã®æ€èšŒã§æã泚ç®ãã¹ãçµæã®ã²ãšã€ã§ãã SAN 㯠VM æ°ã«å¯ŸããŠã»ãŒçæ³çãªç·åœ¢ã¹ã±ãŒãªã³ã° ã瀺ããŠããŸãã3 VM â 6 VM ã§å
šãã¿ãŒã³çŽ 2.0 åãONTAP ã® HA ãã¢ïŒ2 ããŒãïŒã«å¯Ÿã㊠VM ãåçã«åæ£ããããããäž¡ããŒãã®åŠçèœåãå¹ççã«äœ¿ãåã£ãŠããŸãã NAS ã¯äœ iodepth ã§ã¯ç·åœ¢ã«è¿ãïŒ1.8ã1.9 åïŒãã®ã®ãé« iodepth ã§ã¯ã¹ã±ãŒãªã³ã°ãé æã¡ã«ãªã£ãŠããŸãã iodepth=64 ã§ã¯ããã 1.04ã1.07 åãã䌞ã³ãŠããŸãããVM ãåã«ããŠãã»ãšãã©æ§èœãäžãããªããšããããšã§ãã 6 VM ã 3 ããŒãã«åæ£ããåããŒããã nconnect=8 ã§è€æ°ã® TCP ã³ãã¯ã·ã§ã³ã ONTAP ã® NFS ãµãŒããŒã«éäžããããšã§ãNFS ãããã³ã«ã®åŠçïŒRPC ã®åŠçãã³ãã¯ã·ã§ã³ç®¡ççïŒãããã«ããã¯ãšãªããã¹ãã¬ãŒãžåŽã飜åããŠãããšèããããŸããSANïŒiSCSIïŒã¯å LUN ãç¬ç«ããŠåŠçãããããããããã³ã«å±€ã§ã®ãã®çš®ã®ããã«ããã¯ãçãã«ãããšèããããŸãã VMware ã®äžçã«çœ®ãæãããšããNFS ããŒã¿ã¹ãã¢ã«å€æ°ã® VM ãéçŽãããšãVMFSïŒiSCSI / FCïŒããŒã¿ã¹ãã¢ã«æ¯ã¹ãŠã¹ã±ãŒã©ããªãã£ã®å£ã«åœãããããããšããçµéšåã«è¿ãåŸåã§ãããã ããã¹ãã¬ãŒãžã¢ãã«ããããã¯ãŒã¯åž¯åã«ãã£ãŠãã®å£ã®äœçœ®ã¯å€§ããå€ãããŸãã ãŸãšããšéžå®ã¬ã€ã ä»å㯠3 çš®é¡ã® fio ãã³ãããŒã¯ã«ãããAFF C190 äžã® ontap-san ïŒiSCSIïŒãš ontap-nas ïŒNFSv4.2, nconnect=8 ïŒã® IO æ§èœãæ¯èŒããŸããã æ€èšŒçµæãµã㪠æ€èšŒ æ¡ä»¶ Read Write æ€èšŒ 1 1 VM, iodepth=1 , BS å¯å€ SAN 1.2ã2.5xïŒå° BS ã»ã©å·®å€§ïŒ SAN 0.95ã1.4xïŒNAS åªäœãããïŒ æ€èšŒ 2 1 VM, iodepth å¯å€, 4k SAN 1.1ã2.6xïŒé« iodepth ã§å·®çž®å°ïŒ SAN 1.0ã1.3xïŒiodepth=128 ã§åçïŒ æ€èšŒ 3ïŒ3 VMïŒ 3 VM, iodepth å¯å€, 4k äœ iodepth: SANãé« iodepth: ã»ãŒåç äœ iodepth: SANãé« iodepth: NAS 1.4x æ€èšŒ 3ïŒ6 VMïŒ 6 VM, iodepth å¯å€, 4k SAN 1.4ã1.8x SAN 1.3ã1.4x çµæããèŠããããš 1 VM ã§ã¯ Read 㯠SAN ãå®å®ããŠéãã Write ãå€ãã®æ¡ä»¶ã§ SAN ãåªäœã§ãããiodepth ãäžããã«ã€ããŠå·®ã¯çž®ãŸããWrite ã® iodepth=128 ã§ã¯ NAS nconnect=8 ã SAN ãšåçãŸã§è¿œãã€ããŸãã nconnect=8 ã¯é« iodepth ã§å¹æãçºæ®ããã nconnect=8 ã®å¹æã¯ iodepth 32 以äžã§çŸããŸããRead ã§ã¯ nconnect=1 ã®æå€§ 2.5 åãWrite ã§ã 1.1ã1.2 åã®åäžãåŸãããŸããéã« iodepth ãäœãç¶æ
ã§ã¯ nconnect ãå¢ãããŠã广ã¯ã»ãŒãããŸãããNFS ã䜿ããªã nconnect ã®èšå®ãæ§èœã«å€§ãã圱é¿ããŸããããã®å¹æã掻ããã®ã¯ IO ã®äžŠå床ãé«ãç°å¢ã§ãã NFS 㯠NFSv4 以äžã䜿ãã ä»åã®æ€èšŒã§ã¯ NFSv4.2 ã䜿çšããŠããŸããNFSv4 ã® Compound Operations ã«ããã©ãŠã³ãããªããåæžãã»ãã·ã§ã³ãã©ã³ãã³ã°ã¯ nconnect ãšçžæ§ãè¯ããNFSv3 ãããå¹ççã«äžŠå IO ãåŠçã§ããŸãã VM æ°ãã¹ã±ãŒã«ããç°å¢ã§ã¯ SAN ã«ã¹ã±ãŒã©ããªãã£ã®å©ãããã SAN 㯠VM æ°ã«å¯ŸããŠç·åœ¢ã«ã¹ã±ãŒã«ããŸãããNAS ã¯é« iodepth ã§é æã¡ã«ãªãåŸåãèŠãããŸããïŒãã® AFF C190 æ§æã®å ŽåïŒã倿°ã® VM ãåæã«é« IO ãçºè¡ããç°å¢ã§ã¯ãSAN ã®ã¹ã±ãŒã©ããªãã£ãå
ããŸãã å°æ° VM ãªã NAS ãååç«¶äºåãããã 3 VM çšåºŠã§ããã° NAS nconnect=8 ã¯é« iodepth ã§ SAN ãšåç以äžã®æ§èœãåºããŠãããçšéã«ãã£ãŠã¯ ontap-nas ãæå©ãªå Žé¢ããããŸãã ãããã³ã«éžå®ã¬ã€ã : æ§èœã ãã§ã¯æ±ºãŸããªã VMware ãã OpenShift Virtualization ãžç§»è¡ããéãã¹ãã¬ãŒãžãããã³ã«ã®éžå®ã¯æ§èœã ãã§æ±ºãŸããã®ã§ã¯ãããŸãããVMware ã®ãšããšåæ§ã«ãç·åçãªå€æãå¿
èŠã§ãã以äžã®èгç¹ã§æŽçããŸãã èŠ³ç¹ ontap-nas ïŒNFSïŒ ontap-san ïŒiSCSIïŒ IOPS æ§èœïŒ1 VMïŒ SAN ã«å£ãã nconnect=8 + é« iodepth ã§å·®ã¯çž®ãŸã å
šè¬çã«å®å®ããŠé«ã ã¹ã±ãŒã©ããªãã£ïŒå€ VMïŒ é« iodepth ã§é æã¡åŸåãã VM æ°ã«å¯ŸããŠç·åœ¢ã«ã¹ã±ãŒã« Live migration RWX ãæšæºãµããŒããVMware ã® vMotion ãšåæ§ã«ã¹ãã¬ãŒãžåŽã®è¿œå èšå®äžèŠ RWXïŒ volumeMode: Block ïŒã§å¯Ÿå¿å¯èœ éçšã®ã·ã³ãã«ã NFS ããŠã³ãã®ã¿ãVMware ã® NFS ããŒã¿ã¹ãã¢ãšåæ§ã«ç®¡çãã·ã³ãã« iSCSI initiatorãLUN ãããã³ã°ãå¿
èŠ nconnect ã«ãããã¥ãŒãã³ã° TBC ã« 1 è¡è¿œå ã§ TCP å€éåãKubernetes / Trident ãªãã§ã¯ã®æè»œã TCP å€éåã«ã¯ multipathïŒè€æ° NICïŒãå¿
èŠ ããã¯ã¢ããã»DR ONTAP SnapMirrorãTrident Protect ã§ NFS / iSCSI ãšãã«å¯Ÿå¿ããããã³ã«å·®ã¯å°ãã åå·Š æ¢åã®éçšãã¬ããž VMware ã§ NFS ããŒã¿ã¹ãã¢ãéçšããŠããçµç¹ã«ã¯èŠªåæ§ãé«ã VMware ã§ VMFS + iSCSI / FC ãéçšããŠããçµç¹ã«ã¯èŠªåæ§ãé«ã æ§èœèŠä»¶ãèŠããŠããå Žå : äŸãã°ããŒã¯ IOPS ã 20,000 IOPS çšåºŠã§ããã°ã ontap-nas ã ontap-san ãã©ã¡ãããµãããªç§ã§è¿ãã®ã§ãã¶ã£ã¡ãã倧差ãããŸããããã®å Žåã¯è«çæ§æãã·ã³ãã«ãª ontap-nas ãéžã¶å€æã¯å
šããã£ãŠåŠ¥åœã§ãã倿° VM ã®é« IOPS ãå¿
èŠãªç°å¢ã§ã¯ ontap-san ã®ã¹ã±ãŒã©ããªãã£ã掻ããŸãã æ§èœèŠä»¶ãèŠããŠããªãå Žå : æè¿ã¯ Storage live migration ã GA ã«ãªã£ãŠ ontap-san â ontap-nas ã®ç§»åã¯ç°¡åã«ã§ããã®ã§ããŸãã¯æ°æ¥œã« ontap-nas + nconnect=8 ã§å§ããŠãæ§èœèŠä»¶ã«å¿ã㊠ontap-san ã«åãæ¿ãããšããã¢ãããŒããååçŸå®çã§ãã VMware ã§ NFS ããŒã¿ã¹ãã¢ã䜿ã£ãŠããå Žå : NFS ã®éçšãã¬ããžãããçµç¹ãªãã ontap-nas ã¯ç§»è¡ã®ããŒãã«ãäœãã§ãã nconnect=8 ãå¿ããã«èšå®ããã°ãæ§èœé¢ã§ãååæŠããŸãã VMware ã§ VMFS + iSCSI / FC ã䜿ã£ãŠããå Žå : SAN ã®éçšãã¬ããžãããçµç¹ãªãã ontap-san ãžã®ç§»è¡ã¯èªç¶ãªéžæã§ãããã ããOpenShift / Kubernetes ç°å¢ã§ã¯ Trident ã LUN ã®ç®¡çãèªååããŠããããããVMware æä»£ã® LUN 管çã®æéã¯å€§å¹
ã«è»œæžãããŸãã ãªããQoS ã«ãã VM ããšã®æ§èœå¶åŸ¡ã«ã€ããŠã¯ç¬¬5åïŒãã«ãããã³ãèšèšïŒã§ãæ¥åžžçãªæ§èœç£èŠã®æ¹æ³ã«ã€ããŠã¯ç¬¬8åïŒç£èŠãšéçšïŒã§è©³ããæ±ããŸãã ãšããããã§ä»æ¥ã¯ãããŸã§ã次åã®ç¬¬4åã§ã¯ãNetApp Shift Toolkit ãš MTV ã䜿ã£ã VM ã®ç§»è¡ã«ã€ããŠç޹ä»ããŸãã ããŸã : ãã³ãããŒã¯äžã«èµ·ãããã©ãã« ä»åã®ãã³ãããŒã¯ã§ã snapshotPolicy: default ã®ãŸãŸ fio ãå®è¡ããçµæãONTAP ã® Snapshot å·®åã snapshotReserve ïŒ10%ïŒãè¶
ã㊠FlexVol ãåãå°œãããVM ã PausedIOError ã§åæ¢ãããšãããã©ãã«ãçºçããŸããã 容éç·š ã§ã snapshotReserve: 10 ãèšå®ããŠããã° Snapshot é åã確ä¿ã§ããããšæžããŸãããã倿Žçã®é«ãã¯ãŒã¯ããŒãã§ã¯ 10% ã§ãäžè¶³ãåŸããšããå®äŸã§ãã fio ã®ãããªå€§éæžã蟌ã¿ãè¡ãã¯ãŒã¯ããŒãã§ã¯ã1 æéããšã®å®æ Snapshot ãååŸããããšã倿Žãããã¯ã Snapshot ã«ä¿æããç¶ããŸãã450G ã®ãã¡ã€ã«ãç¹°ãè¿ãäžæžããããšãå·®åãèšåŒµããä»å㯠snapshot-space-used ã 195% ã«éããŸããã Write ã®ãã³ã宿œæã«ã¯ã察象ããªã¥ãŒã ã® snapshotPolicy ãäžæçã« none ã«å€æŽããŸãããããŸããæ¢åã® Snapshot ãåé€ããŠããããšã匷ãããããããŸããWAFL ã® Snapshot 㯠Redirect-on-Write ã§ãSnapshot äœææã¯ã»ãŒç¡é¢šã§ããå逿ã«ãããã¯ã®è§£æŸåŠçãããŒã£ãšçºçããŠæ§èœã«åœ±é¿ãåãŒãã®ã§ãå®å
šã®ãããã³ãã®åã«æ¶ããŠãããŸãããã æ¬çªç°å¢ã§ã¯ãã¯ãŒã¯ããŒãã®å€æŽçïŒæ¥æ¬¡ã®å€æŽãããã¯éïŒãš Snapshot ã®ä¿æäžä»£æ°ããå¿
èŠãª snapshotReserve ãèŠç©ãããSnapshot 䜿çšçãç£èŠããããšãéèŠã§ããã10% ã«ããŠããã°å®å¿ãã§ã¯ãªããå®éã®å€æŽãã¿ãŒã³ã«åãããèšèšãå¿
èŠã§ãã