Red Hatã§ã³ã³ãµã«ã¿ã³ããããŠããç¹ã§ãã ãã¡ãã®èšäº ã§ã¯ãupstreamã®KubernetesãšMultusã䜿ã£ãŠKVMç°å¢äžã§Multusãæ€èšŒããŸãããæ¬èšäºã§ã¯ãIPIæ¹åŒã§AWSäžã«æ§ç¯ããOCP (OpenShift Container Platform) v4.3ã䜿ã£ãŠMultusãåãããŠã¿ãŸãã ãããªãIPIãšæžããŸããããOpenShift v4ã§ã¯ãç°å¢æ§ç¯ã®æ¹æ³ã倧ãã2çš®é¡ãããIPIã¯ãã®ãã¡ã®ã²ãšã€ã§ãã IPI(Installer Provisioned Infrastructure)æ¹åŒã®ã€ã³ã¹ããŒã« - ãããã¯ãŒã¯ãããŒããã©ã³ãµãDNSã¬ã³ãŒããä»®æ³ãã·ã³ã€ã³ã¹ã¿ã³ã¹çãOpenShiftã®ã€ã³ã¹ããŒã«ã«å¿
èŠãªã€ã³ãã©ã®ã³ã³ããŒãã³ããã€ã³ã¹ããŒã©ãèªåçã«äœæããŸããworkerããŒãã®ãªãŒãã¹ã±ãŒã«ãã§ããããã«ãªã£ããããŸããã€ã³ãã©ãAPIã§å¶åŸ¡ã§ããç°å¢(äŸãã°IaaS)äžã«OpenShiftã¯ã©ã¹ã¿ãŒãæ§ç¯ãããšãã¯ãã¡ãã䟿å©ã§ãã UPI(User Provisioned Infrastructure)æ¹åŒã®ã€ã³ã¹ããŒã« - ãããã¯ãŒã¯ãããŒããã©ã³ãµãDNSã¬ã³ãŒããä»®æ³ãã·ã³ã€ã³ã¹ã¿ã³ã¹çãOpenShiftã®ã€ã³ã¹ããŒã«ã«å¿
èŠãªã€ã³ãã©ã®ã³ã³ããŒãã³ãã¯ãããããããŠãŒã¶ãŒãèªèº«ã§çšæããŠããå¿
èŠããããŸããã€ã³ã¹ããŒã©ã¯ãã®äžã«OpenShiftãã€ã³ã¹ããŒã«ããŸããã€ã³ãã©ãAPIã§å¶åŸ¡ã§ããªãç°å¢(äŸãã°ãªã³ãã¬ã®ãã¢ã¡ã¿ã«ç°å¢)ããIPIãäœãæ§æã ãšèŠä»¶ã«åããªãå ŽåãworkerããŒãã«(Red Hat CoreOSã§ã¯ãªã)RHELã䜿ãããå Žåçã¯UPIã䜿ããŸãã IPIã§AWSäžã«OpenShiftã¯ã©ã¹ã¿ãŒãæ§ç¯ãããšã äœæãããEC2ã€ã³ã¹ã¿ã³ã¹ã¯NIC(ENI)ã1åããæããªã ããŒãã¯ç°ãªãAZã«ãŸããã£ãŠåæ£é
眮 ãšããæ§æã«ãªããŸããAWSäžã®OpenShiftã§Multusã䜿ãå Žåã¯ãããããUPIã§ç°å¢ãäœãããšã«ãªãããšæãã®ã§ãããä»åã¯IPIã€ã³ã¹ããŒã«ã§æ§ç¯ããç°å¢ã«å¯ŸããŠãå°ãã«ã¹ã¿ãã€ãºããŠMultusã䜿ããããã«ããŸãããç¹ã«æ·±ãçç±ã¯ãããŸããã å®éã«è¡ã£ãã«ã¹ã¿ãã€ãºã¯ä»¥äžã§ãã åããŒãã«å¯ŸããŠNICã远å ãã远å NICã§éä¿¡ããããã®VPCãµãããããäœæãã 远å NICã䜿ã£ãŠç°ãªãAZã«ããworkerããŒãäžã®Podãšéä¿¡ã§ããããã«static routeãèšå®ãã OpenShift v4ã§ã¯ãã¯ã©ã¹ã¿ãŒã«å¯Ÿããèšå®ã¯å
šãŠOperatorçµç±ã§è¡ãäœãã«ãªã£ãŠããŸããäŸãã°ãããã¯ãŒã¯åšãã®èšå®ã¯Cluster Network Operator(CNO)ãåžããŸããMultusã®èšå®ãããéã åã®èšäº ã§ã¯NetworkAttachmentDefinitionã®Custom Resourceãå®çŸ©ããmanifestãæžã㊠kubectl create ããŸããããOpenShiftã§ã¯CNOçµç±ã§NetworkAttachmentDefinitionãäœããããCNOã«å¯ŸããŠæç€ºããŸãããã®èŸºããæ¬èšäºã§äžçªèšãããã£ãããšãªã®ã§ãããããããäœèšãªããšãæžããã®ã§é·ããªã£ãŠããŸããŸãããã容赊ãã ãã... OpenShiftã¯ã©ã¹ã¿ãŒã®æ§ç¯ AWSäžã«OpenShift v4.3.0[ 1 ]ã®ç°å¢ãæ§ç¯ããŸããä»åæ§ç¯ã«äœ¿ã£ãIPIãšããæ¹åŒã§ã¯ãVPCãELBãRoute53ã®ãŸãŒã³ç»é²çãå¿
èŠãªAWSãªãœãŒã¹ã¯å
šãŠèªåã§äœæãããã®äžã§RHCOS(Red Hat CoreOS)ã䜿ã£ãEC2ã€ã³ã¹ã¿ã³ã¹ãèµ·åããŠOpenShiftã¯ã©ã¹ã¿ãŒãæ§æããããšããäžé£ã®äœæ¥ãã³ãã³ãäžæã§ãã£ãŠãããŸãã $ openshift-install create cluster AWSäžã§IPIæ¹åŒã§OpenShiftãã€ã³ã¹ããŒã«ããæé ã®è©³çްã¯äžèšããåç
§ãã ããã å
¬åŒã®ã€ã³ã¹ããŒã«æé [ 2 ] ç¿ã§ãã§ããOpenShift4ã€ã³ã¹ããŒã«ç·š ãŸããæ¬æ€èšŒã§ã¯ãããŸããã§ããããããUPIã䜿ã£ãŠã€ã³ã¹ããŒã«ããå Žåã¯äžèšããåç
§ãã ããã UPIã§ã®ãã¢ã¡ã¿ã«ç°å¢ãžã®ã€ã³ã¹ããŒã«æé [ 3 ] OpenShiftãçéã§ã€ã³ã¹ããŒã«ãããããªããž Multusã®ããã«ç°å¢ãã«ã¹ã¿ãã€ãº IPIã€ã³ã¹ããŒã«ã§AWSã«OpenShiftã¯ã©ã¹ã¿ãŒãæ§ç¯ãããšãäžå³ã®ãããªæ§æã«ãªããŸã(EC2ã€ã³ã¹ã¿ã³ã¹ãšVPCã®ãã©ã€ããŒããããã¯ãŒã¯ã®ã¿èšèŒããŠããŸã)ã IPIã€ã³ã¹ããŒã«ã§AWSã«ã§ããOpenShiftã¯ã©ã¹ã¿ãŒ Multusã§ããã²ãšã€NICã䜿ãã®ã§ããããã«ã¹ã¿ãã€ãºããŠãæçµçã«äžå³ã®ããã«ãããã§ãã Multusãåããããæ§æ ãã®ããã«ã åAZã«ãµããããã远å ã 远å ãµããããããšã«æ¥ç¶ããENIãäœæã äœæããENIãåworkerããŒãã«ã¢ã¿ãããã ãšããã«ã¹ã¿ãã€ãºãããŸãã ãµããããã®è¿œå Multusã§è¿œå æ¥ç¶ããããã®ãµãããããåãªãŒãžã§ã³ã«äœæããŸããIPv4ã®ã¢ãã¬ã¹ãããã¯ã¯ãVPCã®ã¢ãã¬ã¹ãããã¯ããé©åœã«åãåºããŸããæ¬ç°å¢ã§ã¯ ãªãŒãžã§ã³ ãµããããå ã¢ãã¬ã¹ããã㯠ap-northeast-1a ocp43-multus-1a 10.0.201.0/24 ap-northeast-1c ocp43-multus-1c 10.0.202.0/24 ap-northeast-1d ocp43-multus-1d 10.0.203.0/24 ãäœæããŸãããAWS CLIã§ãããªã $ aws ec2 create-subnet --availability-zone ${az} --cidr-block ${cidr} --vpc-id ${vpc_id} $ aws ec2 create-tags --resources ${multus_subnet_id} --tags Key=Name,Value=${multus_subnet_name} ã¿ãããªæãã§ãã ENIã®äœæ åãªãŒãžã§ã³ã®workerããŒãã远å ãããµããããã«æ¥ç¶ããããã远å ãµããããããšã«ENIãäœæããŸãã ä»åã¯äžèšã®IPã¢ãã¬ã¹ã䜿ãããšã«ããŸãã ãµãããã ãã©ã€ããªã®IPv4ã¢ãã¬ã¹ ã»ã«ã³ããªã®IPv4ã¢ãã¬ã¹ ocp43-multus-1a 10.0.201.11 10.0.201.21 ã 10.0.201.29 ocp43-multus-1c 10.0.202.11 10.0.202.21 ã 10.0.202.29 ocp43-multus-1d 10.0.203.11 10.0.203.21 ã 10.0.203.29 ipvlanã§Podã«ä»äžããIPã¢ãã¬ã¹ãããããããENIã®ã»ã«ã³ããªãã©ã€ããŒãIPv4ã¢ãã¬ã¹ãšããŠåæããŠãããŸãã $ aws ec2 create-network-interface --subnet-id ${multus_subnet_id} \ --description "for multus, subnet: ${multu_subnet_name}, az:${az}" \ --private-ip-addresses Primary=true,PrivateIpAddress=${prefix}.11 \ Primary=false,PrivateIpAddress=${prefix}.21 \ Primary=false,PrivateIpAddress=${prefix}.22 \ Primary=false,PrivateIpAddress=${prefix}.23 \ Primary=false,PrivateIpAddress=${prefix}.24 \ Primary=false,PrivateIpAddress=${prefix}.25 \ Primary=false,PrivateIpAddress=${prefix}.26 \ Primary=false,PrivateIpAddress=${prefix}.27 \ Primary=false,PrivateIpAddress=${prefix}.28 \ Primary=false,PrivateIpAddress=${prefix}.29 $ aws ec2 create-tags --resources ${interface_id} --tags Key=Name,Value=eni-${multus_subnet_name} ENIãworkerããŒããžã¢ã¿ãã äœæããENIããåworkerããŒãã«ã¢ã¿ããããŸãã $ aws ec2 attach-network-interface --device-index 1 --instance-id ${instance_id} --network-interface-id ${interface_id} workerããŒãã®ç¢ºèª ENIãworkerããŒãã«ã¢ã¿ãããããšãworkerããŒãã®RHCOS(ã«ãŒãã«ããŠãŒã¶ãŒã©ã³ãã®ã³ãŒãã¯RHEL8ãšå
šãåã[ 4 ])ã®ã«ãŒãã«ãåçã«è¿œå ã€ã³ã¿ãŒãã§ãŒã¹ã ens4 ãšããŠèªèããŠãããŸããworkerããŒãã§ãã®ããšã確èªããŠã¿ãŸãã OpenShift v4.xã§master/workerããŒãã«å
¥ã£ãŠOSã³ãã³ããå®è¡ããæ¹æ³ã¯2ã€ãããŸãã sshãã°ã€ã³ãã - ã€ã³ã¹ããŒã©( openshift-installer ã³ãã³ã)å®è¡ããŒãã®ssh-agentã«éµãç»é²ãããŠããã°ããããRHCOSã®ãŠãŒã¶ãŒ core ã®authorized_keysã«ç»é²ããŠãããŸãã oc debug node ã䜿çšãã - oc get node ã§ããŒãåã調ã¹ã oc debug node/NODENAME ãå®è¡ããŸããäŸ: oc debug node/ip-10-0-133-117.ap-northeast-1.compute.internal ã©ã¡ãã®æ¹æ³ã§ç¢ºèªããŠãæ§ããŸãããã1.ã宿œããã«ã¯ãIPIã€ã³ã¹ããŒã«ã§äœæããVPCãµããããäžã«å€éšããã¢ã¯ã»ã¹ããããã®èžã¿å°ãµãŒããå¿
èŠã«ãªããŸãã ã©ã¡ããã®æ¹æ³ã§workerããŒãã«å
¥ã£ãŠãã€ã³ã¹ã¿ã³ã¹ã®ã€ã³ã¿ãŒãã§ãŒã¹ã確èªããŸãã # ip addr show dev ens3 2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000 link/ether 06:41:25:41:65:46 brd ff:ff:ff:ff:ff:ff inet 10.0.133.117/20 brd 10.0.143.255 scope global dynamic noprefixroute ens3 valid_lft 3255sec preferred_lft 3255sec inet6 fe80::b6d2:f22:84d1:e350/64 scope link noprefixroute valid_lft forever preferred_lft forever # ip addr show dev ens4 31: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000 link/ether 06:e5:2e:bf:58:3a brd ff:ff:ff:ff:ff:ff inet 10.0.201.11/24 brd 10.0.201.255 scope global dynamic noprefixroute ens4 valid_lft 2308sec preferred_lft 2308sec inet6 fe80::2af9:7855:12d6:f03d/64 scope link noprefixroute valid_lft forever preferred_lft forever ens3 ãå
ã
ã€ã³ã¹ã¿ã³ã¹ãæã£ãŠããENIã§ã ens4 ãå
ã»ã©è¿œå ããENIã§ãã ã«ãŒãã£ã³ã°ããŒãã«ãèŠãŠã¿ãŸãã # ip route show default via 10.0.128.1 dev ens3 proto dhcp metric 100 default via 10.0.201.1 dev ens4 proto dhcp metric 101 10.0.128.0/20 dev ens3 proto kernel scope link src 10.0.140.137 metric 100 10.0.201.0/24 dev ens4 proto kernel scope link src 10.0.201.11 metric 101 10.128.0.0/14 dev tun0 scope link 172.30.0.0/16 dev tun0 ens4ã«ãDHCPããããã£ãããã©ã«ãã²ãŒããŠã§ã€ãèšå®ãããŠããŠæ°æã¡æªãã®ã§ãens4ã§ã¯DHCPã§IPã¢ãã¬ã¹ã ããã€ãã(DHCPã«ããããã©ã«ãã²ãŒããŠã§ã€ãšDNSèšå®ãããªã)ããã«ããŸãã # nmcli c s NAME UUID TYPE DEVICE Wired connection 1 e2fe62b6-e972-3d43-8023-320f467fec57 ethernet ens3 Wired connection 2 76577e5a-43d2-3f3d-b2d9-a93a067cf42c ethernet ens4 ens4ã®Connection nameã§ãã Wired connection 2 ã«å¯ŸããŠã ipv4.ignore-auto-dns ãš ipv4.ignore-auto-routes ãæå¹åããŸãã # nmcli c m 'Wired connection 2' ipv4.ignore-auto-dns yes ipv4.ignore-auto-routes yes # sudo nmcli c d 'Wired connection 2' Connection 'Wired connection 2' successfully deactivated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/2) # sudo nmcli c u 'Wired connection 2' Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/3) ããã§ãäœåãªããã©ã«ãã²ãŒããŠã§ã€ã®èšå®ãæ¶ããŸããã # ip r default via 10.0.128.1 dev ens3 proto dhcp metric 100 10.0.128.0/20 dev ens3 proto kernel scope link src 10.0.140.137 metric 100 10.0.201.0/24 dev ens4 proto kernel scope link src 10.0.201.11 metric 101 10.128.0.0/14 dev tun0 scope link 172.30.0.0/16 dev tun0 åworkerããŒãã«å¯ŸããŠãã®æäœã宿œããŠãããŸãã Multusã®ã»ããã¢ãã æ¬¡ã«Multusã®èšå®ãè¡ããŸããåºæ¬çã« ãã¡ã ã®æé ã«åŸããŸãã ãŸãããããã§ãããupstreamã®Kubernetes+Multusã䜿ã£ãåã®èšäºã§ã¯ã --- apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: ipvlan-conf spec: config: '{ "cniVersion": "0.3.1", "plugins": [ { "type": "ipvlan", "capabilities": { "ips": true }, "master": "eth1", "ipam": { "type": "static" } }, { "capabilities": { "mac": true }, "type": "tuning" } ] }' ã®ãããªmanifestã䜿ã£ãŠNewtorkAttachmentDefinitionã®Custom ResourceãäœæããŸããã äžæ¹OpenShift v4.xã§ã¯ãCNIã®èšå®ã¯å
šãŠ CNO (Cluster Network Operator) [ 5 ]ãåžãä»çµã¿ã«ãªã£ãŠããŸãããããããŸããŠãCNOã«å¯ŸããŠNetworkAttachmentDefinitionãäœæããŠãããããæç€ºããå¿
èŠããããŸããå
·äœçã«ã¯ã $ oc edit networks.operator.openshift.io cluster ãå®è¡ãã spec.additionalNetworks ã®äžã«ãNetworkAttachmentDefinitionã®Custom Resourceã§èšå®ããå
容ãèšèŒããŸããæ¬èšäºã®æ€èšŒã§ã¯ spec: additionalNetworks: - name: network-ipvlan type: Raw rawCNIConfig: '{ "cniVersion": "0.3.1", "type": "ipvlan", "mode": "l2", "capabilities": {"ips": true}, "master": "ens4", "ipam": { "type": "static", "routes": [ {"dst": "10.0.202.0/24", "gw": "10.0.201.1"}, {"dst": "10.0.203.0/24", "gw": "10.0.201.1"} ] } }' ã®ãããªå
容ãèšèŒããŸãã[ 6 ]ã ãã®å
容ã¯ã ap-northeast-1a ãªãŒãžã§ã³(远å ãµããããã®IPv4ã¢ãã¬ã¹ãããã¯ã 10.0.201.0/24 )ã®workerããŒãã«Podããããã€ãããå Žåã®èšå®ã§ããstatic routeã®èšå®ã¯ããããã€å
ã®workerããŒãããããªãŒãžã§ã³ããšã«å€ããå¿
èŠããããŸãã®ã§ã泚æãã ããã äžèš oc edit networks.operator.openshift.io cluster ã§ç·šéããå
容ã¯ã oc patch ã䜿ã£ãŠ oc patch networks.operator.openshift.io cluster --type merge -p '{"spec": {"additionalNetworks": [{"name": "network-ipvlan", "type": "Raw", "rawCNIConfig": "{\"cniVersion\": \"0.3.1\", \"type\": \"ipvlan\", \"mode\": \"l2\", \"capabilities\": { \"ips\": true }, \"master\": \"ens4\", \"mode\": \"l2\", \"capabilities\": { \"ips\": true }, \"ipam\": {\"type\": \"static\", \"routes\": [{ \"dst\": \"10.0.202.0/24\", \"gw\": \"10.0.201.1\" }, { \"dst\": \"10.0.203.0/24\", \"gw\": \"10.0.201.1\" }]}}"}]}}' ããã㯠oc patch networks.operator.openshift.io cluster --type json -p ' [{ "op": "add", "path": "/spec/additionalNetworks", "value": [] },{ "op": "add", "path": "/spec/additionalNetworks/0", "value": { name: "network-ipvlan", type: "Raw", rawCNIConfig: "{ \"cniVersion\": \"0.3.1\", \"type\": \"ipvlan\", \"mode\": \"l2\", \"capabilities\": { \"ips\": true }, \"master\": \"ens4\", \"ipam\": { \"type\": \"static\", \"routes\": [ { \"dst\": \"10.0.202.0/24\", \"gw\": \"10.0.201.1\" }, { \"dst\": \"10.0.203.0/24\", \"gw\": \"10.0.201.1\" } ] } }" } } ]' ã®ããã«åæ ãããããšãã§ããŸãã CNOã®èšå®å€æŽãçµãã£ããã oc get net-attach-def ãå®è¡ããäœæããNetworkAttachmentDefinitionã®å
容ã確èªããŸãã $ oc get net-attach-def network-ipvlan -o yaml apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: creationTimestamp: "2020-01-24T14:57:36Z" generation: 1 name: network-ipvlan <snip> spec: config: '{"cniVersion": "0.3.1", "type": "ipvlan", "mode": "l2", "capabilities": { "ips": true }, "master": "ens4", "mode": "l2", "capabilities": { "ips": true }, "ipam": {"type": "static", "routes": [{ "dst": "10.0.202.0/24", "gw": "10.0.201.1" }, { "dst": "10.0.203.0/24", "gw": "10.0.201.1" }]}}' Podã®ããã〠1ã€ç®ã®Podã®ãããã€ æ¬æ€èšŒã§ã¯ãäžèšã®ãããªèšå®ã®Podããããã€ããŸãã ipvlanã§ãã¹ãã®ens4çµç±ã§è¿œå ãµããããã«æ¥ç¶ãã Podã®ããã©ã«ãã²ãŒããŠã§ã€ã¯ãããã©ã«ãCNIãã©ã°ã€ã³ã§ããOpenShift-SDNåŽ(ens3ã§æ¥ç¶ããŠãããIPIã€ã³ã¹ããŒã©ãäœæãããµãããã)ã«ãã ipvlanã§æ¥ç¶ãã远å ãµããããçµç±ã§Podééä¿¡ãã§ãããããstatic routeãèšå®ãã ãŸããap-northeast-1aã®AZã«ããworkerããŒãã«ãããã€ããããšãèããŸããæºåãšããŠãåããŒããã©ã®AZã«ãããã確èªããŸãã $ aws ec2 describe-instances --filter Name=vpc-id,Values=${vpc_id} | jq -r '.Reservations[] | .Instances[] | [.PrivateDnsName, (.Tags[] | select(.Key == "Name").Value)] | @text' ["ip-10-0-158-197.ap-northeast-1.compute.internal","ocp43-9ld5l-worker-ap-northeast-1c-cphxt"] ["ip-10-0-149-79.ap-northeast-1.compute.internal","ocp43-9ld5l-master-1"] ["ip-10-0-143-118.ap-northeast-1.compute.internal","ocp43-9ld5l-master-0"] ["ip-10-0-133-117.ap-northeast-1.compute.internal","ocp43-9ld5l-worker-ap-northeast-1a-vcnxn"] ["ip-10-0-167-226.ap-northeast-1.compute.internal","ocp43-9ld5l-master-2"] ["ip-10-0-162-156.ap-northeast-1.compute.internal","ocp43-9ld5l-worker-ap-northeast-1d-x62nl"] ["ip-10-0-46-214.ap-northeast-1.compute.internal","bastion-ocp43"] ap-northeast-1aã«ããworkerããŒã㯠ip-10-0-133-117.ap-northeast-1.compute.internal ã§ããããšãããããŸããã æ¬¡ã«ãã®ããŒãã®ã©ãã«ã確èªããŸãã $ oc get node ip-10-0-133-117.ap-northeast-1.compute.internal --show-labels NAME STATUS ROLES AGE VERSION LABELS ip-10-0-133-117.ap-northeast-1.compute.internal Ready worker 3d12h v1.16.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m4.large,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=ap-northeast-1,failure-domain.beta.kubernetes.io/zone=ap-northeast-1a,kubernetes.io/arch=amd64,kubernetes.io/hostname=ip-10-0-133-117,kubernetes.io/os=linux,node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhcos ããŒãã®ã»ã¬ã¯ã¿ãšããŠãã©ãã« kubernetes.io/hostname=ip-10-0-133-117 ã䜿ãããšã«ããŸãã ãã®ã©ãã«ã nodeSelector ã§æå®ããipvlanã®è¿œå ã€ã³ã¿ãŒãã§ãŒã¹ã«å¯Ÿã㊠10.0.201.27/24 ãä»äžããPod ipvlsan-1 ã®manifestã¯æ¬¡ã®ããã«ãªããŸãã --- apiVersion: v1 kind: Pod metadata: name: ipvlan-1 labels: app: multus-ipvlan annotations: k8s.v1.cni.cncf.io/networks: '[ { "name": "network-ipvlan", "ips": [ "10.0.201.27/24" ] } ]' spec: containers: - name: centos-tools image: docker.io/centos/tools:latest command: - /sbin/init securityContext: privileged: true nodeSelector: kubernetes.io/hostname: ip-10-0-133-117 ãã®ãã¡ã€ã«ã ipvlan-ocp-27-1.yaml ã«ä¿åããŠãäžèšã³ãã³ãã§Podããããã€ããŸãã $ oc create -f ipvlan-ocp-27-1.yaml Podãèµ·åããããæåŸ
ããèšå®ãåæ ãããŠããã確èªããŸãã $ oc rsh ipvlan-1 ip -d addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 3: eth0@if35: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8951 qdisc noqueue state UP group default link/ether 0a:58:0a:80:02:1d brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 veth numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 inet 10.128.2.29/23 brd 10.128.3.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::58e8:eeff:fe0a:45bf/64 scope link valid_lft forever preferred_lft forever 4: net1@if31: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UNKNOWN group default link/ether 06:e5:2e:bf:58:3a brd ff:ff:ff:ff:ff:ff promiscuity 0 ipvlan mode l2 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 inet 10.0.201.27/24 brd 10.0.201.255 scope global net1 valid_lft forever preferred_lft forever inet6 fe80::6e5:2e00:1bf:583a/64 scope link valid_lft forever preferred_lft forever Multusã«ããipvlanã®è¿œå ã€ã³ã¿ãŒãã§ãŒã¹ã net1 ãšããŠãæå®ããIPã¢ãã¬ã¹ãä»äžããç¶æ
ã§ååšããããšã確èªã§ããŸãããæ¬¡ã«Podã®ã«ãŒãã£ã³ã°ããŒãã«ã確èªããŸãã $ oc rsh ipvlan-1 ip route show default via 10.128.2.1 dev eth0 10.0.201.0/24 dev net1 proto kernel scope link src 10.0.201.27 10.0.202.0/24 via 10.0.201.1 dev net1 10.0.203.0/24 via 10.0.201.1 dev net1 10.128.0.0/14 dev eth0 10.128.2.0/23 dev eth0 proto kernel scope link src 10.128.2.29 172.30.0.0/16 via 10.128.2.1 dev eth0 224.0.0.0/4 dev eth0 ç¡äºãNetworkAttachmentDefinitionã§ä»èŸŒãã§ããã 10.0.202.0/24 ããã³ 10.0.203.0/24 å®ãŠã®static routeãèšå®ãããŠããŸãã ãã®static routeã«ãããipvlanã®è¿œå ã€ã³ã¿ãŒãã§ãŒã¹ã䜿ã£ãéä¿¡ããªãŒãžã§ã³(workerããŒã)ããŸããã§è¡ãããšãã§ããããã«ãªããŸãã 2ã€ç®ã®Podã®ããã〠次ã¯ãap-northeast-1cã®AZã«ããworkerããŒãã«Podããããã€ããŸãã ãã®AZã®è¿œå ãµãããã㯠10.0.202.0/24 ã§ãããããä»ãªãŒãžã§ã³ã®è¿œå ãµãããããšéä¿¡ããããã«ã¯ å®å
nexthop 10.0.201.0/24 10.0.202.1 10.0.203.0/24 10.0.202.1 ã®ãããªstatic routeãå¿
èŠã«ãªããŸãããããåæ ããããã«ãNetworkAttachmentDefinition network-ipvlan ã«æžããstatic routeèšå®ã®å
å®¹ãæ¬¡ã®ããã«å€æŽããŸãã spec: additionalNetworks: - name: network-ipvlan type: Raw rawCNIConfig: '{ "cniVersion": "0.3.1", "type": "ipvlan", "mode": "l2", "capabilities": {"ips": true}, "master": "ens4", "ipam": { "type": "static", "routes": [ {"dst": "10.0.201.0/24", "gw": "10.0.202.1"}, {"dst": "10.0.203.0/24", "gw": "10.0.202.1"} ] } }' åè¿°ã® oc edit networks.operator.openshift.io cluster ãããã¯äžèšã® oc patch ã§å€æŽã宿œããŸãã $ oc patch networks.operator.openshift.io cluster --type merge -p '{"spec": {"additionalNetworks": [{"name": "network-ipvlan", "type": "Raw", "rawCNIConfig": "{\"cniVersion\": \"0.3.1\", \"type\": \"ipvlan\", \"mode\": \"l2\", \"capabilities\": { \"ips\": true }, \"master\": \"ens4\", \"mode\": \"l2\", \"capabilities\": { \"ips\": true }, \"ipam\": {\"type\": \"static\", \"routes\": [{ \"dst\": \"10.0.201.0/24\", \"gw\": \"10.0.202.1\" }, { \"dst\": \"10.0.203.0/24\", \"gw\": \"10.0.202.1\" }]}}"}]}}' network-ipvlan ãæåŸ
ããå
容ã«å€ãã£ãããšã確èªã§ããããæ¬¡ã®ãããªmanifestã§Podãèµ·åããŸãã --- apiVersion: v1 kind: Pod metadata: name: ipvlan-2 labels: app: multus-ipvlan annotations: k8s.v1.cni.cncf.io/networks: '[ { "name": "network-ipvlan", "ips": [ "10.0.202.27/24" ] } ]' spec: containers: - name: centos-tools image: docker.io/centos/tools:latest command: - /sbin/init securityContext: privileged: true nodeSelector: kubernetes.io/hostname: ip-10-0-158-197 nodeSelectorãšããŠãããã€å
ã®AZã«ããworkerããŒãã«ã€ããã©ãã«ãæå®ããannotationã§Podã®IPã¢ãã¬ã¹ãæå®ããŠããŸãã Podã®ãããã€ãã§ããããPodã«å
¥ã£ãŠIPã¢ãã¬ã¹ã確èªããŸãã $ oc rsh ipvlan-2 ip -d addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 3: eth0@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8951 qdisc noqueue state UP group default link/ether 0a:58:0a:83:00:0e brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 veth numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 inet 10.131.0.14/23 brd 10.131.1.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::9010:bfff:fefb:aad7/64 scope link valid_lft forever preferred_lft forever 4: net1@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UNKNOWN group default link/ether 0a:cd:aa:50:a0:b8 brd ff:ff:ff:ff:ff:ff promiscuity 0 ipvlan mode l2 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 inet 10.0.202.27/24 brd 10.0.202.255 scope global net1 valid_lft forever preferred_lft forever inet6 fe80::acd:aa00:150:a0b8/64 scope link valid_lft forever preferred_lft forever ã«ãŒãã£ã³ã°ããŒãã«ã確èªããŸãã $ oc rsh ipvlan-2 ip route show default via 10.131.0.1 dev eth0 10.0.201.0/24 via 10.0.202.1 dev net1 10.0.202.0/24 dev net1 proto kernel scope link src 10.0.202.27 10.0.203.0/24 via 10.0.202.1 dev net1 10.128.0.0/14 dev eth0 10.131.0.0/23 dev eth0 proto kernel scope link src 10.131.0.14 172.30.0.0/16 via 10.131.0.1 dev eth0 224.0.0.0/4 dev eth0 ãã®AZçšã®static routeèšå®ãç¡äºåæ ãããŠããããšã確èªã§ããŸãããããã«Pod ipvlan-1 ãšçéã§ããããšã確èªããŠãããŸãã $ oc rsh ipvlan-2 ping -c 3 10.0.201.27 PING 10.0.201.27 (10.0.201.27) 56(84) bytes of data. 64 bytes from 10.0.201.27: icmp_seq=1 ttl=64 time=2.35 ms 64 bytes from 10.0.201.27: icmp_seq=2 ttl=64 time=2.34 ms 64 bytes from 10.0.201.27: icmp_seq=3 ttl=64 time=2.43 ms --- 10.0.201.27 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 2.345/2.378/2.432/0.038 ms ã³ã³ãããã¹ãã§ã®tcpdumpã®å®è¡ å®éã«è¿œå ãµãããããéã£ãŠéä¿¡ã§ããŠããããããã¹ãäžã§tcpdumpããŠç¢ºèªããããªããããããŸãããã³ã³ãããã¹ãã®RHCOSã¯ã³ã³ããå®è¡ã«ç¹åããŠãããããtcpdumpçã®ããŒã«ã¯å
¥ã£ãŠããªãã®ã§ããã support-tools ãšããã³ã³ããã䜿ããštcpdumpã§ããããã«ãªããŸãã ãŸãã1ã€ç®ã®Podããããã€ããããŒãã«å
¥ããŸãã $ oc debug node/ip-10-0-133-117.ap-northeast-1.compute.internal Starting pod/ip-10-0-133-117ap-northeast-1computeinternal-debug ... To use host binaries, run `chroot /host` Pod IP: 10.0.133.117 If you don't see a command prompt, try pressing enter. sh-4.2# chroot /host sh-4.4# 次㫠toolbox ãšããã³ãã³ããå®è¡ããŸã[ 7 ]ããã®ã³ãã³ããå®è¡ãããšã support-tools ã³ã³ãããååŸããprivilegedãã€ãã¹ããããã¯ãŒã¯ãæŽãã ç¶æ
ã§ãããã€ãããã®äžã«å
¥ããŸããããã§ããã¹ãã®ã€ã³ã¿ãŒãã§ãŒã¹ã«å¯ŸããŠtcpdumpã䜿ããããã«ãªããŸã[ 8 ]ã sh-4.4# toolbox Trying to pull registry.redhat.io/rhel8/support-tools... Getting image source signatures Copying blob 1457434f891b done Copying blob cb3c77f9bdd8 done Copying blob fd8daf2668d1 done Copying config 517597590f done Writing manifest to image destination Storing signatures 517597590ff4236b0e5e3efce75d88b2b238c19a58903f59a018fc4a40cd6cce Spawning a container 'toolbox-' with image 'registry.redhat.io/rhel8/support-tools' Detected RUN label in the container image. Using that as the default... command: podman run -it --name toolbox- --privileged --ipc=host --net=host --pid=host -e HOST=/host -e NAME=toolbox- -e IMAGE=registry.redhat.io/rhel8/support-tools:latest -v /run:/run -v /var/log:/var/log -v /etc/machine-id:/etc/machine-id -v /etc/localtime:/etc/localtime -v /:/host registry.redhat.io/rhel8/support-tools:latest [root@ip-10-0-133-117 /]# è£ã§ ipvlan-2 ãã ipvlan-1 ã«å¯ŸããŠpingãæã¡ç¶ããªããããã® support-tools ã³ã³ããã®äžã§ ens4 ã«å¯ŸããŠtcpdumpãå®è¡ããŠã¿ãŸãã [root@ip-10-0-133-117 /]# tcpdump -enni ens4 icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ens4, link-type EN10MB (Ethernet), capture size 262144 bytes 15:59:21.983883 0a:cd:aa:50:a0:b8 > 0a:65:d7:a5:30:62, ethertype IPv4 (0x0800), length 98: 10.0.202.27 > 10.0.201.27: ICMP echo request, id 16088, seq 4, length 64 15:59:21.986235 0a:65:d7:a5:30:62 > 0a:cd:aa:50:a0:b8, ethertype IPv4 (0x0800), length 98: 10.0.201.27 > 10.0.202.27: ICMP echo reply, id 16088, seq 4, length 64 15:59:22.985390 0a:cd:aa:50:a0:b8 > 0a:65:d7:a5:30:62, ethertype IPv4 (0x0800), length 98: 10.0.202.27 > 10.0.201.27: ICMP echo request, id 16088, seq 5, length 64 15:59:22.987738 0a:65:d7:a5:30:62 > 0a:cd:aa:50:a0:b8, ethertype IPv4 (0x0800), length 98: 10.0.201.27 > 10.0.202.27: ICMP echo reply, id 16088, seq 5, length 64 15:59:23.985892 0a:cd:aa:50:a0:b8 > 0a:65:d7:a5:30:62, ethertype IPv4 (0x0800), length 98: 10.0.202.27 > 10.0.201.27: ICMP echo request, id 16088, seq 6, length 64 15:59:23.988242 0a:65:d7:a5:30:62 > 0a:cd:aa:50:a0:b8, ethertype IPv4 (0x0800), length 98: 10.0.201.27 > 10.0.202.27: ICMP echo reply, id 16088, seq 6, length 64 ipvlan-1 ã ipvlan-2 ã®Podéã§ããã¹ãã®ens4ã䜿ã£ãŠICMP echo request/replyã®ããåããããŠããããšã確èªã§ããŸããã åé¡å€å¥ã®ãã³ã NetworkAttachmentDefinitionãæåŸ
ã©ããèšå®ãããªãå Žå NetworkAttachmentDefinitionãæåŸ
ã©ããèšå®ãããªãå Žåã¯CNO Podã®ãã°ã確èªããŠãã ããã ãŸãCNOã®Podåã確èªããŸãã $ oc -n openshift-network-operator get pod NAME READY STATUS RESTARTS AGE network-operator-5c7c7dc988-vlw6v 1/1 Running 0 25h ãã®Podã«å¯ŸããŠã oc logs ãå®è¡ããŸãã $ oc -n openshift-network-operator logs -f network-operator-5c7c7dc988-vlw6v Podã®èµ·åã«å€±æããããPod manifestã®èšå®ãããŸãPodã«åæ ãããªãå Žå Podã®èµ·åã«å€±æãããPod manifestã®èšå®ãããŸãPodã«åæ ãããªãå Žåã¯ããŸã oc describe pod ã®åºåã確èªããŠãã ãããããã§åœãããã€ããªãå Žåã¯ããããã€å
ã®workerããŒãã®ãã°ã確èªããŸããcrioãµãŒãã¹ã®ãã°ã«ãšã©ãŒã¡ãã»ãŒãžãåºãŠããªãããæåã«èŠãã®ããããšæããŸãã å
·äœçã«ã¯ãworkerããŒãã«sshããã㯠oc debug node ã§ãã°ã€ã³ããjournalctlã§ç¢ºèªããŸã[ 9 ]ã # journalctl -u crio -f æåŸã« IPIã€ã³ã¹ããŒã«ã§æ§ç¯ããOpenShiftã¯ã©ã¹ã¿ã«å¯ŸããŠãµãããããšNICã远å ããMultusã䜿ã£ãŠ2ã€ã®ãããã¯ãŒã¯ã«æ¥ç¶ããPodããããã€ãã远å ãµããããçµç±ã§Podééä¿¡ãã§ããããšã確èªããŸããã ãããã¯ãŒã¯æ§æäžãPodããããã€ããAZããšã«ç°ãªãstatic routeãå
¥ããå¿
èŠãããããã®ãããããã€å
ã®AZããšã«NetworkAttachmentDefinitionã®Custom ResourceãäœãçŽããŸãããããã¯ãšãŠãç
©éã§ãã§ããã°ãããããããŸããã察å¿çã¯ããã€ãèããããŸãã 察å¿ç1: ã³ã³ãããã¹ãåŽã§static routeãèšå®ãã Podã«ã¯static routeãæžãããã³ã³ãããã¹ãäžã§static routeãèšå®ããŠããåãããã«ãã±ãããæµãããšãã§ããŸãã å
·äœçã«ã¯ãã³ã³ãããã¹ãã«ãã°ã€ã³ãã远å ãµããããåŽã®ã€ã³ã¿ãŒãã§ãŒã¹ ens4 ã«å¯Ÿã㊠# ip route add to 10.0.202.0/24 via 10.0.201.1 ããã㯠# nmcli con mod ens4 +ipv4.routes "10.0.202.0/24 10.0.201.1" ã®ããã«static routeãèšå®ããŸã (åè
ã¯åèµ·åãããšèšå®ãæ¶ããŸãããåŸè
ã¯åèµ·åããŠãstatic routeèšå®ãåæ ãããŸã)ã 察å¿ç2: Pod ã®manifestã«æžã äŸãã°staticã«IPã¢ãã¬ã¹ãæ¯ãIPAMã®èšå®ãPod manifestã®annotationã«åã蟌ãã ããã«ãPodã«æ³šå
¥ããstatic routeãPod manifestã«åã蟌ãããšãã§ããã°ããããã€å
ã®AZãå€ãããã³ã«NetworkAttachmentDefinitionã®static routeèšå®ã倿Žããå¿
èŠããªããªããŸãããŸãã察å¿ç1ã®å Žåã¯ãã³ã³ãããã¹ãã§äžåŸã«static routeãæ±ºãŸã£ãŠããŸããŸãã®ã§ãPodããšã«static routeã®èšå®ãåãæ¿ãããå Žåã¯ãã¡ãã®æ¹æ³ã䜿ãããšãããããã§ãããã®æ¹æ³ã®å®çŸæ¹æ³ã«ã€ããŠã¯upstreamã§è°è«äžã§[ 10 ]ãOpenShift v4.3ã§ã¯ãŸã 䜿ããŸãããä»åŸã®Multusã®é²åã«æåŸ
ããŸãããã è¬èŸ ä»åããRed Hatã§NFVPEãããŠããæããã«ããããæããŠããã ããŸãããããããšãããããŸããïŒ ã€ãæè¿åºãã°ããã®æ°ããããŒãžã§ã³ã§ãïŒ ↩ æ¥æ¬èªç㯠ãã¡ã ã«ãããŸããGAçŽåŸãªã®ã«æ¥æ¬èªã®ããã¥ã¡ã³ãããããªããŠçŽ æµã§ããã ↩ æ¥æ¬èªç㯠ãã¡ã ↩ RHCOS(Red Hat CoreOS) ã®OSã€ã¡ãŒãžã¯ãRHEL8ãšåãrpmããã±ãŒãžã䜿ãã€ã€(ã€ãŸãå
¥ã£ãŠãããã€ããªã¯RHEL8ãšåã)ã rpm-ostree ã䜿ã£ãŠOSã€ã¡ãŒãžãäœæããŠããŸããrpm-ostree㯠Project Atomic ç±æ¥ã®ããŒã«ã§ããŠãŒã¶ãŒããŒã¿(/etcã/var)ãžã®åœ±é¿ãªãã«OSã€ã¡ãŒãžãã¢ãããã¯ã«ããŒã«ããã¯ã§ããä»çµã¿ãæäŸããŸããRHCOSã¯ãRHELã®rpmããã±ãŒãžãrpm-ostreeçã®Project Atomicç±æ¥ã®ããŒã«ãCoreOS瀟ã®Container Linuxã§äœ¿ã£ãŠãã Ignition çã®ä»çµã¿ãæªéåäœããããã£ã¹ããªãã¥ãŒã·ã§ã³ãšèšããŸããRHCOSèªäœã¯(ä»ã®ãšãã)ãOCP(OpenShiftã®è£œåç)ã®ããŒããåããOSãšããŠã®äœ¿ãæ¹ãããµããŒããããŸãã(ã€ãŸãæ±çšOSãšããŠäœ¿ããŠãŒã¹ã±ãŒã¹ã¯ãµããŒããããŸãã)ãäžæ¹ãRed Hat CoreOSã®upstreamã§ãã Fedora CoreOS ã¯ãã³ã³ãããã¹ãã«ç¹åããæ±çšOSãšããŠäœ¿ãããããšãæ³å®ããŠéçºãé²ããããŠããŸãã ↩ Cluster Network Operatorã«ã€ããŠã¯ã ãã¡ã ããåç
§ãã ããã ↩ åã®èšäºã§ã¯ãNetworkAttachmentDefinitionã®å
容ãšããŠã plugins é
åã®äžã« type: ipvlan ã®èšå®ãš type: tuning ã®èšå®ãæžããŠããŸããããOpenShift v4.3ã®Multusã ãšãã®æžãæ¹ãCNOã«ã¯ããããŠããŸãã®ã§ãå°ãå€ãæžãæ¹ãããŠããŸãã ↩ toolboxã¯ã·ã§ã«ã¹ã¯ãªããã§ãäžã§ (1) podman login registry.redhat.io (2) podman pull registry.redhat.io/rhel8/support-tools (3) podman container runlabel RUN registry.redhat.io/rhel8/support-tools ã¿ãããªæãã®ããšããã£ãŠããŸãã ↩ Red Hat補åã§åé¡å€å¥æã®æ
å ±åéã«äœ¿ãsosreportã³ãã³ããã support-tools ã³ã³ããçµç±ã§å®è¡ã§ããããã«ãªããŸãã ↩ äœè«ã§ããã export SYSTEMD_LESS=FRXMK ãå®è¡ããŠãããšãjournalctlã³ãã³ãã®ããŒãžã£ã暪ã¹ã¯ããŒã«ããã«ã¿ãŒããã«ã®å³ç«¯ã§æãè¿ããŠãããã®ã§ãã¡ãã£ãšã ã幞ãã«ãªããŸãã ↩ 察å¿ç2ãå®çŸããå®è£
ãšããŠã cni-route-override ãææ¡ãããŠããŸããPod manifestã®annotationã«æžããstatic routeã®èšå®ãã cni-args ããå·®ã蟌ã¿ãŸãã ↩