On Wed, Mar 11, 2020 at 2:08 AM Vadim M Basyrov <vbasy...@technoserv.com> wrote: > > Hello! > > I write to you because I can’t solve one issue regarding open-v-switch. > > We have the OpenStack deployment – version Queens. >
In the case of openstack, neutron ovs agent programs the flows right ? I think you should investigate whether neutron ovs agent is programming the required flows or not. Thanks Numan > 11 nodes – 3 control nodes and 8 compute nodes. > > There are seven compute nodes with ovs-vswitchd (Open vSwitch) 2.9.0 and last > one compute node with ovs-vswitchd (Open vSwitch) 2.9.5 (from the start we > had ovs-vswitchd (Open vSwitch) 2.9.0 like as on other nodes). > > We upgraded ovs-vswitchd (Open vSwitch) 2.9.5 on the last compute node for > test and wanted to figure out that this problem will be persists or not. > > > When we launch virtual machine on first seven nodes, virtual machine works > and gets ip-addresses from dhcp without problem. > > But when we launch virtual machine or migrate virtual machine on the last > compute node virtual machine does not get ip-addresses from dhcp. > > > We investigated this problem and found one interesting moment. > > On the first seven nodes after creating virtual machine we can see correct > dump flow for private network of virtual machine. > > > Correct dump: > > root@dc2-compute03:~# ovs-ofctl dump-flows br-tun | grep 0x10057 > > cookie=0x7e0b101c14974b46, duration=8712088.227s, table=4, > n_packets=58210459, n_bytes=333529723572, idle_age=39, hard_age=65534, > priority=1,tun_id=0x10057 actions=mod_vlan_vid:3,resubmit(,9) > > cookie=0x7e0b101c14974b46, duration=2305838.428s, table=20, n_packets=0, > n_bytes=0, idle_age=65534, hard_age=65534, > priority=2,dl_vlan=3,dl_dst=fa:16:3e:f7:e1:46 > actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:6 > > cookie=0x7e0b101c14974b46, duration=2305820.317s, table=20, n_packets=54, > n_bytes=6045, idle_age=65534, hard_age=65534, > priority=2,dl_vlan=3,dl_dst=fa:16:3e:10:65:23 > actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:3 > > cookie=0x7e0b101c14974b46, duration=2305811.544s, table=20, n_packets=0, > n_bytes=0, idle_age=65534, hard_age=65534, > priority=2,dl_vlan=3,dl_dst=fa:16:3e:2c:92:0d > actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:10 > > cookie=0x7e0b101c14974b46, duration=2305795.834s, table=20, n_packets=45, > n_bytes=4175, idle_age=65534, hard_age=65534, > priority=2,dl_vlan=3,dl_dst=fa:16:3e:ea:d7:e4 > actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:4 > > cookie=0x7e0b101c14974b46, duration=2305795.238s, table=20, n_packets=36, > n_bytes=1865, idle_age=65534, hard_age=65534, > priority=2,dl_vlan=3,dl_dst=fa:16:3e:65:23:5d > actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:5 > > cookie=0x7e0b101c14974b46, duration=1027284.390s, table=20, n_packets=0, > n_bytes=0, idle_age=65534, hard_age=65534, > priority=2,dl_vlan=3,dl_dst=fa:16:3e:90:b8:73 > actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:8 > > cookie=0x7e0b101c14974b46, duration=1027268.690s, table=20, n_packets=0, > n_bytes=0, hard_timeout=300, idle_age=65534, hard_age=38, > priority=1,vlan_tci=0x0003/0x0fff,dl_dst=fa:16:3e:90:b8:73 > actions=load:0->NXM_OF_VLAN_TCI[],load:0x10057->NXM_NX_TUN_ID[],output:8 > > cookie=0x7e0b101c14974b46, duration=23829.098s, table=20, n_packets=220, > n_bytes=14712, hard_timeout=300, idle_age=166, hard_age=124, > priority=1,vlan_tci=0x0003/0x0fff,dl_dst=fa:16:3e:b6:4d:f1 > actions=load:0->NXM_OF_VLAN_TCI[],load:0x10057->NXM_NX_TUN_ID[],output:3 > > cookie=0x7e0b101c14974b46, duration=131.880s, table=20, n_packets=0, > n_bytes=0, hard_timeout=300, idle_age=131, > priority=1,vlan_tci=0x0003/0x0fff,dl_dst=fa:16:3e:10:65:23 > actions=load:0->NXM_OF_VLAN_TCI[],load:0x10057->NXM_NX_TUN_ID[],output:3 > > cookie=0x7e0b101c14974b46, duration=131.880s, table=20, n_packets=0, > n_bytes=0, hard_timeout=300, idle_age=131, > priority=1,vlan_tci=0x0003/0x0fff,dl_dst=fa:16:3e:65:23:5d > actions=load:0->NXM_OF_VLAN_TCI[],load:0x10057->NXM_NX_TUN_ID[],output:5 > > cookie=0x7e0b101c14974b46, duration=131.880s, table=20, n_packets=0, > n_bytes=0, hard_timeout=300, idle_age=131, > priority=1,vlan_tci=0x0003/0x0fff,dl_dst=fa:16:3e:ea:d7:e4 > actions=load:0->NXM_OF_VLAN_TCI[],load:0x10057->NXM_NX_TUN_ID[],output:4 > > cookie=0x7e0b101c14974b46, duration=2305817.320s, table=22, n_packets=27156, > n_bytes=3159361, idle_age=24888, hard_age=65534, priority=1,dl_vlan=3 > actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:3,output:4,output:5,output:6,output:8,output:10 > > > On the last compute node after creating virtual machine we have incorrect > dump flow or incomplete. In this case we don’t see any “NXM_OF_VLAN_TCI” > output strings to MAC-addresses of DHCP. > > > Incorrect dump: > > root@dc2-compute09:~# ovs-ofctl dump-flows br-tun | grep 0x10057 > > cookie=0x189ebcbc0bc4a360, duration=3006.464s, table=4, n_packets=124, > n_bytes=24219, idle_age=21, priority=1,tun_id=0x10057 > actions=mod_vlan_vid:6,resubmit(,9) > > cookie=0x189ebcbc0bc4a360, duration=3003.196s, table=20, n_packets=0, > n_bytes=0, idle_age=3005, priority=2,dl_vlan=6,dl_dst=fa:16:3e:65:23:5d > actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:22 > > cookie=0x189ebcbc0bc4a360, duration=3003.189s, table=20, n_packets=0, > n_bytes=0, idle_age=3005, priority=2,dl_vlan=6,dl_dst=fa:16:3e:10:65:23 > actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:23 > > cookie=0x189ebcbc0bc4a360, duration=3003.183s, table=20, n_packets=0, > n_bytes=0, idle_age=3005, priority=2,dl_vlan=6,dl_dst=fa:16:3e:ea:d7:e4 > actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:24 > > cookie=0x189ebcbc0bc4a360, duration=3003.175s, table=20, n_packets=0, > n_bytes=0, idle_age=3005, priority=2,dl_vlan=6,dl_dst=fa:16:3e:0f:49:0c > actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:26 > > cookie=0x189ebcbc0bc4a360, duration=3003.169s, table=20, n_packets=0, > n_bytes=0, idle_age=3005, priority=2,dl_vlan=6,dl_dst=fa:16:3e:f7:e1:46 > actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:27 > > cookie=0x189ebcbc0bc4a360, duration=3003.160s, table=20, n_packets=0, > n_bytes=0, idle_age=3005, priority=2,dl_vlan=6,dl_dst=fa:16:3e:90:b8:73 > actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:29 > > cookie=0x189ebcbc0bc4a360, duration=3003.152s, table=20, n_packets=0, > n_bytes=0, idle_age=3005, priority=2,dl_vlan=6,dl_dst=fa:16:3e:2c:92:0d > actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:31 > > cookie=0x189ebcbc0bc4a360, duration=2904.220s, table=20, n_packets=0, > n_bytes=0, hard_timeout=300, idle_age=2904, hard_age=21, > priority=1,vlan_tci=0x0006/0x0fff,dl_dst=fa:16:3e:90:b8:73 > actions=load:0->NXM_OF_VLAN_TCI[],load:0x10057->NXM_NX_TUN_ID[],output:29 > > cookie=0x189ebcbc0bc4a360, duration=3003.156s, table=22, n_packets=63, > n_bytes=16788, idle_age=3, priority=1,dl_vlan=6 > actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:22,output:23,output:24,output:25,output:26,output:27,output:28,output:29,output:30,output:31 > > > OS on all hosts: > > NAME="Ubuntu" > > VERSION="16.04.4 LTS (Xenial Xerus)" > > ID=ubuntu > > ID_LIKE=debian > > PRETTY_NAME="Ubuntu 16.04.4 LTS" > > VERSION_ID="16.04" > > HOME_URL="http://www.ubuntu.com/"; > > SUPPORT_URL="http://help.ubuntu.com/"; > > BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"; > > VERSION_CODENAME=xenial > > UBUNTU_CODENAME=xenial > > > Kernel version on the first seven nodes: 4.4.0-130-lowlatency > > Kernel version on the last node: 4.4.0-170-generic (before was > 4.4.0-130-lowlatency when we found problem, we upgraded open-v-switch version > and kernel version) > > > I've attached two conf.db files of correct and incorrect node. Correct node > where network works well and incorrect node where network does not work well > (the last compute node). > > > "ovs-dpctl show" command dump of correct node. > > root@dc2-compute03:~# ovs-dpctl show > > system@ovs-system: > > lookups: hit:304985044225 missed:261957039 lost:108 > > flows: 918 > > masks: hit:4793394128544 total:17 hit/pkt:15.70 > > port 0: ovs-system (internal) > > port 1: vxlan_sys_4789 (vxlan: packet_type=ptap) > > port 2: br-tun (internal) > > port 3: vlan944 (internal) > > port 4: vlan941 (internal) > > port 5: vlan940 (internal) > > port 6: qvo89695e65-21 > > port 7: qvoc1d94fa7-f3 > > port 8: qvofd9e30ab-8b > > port 9: qr-4600256d-b3 (internal) > > port 10: qvo53120329-cc > > port 11: qr-36735708-b9 (internal) > > port 12: qr-2dcb6c75-e5 (internal) > > port 13: qr-de554b3e-ee (internal) > > port 14: qr-132fbe52-82 (internal) > > port 15: qr-3b3d0596-39 (internal) > > port 16: qvo21070b6b-44 > > port 17: qr-11fb5e85-16 (internal) > > port 18: qr-1d58adb4-45 (internal) > > port 19: qr-5a3db12a-f9 (internal) > > port 20: br-int (internal) > > port 21: qr-cfe0668e-2f (internal) > > port 22: qr-e04a70fc-ff (internal) > > port 23: qr-9f0848e5-e7 (internal) > > port 24: qr-6a9dec56-3f (internal) > > port 25: br0 (internal) > > port 26: enp59s0f0 > > port 27: enp216s0f1 > > port 28: vlan123 (internal) > > port 29: vlan943 (internal) > > port 30: qr-fc0718b4-e8 (internal) > > port 31: qr-ed5db182-b6 (internal) > > port 32: qvob6b62461-6c > > port 33: qvod2d02a50-13 > > port 36: qr-2fdc16d7-a3 (internal) > > port 39: qvoed4060b4-ed > > port 42: qvob6dd5c36-c5 > > port 43: qr-c091f908-ba (internal) > > port 45: fg-d63f59c5-8f (internal) > > port 46: qvo478e4755-8d > > port 47: qr-72a3e62b-c2 (internal) > > port 48: qvo7c88a4c8-31 > > port 49: qvoc8505821-7a > > port 50: qvo374ab104-81 > > port 52: qvode477341-12 > > port 53: qvo74fa0739-24 > > port 55: qvof415d31d-0e > > port 56: qvob727e682-40 > > port 57: qvo2676d8ec-53 > > port 58: qvoa87b5c32-85 > > port 59: qr-785b0b3f-78 (internal) > > port 60: qvo1192309a-b7 > > port 61: qvo736dffdf-8e > > port 62: qvo673e82c7-3c > > port 63: qvoe0ffcdfa-bd > > port 64: qvo34b3d411-dd > > port 65: qvoc032b53f-0b > > port 66: qvo7a77a59e-84 > > port 68: qvo859183e9-30 > > port 69: qvo70596e32-ba > > port 70: qvoebbb1420-a7 > > port 71: qvo800895af-c8 > > port 73: qvo186f5253-12 > > port 75: qr-a1164d85-22 (internal) > > port 76: qvob1c5ab98-ac > > port 79: qr-5ea9b7ad-8e (internal) > > port 81: qr-d20a18c2-0c (internal) > > port 84: qvo7b1ef463-1a > > port 85: qr-d14d71f2-4c (internal) > > port 86: qr-f6b5a89e-16 (internal) > > port 88: qvo0bc1c446-d0 > > port 89: qr-20b48617-ce (internal) > > port 98: qvo4a760e85-7e > > port 99: qvofcb1f977-e9 > > > "ovs-dpctl show" command dump of incorrect node. > > root@dc2-compute09:~# ovs-dpctl show > > system@ovs-system: > > lookups: hit:2747212618 missed:79153734 lost:0 > > flows: 161 > > masks: hit:16755259144 total:9 hit/pkt:5.93 > > port 0: ovs-system (internal) > > port 1: vxlan_sys_4789 (vxlan: packet_type=ptap) > > port 2: br-tun (internal) > > port 3: br0 (internal) > > port 4: enp216s0f1 > > port 5: enp59s0f0 > > port 6: bond0 (internal) > > port 7: vlan123 (internal) > > port 8: vlan943 (internal) > > port 9: vlan944 (internal) > > port 10: br-int (internal) > > port 11: vlan941 (internal) > > port 12: vlan940 (internal) > > port 13: qvoad8dfa6f-cf > > port 14: qr-1d58adb4-45 (internal) > > port 15: qr-c091f908-ba (internal) > > port 16: fg-d52bf3cc-aa (internal) > > > Have I right found that flow hasn't TCI string to DHCP? > > > Best regards, > > Vadim Basyyrov > > System engineer of > > Cloud solutions department > > > > _______________________________________________ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
