Hi Ben, Thanks for the patch. The fix looks good. We’ve also verified the fix. Eth0 qos is left intact.
root@59c90586bd96:/# /usr/share/openvswitch/scripts/ovn-ctl start_controller * Starting ovn-controller root@59c90586bd96:/# root@59c90586bd96:/# root@59c90586bd96:/# root@59c90586bd96:/# !ping ping 10.4.163.185 PING 10.4.163.185 (10.4.163.185) 56(84) bytes of data. 64 bytes from 10.4.163.185: icmp_seq=1 ttl=63 time=0.525 ms 64 bytes from 10.4.163.185: icmp_seq=2 ttl=63 time=0.368 ms ^C --- 10.4.163.185 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1000ms rtt min/avg/max/mdev = 0.368/0.446/0.525/0.081 ms root@59c90586bd96:/# root@59c90586bd96:/# root@59c90586bd96:/# tc qdisc show qdisc noqueue 0: dev lo root refcnt 2 qdisc noqueue 0: dev eth0 root refcnt 2 qdisc noqueue 0: dev vxlan_sys_4789 root refcnt 2 root@59c90586bd96:/# ps aux|grep ovn-controller root 22611 0.0 0.0 27336 2212 ? S<s 18:37 0:00 ovn-controller: monitoring pid 22612 (healthy) root 22612 0.0 0.0 27468 3832 ? S< 18:37 0:00 ovn-controller unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --no-chdir --log-file=/var/log/openvswitch/ovn-controller.log --pidfile=/var/run/openvswitch/ovn-controller.pid --detach --monitor root@59c90586bd96:/# Regards, Hexin On 2/1/17, 9:23 AM, "Ben Pfaff" <b...@ovn.org> wrote: >Thanks for locating the root cause. > >I sent out a pair of patches that, together, I expect would at least >work around the problem in the common case: > https://patchwork.ozlabs.org/patch/722628/ > https://patchwork.ozlabs.org/patch/722629/ >Do you think you could try them out? > >Thanks, > >Ben. > >On Wed, Feb 01, 2017 at 07:43:53AM +0000, Hexin Wang wrote: >> So this is a docker issue, >> http://stackissue.com/openshift/origin/sdn-set-veth-tx-queue-length-to-unblock-qos-11126.html >> Basically docker creates interface with an explicit 0 txqlen which causes >> the problem when QoS is switched to htb. >> >> Thanks. >> >> Hexin >> >> >> >> >> On 1/31/17, 6:18 PM, "ovs-discuss-boun...@openvswitch.org on behalf of Hexin >> Wang" <ovs-discuss-boun...@openvswitch.org on behalf of >> hexin.w...@nutanix.com> wrote: >> >> >Looks like the problem is because ovn is trying to touch eth0 qos. The >> >moment it changes qos_type from “linux-other” to “linux-htb”. Traffic stops >> >working. >> > >> >setup_qos (queue_map=0x7fffffffc230, egress_iface=0x63ae10 "eth0") at >> >ovn/controller/binding.c:240 >> >240 int error = netdev_open(egress_iface, NULL, &netdev_phy); >> >(gdb) n >> >241 if (error) { >> >(gdb) >> >251 smap_init(&qdisc_details); >> >(gdb) >> >252 if (netdev_get_qos(netdev_phy, &qdisc_type, &qdisc_details) != >> >0 || >> >(gdb) >> >253 qdisc_type[0] == '\0') { >> >(gdb) >> >252 if (netdev_get_qos(netdev_phy, &qdisc_type, &qdisc_details) != >> >0 || >> >(gdb) >> >257 if (strcmp(qdisc_type, OVN_QOS_TYPE)) { >> >(gdb) print qdisc_type >> >$12 = 0x7ffff78ef8ec "linux-other" >> >(gdb) print qdisc_details >> >$13 = {map = {buckets = 0x7fffffffc258, one = 0x0, mask = 0, n = 0}} >> >(gdb) n >> >258 error = netdev_set_qos(netdev_phy, OVN_QOS_TYPE, >> >&qdisc_details); >> >(gdb) n >> >259 if (error) { >> >(gdb) print error >> >$14 = 0 >> > >> > >> >Thanks. >> > >> >Hexin >> > >> > >> > >> >On 1/31/17, 6:12 PM, "Ritesh Rekhi" <ritesh.re...@nutanix.com> wrote: >> > >> >>+ Hexin >> >> >> >>-----Original Message----- >> >>From: Ben Pfaff [mailto:b...@ovn.org] >> >>Sent: Tuesday, January 31, 2017 3:49 PM >> >>To: Ritesh Rekhi <ritesh.re...@nutanix.com> >> >>Cc: ovs-discuss@openvswitch.org >> >>Subject: Re: [ovs-discuss] after setting ovs-vsctl set open . >> >>external-ids:ovn-encap-ip=<IP>, eth0 stops tansmitting any packets in >> >>docker container >> >> >> >>OK, what is OVN doing to the routing table or IP address configuration for >> >>the docker container? >> >> >> >>On Tue, Jan 31, 2017 at 11:12:40PM +0000, Ritesh Rekhi wrote: >> >>> That is what I am not able to find.If I have only 1 chassis connected >> >>> things work fine. Looks like OVN is doing something to the docker eth0 >> >>> interface .Again this problem does't happen on a Vm. >> >>> >> >>> It only happens on a docker container (:. >> >>> >> >>> Thx >> >>> Ritesh >> >>> >> >>> -----Original Message----- >> >>> From: Ben Pfaff [mailto:b...@ovn.org] >> >>> Sent: Tuesday, January 31, 2017 1:38 PM >> >>> To: Ritesh Rekhi <ritesh.re...@nutanix.com> >> >>> Cc: ovs-discuss@openvswitch.org >> >>> Subject: Re: [ovs-discuss] after setting ovs-vsctl set open . >> >>> external-ids:ovn-encap-ip=<IP>, eth0 stops tansmitting any packets in >> >>> docker container >> >>> >> >>> OK, well then, what is happening to the networking stack that causes the >> >>> problem? OVN doesn't do anything to the host routing table or IP >> >>> address configuration, at least not intentionally. What's happening to >> >>> the host? >> >>> >> >>> On Tue, Jan 31, 2017 at 06:36:10PM +0000, Ritesh Rekhi wrote: >> >>> > Hi Ben, >> >>> > >> >>> > Thx for the reply >> >>> > >> >>> > To repro this issue we don't need any entries in North bound and >> >>> > south bound database. >> >>> > >> >>> > Here are the steps >> >>> > >> >>> > 1) run my docker container >> >>> > >> >>> > docker run -itd --privileged -e CTL=10.4.163.249 >> >>> > riteshrekhi/ovnstuff:2.6.90 >> >>> > >> >>> > do docker attach and go inside docker container , there execute >> >>> > script test.sh which starts ovsdb-server, ovs-vswitchd and >> >>> > ovn-controller.After that it will connect to the south-db >> >>> > >> >>> > using command >> >>> > >> >>> > ovs-vsctl set open . external-ids:system-id=$uuid ovs-vsctl set open . >> >>> > external-ids:ovn-remote=tcp:$CTL:6642 >> >>> > ovs-vsctl set open . external-ids:ovn-encap-type=vxlan ovs-vsctl >> >>> > set open . external-ids:ovn-encap-ip=$IP >> >>> > >> >>> > >> >>> > Right after you execute this script and if connection to south-db >> >>> > establishes, you won't be able to ping out of eth0 >> >>> > >> >>> > >> >>> > 2) For version which is working you can do the same thing instead >> >>> > use riteshrekhi/ovnstuff >> >>> > >> >>> > Ritesh >> >>> > >> >>> > =================================================================== >> >>> > >> >>> > South-bound DB >> >>> > >> >>> > [root@controller ~]# ovn-sbctl show >> >>> > Chassis "0b50e2f7-7421-4536-9441-447af46c4cc4" >> >>> > hostname: "fe11f7697841" >> >>> > Encap vxlan >> >>> > ip: "10.4.150.6" >> >>> > options: {csum="true"} >> >>> > Chassis "0ebf5a6d-230a-4b9a-938e-976565d6993a" >> >>> > hostname: "23cc31791a67" >> >>> > Encap vxlan >> >>> > ip: "10.4.150.6" >> >>> > options: {csum="true"} >> >>> > Chassis "ahv-a-10.11.2.21-newname" >> >>> > hostname: "NTNX-16SM6C380006-A" >> >>> > Encap vxlan >> >>> > ip: "10.11.2.21" >> >>> > options: {csum="true"} >> >>> > Encap geneve >> >>> > ip: "10.11.2.21" >> >>> > options: {csum="true"} >> >>> > Chassis "2a083861-9841-4674-80ba-ed3c1bcf6918" >> >>> > hostname: "5af088595d12" >> >>> > Encap vxlan >> >>> > ip: "172.17.0.5" >> >>> > options: {csum="true"} >> >>> > Port_Binding "f4948a0f-6794-4140-9e4a-9616308467ee" >> >>> > Port_Binding "5c0e4397-7334-4f03-9ec3-fdc025588462" >> >>> > Port_Binding "f1fe3ff5-51ec-4790-ada8-c6e244e0cdd6" >> >>> > Port_Binding "f9e86d51-9b26-48f3-b2ad-8df492022a77" >> >>> > Chassis "de1cc8d2-fb4b-4819-822a-c68d921a84f3" >> >>> > hostname: "cfa5bc3de51c" >> >>> > Encap vxlan >> >>> > ip: "10.4.150.6" >> >>> > options: {csum="true"} >> >>> > >> >>> > >> >>> > North bound DB >> >>> > >> >>> > ovn-nbctl show >> >>> > switch 3a2d4174-27c9-46bc-9191-1d52ab16c057 >> >>> > (neutron-f0a1215f-cd3c-4c0e-8653-95c7d115731a) >> >>> > port c1c22fbb-5928-41c8-aa1a-c5a746c22118 >> >>> > addresses: ["fa:16:3e:6f:ed:ce 10.20.4.1"] >> >>> > port f1fe3ff5-51ec-4790-ada8-c6e244e0cdd6 >> >>> > addresses: ["fa:16:3e:68:cb:45 10.20.3.3"] >> >>> > port 864f7948-f254-4df2-b1cb-39138293a80d >> >>> > addresses: ["fa:16:3e:a7:d1:38 10.20.3.1"] >> >>> > port 5c0e4397-7334-4f03-9ec3-fdc025588462 >> >>> > addresses: ["fa:16:3e:4e:d2:0f 10.20.4.3"] >> >>> > switch 1462d793-7fc5-47a2-8ede-23860a61353e >> >>> > (neutron-3f99e567-26da-44e3-bc41-c90572f45028) >> >>> > port f4948a0f-6794-4140-9e4a-9616308467ee >> >>> > addresses: ["fa:16:3e:eb:85:2c 10.20.3.3"] >> >>> > port f9e86d51-9b26-48f3-b2ad-8df492022a77 >> >>> > addresses: ["fa:16:3e:e8:28:75 10.20.4.3"] >> >>> > port 9504652a-6cd7-45a2-8edf-411e23fe4204 >> >>> > addresses: ["fa:16:3e:de:8b:1d 10.20.4.1"] >> >>> > port f3abc8da-5a4a-4a83-977e-7bffc51003b9 >> >>> > addresses: ["fa:16:3e:96:08:ea 10.20.3.1"] >> >>> > router bc57dc9c-1032-41fd-8768-f0c7536d9558 >> >>> > (neutron-9bf118df-154b-4659-97f4-d0c2ce6394cd) >> >>> > port lrp-864f7948-f254-4df2-b1cb-39138293a80d >> >>> > mac: "fa:16:3e:a7:d1:38" >> >>> > networks: ["10.20.3.1/24"] >> >>> > port lrp-c1c22fbb-5928-41c8-aa1a-c5a746c22118 >> >>> > mac: "fa:16:3e:6f:ed:ce" >> >>> > networks: ["10.20.4.1/24"] >> >>> > router 9cc399f6-60bc-4117-8f0a-941711697c8e >> >>> > (neutron-39c6e2b9-4e0b-4641-9ca0-2a4ccc06741c) >> >>> > router 7636ef03-96ad-470b-b343-49e75211a506 >> >>> > (neutron-df27d70e-88d9-469f-ba83-e2e2034e389a) >> >>> > router 25cb2627-c45a-40af-a49d-f6fc727580b1 >> >>> > (neutron-9b7bd489-d872-4ffc-b4e3-a142f99ab00f) >> >>> > port lrp-9504652a-6cd7-45a2-8edf-411e23fe4204 >> >>> > mac: "fa:16:3e:de:8b:1d" >> >>> > networks: ["10.20.4.1/24"] >> >>> > port lrp-f3abc8da-5a4a-4a83-977e-7bffc51003b9 >> >>> > mac: "fa:16:3e:96:08:ea" >> >>> > networks: ["10.20.3.1/24"] >> >>> > >> >>> > -----Original Message----- >> >>> > From: Ben Pfaff [mailto:b...@ovn.org] >> >>> > Sent: Tuesday, January 31, 2017 8:26 AM >> >>> > To: Ritesh Rekhi <ritesh.re...@nutanix.com> >> >>> > Cc: ovs-discuss@openvswitch.org >> >>> > Subject: Re: [ovs-discuss] after setting ovs-vsctl set open . >> >>> > external-ids:ovn-encap-ip=<IP>, eth0 stops tansmitting any packets >> >>> > in docker container >> >>> > >> >>> > On Mon, Jan 30, 2017 at 02:11:15AM +0000, Ritesh Rekhi wrote: >> >>> > > To test scalability of controller, I made a docker container which >> >>> > > simulates as chassis using the instructions given by Dustin >> >>> > > http://blog.spinhirne.com/2016/09/a-primer-on-ovn.html. >> >>> > > >> >>> > > Everything was working fine, I was able to connect to my controller >> >>> > > and can see the flows coming back to my controller. I was using >> >>> > > released version 2.6.0. >> >>> > > >> >>> > > For some patches wanted to get latest version and found that after I >> >>> > > upgrade to latest patch , my script doesn't work. Upon >> >>> > > troubleshooting found that when I give the last command which >> >>> > > establishes the connection to controller i.e ovs-vsctl set open . >> >>> > > external-ids:ovn-encap-ip=<IP> , eth0 stopped Transmitting any >> >>> > > packets on the container. >> >>> > > >> >>> > > I started poking around and found that commit >> >>> > > "1ea9b847135eddca39ed95c0811edd605f6ca114" introduced this issue. If >> >>> > > I use a commit before that things are working fine. >> >>> > >> >>> > OK... I'll need more information about your setup. To start, can you >> >>> > provide your OVN northbound and southbound databases? >> >_______________________________________________ >> >discuss mailing list >> >disc...@openvswitch.org >> >https://mail.openvswitch.org/mailman/listinfo/ovs-discuss _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
