Thanks for reconfirming Jing Darrell
On Fri, May 3, 2019 at 3:02 PM Zhang, Jing C. (Nokia - CA/Ottawa) < jing.c.zh...@nokia.com> wrote: > The thing is, I don’t see empty TCP packet drops on DPDK computes, I > nevertheless applied the patch HAN mentioned on DPDK computes, no > difference. > > > > The issues we see is on OVS computes. > > > > Jing > > > > *From:* Darrell Ball <dlu...@gmail.com> > *Sent:* Friday, May 03, 2019 3:34 PM > *To:* Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zh...@nokia.com> > *Cc:* Han Zhou <zhou...@gmail.com>; ovs-discuss@openvswitch.org > *Subject:* Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty > payload TCP packets continued > > > > > > > > On Fri, May 3, 2019 at 10:44 AM Zhang, Jing C. (Nokia - CA/Ottawa) < > jing.c.zh...@nokia.com> wrote: > > > 1. The hybrid firewall refers to Linux bridge based firewall. To debug > the issue, we switch the neutron OVS agent to use native firewall. > > > > [securitygroup] > > #firewall_driver=iptables_hybrid > > firewall_driver=openvswitch > > > > # ovs-ofctl dump-flows br-int | grep ct_state > > cookie=0xddb977285e2ba9b6, duration=185.322s, table=71, n_packets=0, > n_bytes=0, idle_age=185, priority=110,ct_state=+trk > actions=ct_clear,resubmit(,71) > > cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, > n_bytes=0, idle_age=184, priority=75,ct_state=+est-rel-rpl,icmp,reg5=0x1 > actions=resubmit(,73) > > cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=204, > n_bytes=16642, idle_age=18, priority=77,ct_state=+est-rel-rpl,udp,reg5=0x1 > actions=resubmit(,73) > > cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, > n_bytes=0, idle_age=184, priority=77,ct_state=+est-rel-rpl,tcp,reg5=0x1 > actions=resubmit(,73) > > cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, > n_bytes=0, idle_age=184, priority=75,ct_state=+new-est,icmp,reg5=0x1 > actions=resubmit(,73) > > cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=204, > n_bytes=16642, idle_age=18, priority=77,ct_state=+new-est,udp,reg5=0x1 > actions=resubmit(,73) > > cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, > n_bytes=0, idle_age=184, priority=77,ct_state=+new-est,tcp,reg5=0x1 > actions=resubmit(,73) > > cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, > n_bytes=0, idle_age=184, priority=74,ct_state=+est-rel-rpl,ipv6,reg5=0x1 > actions=resubmit(,73) > > cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, > n_bytes=0, idle_age=184, priority=74,ct_state=+est-rel-rpl,ip,reg5=0x1 > actions=resubmit(,73) > > cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, > n_bytes=0, idle_age=184, priority=74,ct_state=+new-est,ipv6,reg5=0x1 > actions=resubmit(,73) > > cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, > n_bytes=0, idle_age=184, priority=74,ct_state=+new-est,ip,reg5=0x1 > actions=resubmit(,73) > > > > My understanding is Centos 7 packs the OVS tree, that how conntrack is > supported before kernel 4.3. > > > > https://cbs.centos.org/koji/buildinfo?buildID=24381 > > > > I assumed you are using Linux tree OVS kernel module > > > > > > > > > > 1. I back-ported the patch pointed by Han to OVS v2.9.0, it does not > solve the packet drop on the OVS compute. > > > > Thanks for confirming > > We don't know your full topology, but if you want to send packets > following a path that goes thru an OVS userspace datapath then > > that patch would be applicable. > > Did you apply the patch to ALL userspace dataspath instances that could be > in you packet path ? > > What is the path followed in the problem case ? > > > > > > > 1. > > > > The nature of the issue is same, OVS native firewall drops packets less > than 60 bytes. > > > > Pls correct me and advise if the issue on OVS compute is fixable. > > > > You could check the OVS tree kernel module. > > > > > > > > JIng > > > > *From:* Darrell Ball <dlu...@gmail.com> > *Sent:* Friday, May 3, 2019 11:55 AM > *To:* Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zh...@nokia.com> > *Cc:* Han Zhou <zhou...@gmail.com>; ovs-discuss@openvswitch.org > *Subject:* Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty > payload TCP packets continued > > > > couple corrections inline > > > > On Fri, May 3, 2019 at 8:52 AM Darrell Ball <dlu...@gmail.com> wrote: > > > > > > On Fri, May 3, 2019 at 8:29 AM Zhang, Jing C. (Nokia - CA/Ottawa) < > jing.c.zh...@nokia.com> wrote: > > > 1. This issue is with native OVS firewall where the data flows are > subject to conntrack rules, there is no issue for hybrid firewall > > > > 1/ Does 'native OVS firewall' mean either kernel datapath or userpace > datapath ? > > 2/ Pls define 'hybrid datapath' in your context ? > > > > 2/ Pls define 'hybrid firewall' in your context ? > > > > > > > 1. > > > > 1. Below is from DPDK compute: > > > > # ovs-vsctl --no-wait get Open_vSwitch . other_config > > > > 3/ dpdk is not initialized > > > > An info log is also present when dpdk is initialized: "DPDK Enabled - > initialized" > > > > btw: '--no-wait' is needed for get commands > > > > btw: '--no-wait' is NOT needed for get commands > > > > > > > > # ovs-vsctl -- list bridge br-int | grep datapath > > datapath_id : "00001a9b5b9ec94e" > > datapath_type : netdev > > datapath_version : "<built-in>" > > > > 4/ You are using userspace datapath on this particular node without dpdk > support > > Is that intentional ? > > > > > > > > *From:* Darrell Ball <dlu...@gmail.com> > *Sent:* Friday, May 3, 2019 11:24 AM > *To:* Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zh...@nokia.com> > *Cc:* Han Zhou <zhou...@gmail.com>; ovs-discuss@openvswitch.org > *Subject:* Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty > payload TCP packets continued > > > > The node you are displaying below is running kernel datapath > > > > fyi: The fix Han pointed you to is for userspace datapath/conntrack > > > > > > > > On Fri, May 3, 2019 at 8:14 AM Zhang, Jing C. (Nokia - CA/Ottawa) < > jing.c.zh...@nokia.com> wrote: > > We have both OVS and OVS-dpdk computes. > > > > Below is from OVS compute: > > > > # ovs-vsctl --no-wait get Open_vSwitch . other_config > > {} > > # ovs-vsctl -- list bridge br-int | grep datapath > > datapath_id : "0000aaf62aaf3546" > > datapath_type : system > > datapath_version : "<unknown>" > > > > *From:* Darrell Ball <dlu...@gmail.com> > *Sent:* Friday, May 3, 2019 12:19 AM > *To:* Han Zhou <zhou...@gmail.com>; Zhang, Jing C. (Nokia - CA/Ottawa) < > jing.c.zh...@nokia.com>; ovs-discuss@openvswitch.org > *Subject:* Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty > payload TCP packets continued > > > > What do the following commands yield ? > > > > sudo ovs-vsctl -- get bridge <bridge name> datapath_type > > > > sudo ovs-vsctl --no-wait get Open_vSwitch . other_config > > > > > > > > *From: *<ovs-discuss-boun...@openvswitch.org> on behalf of Han Zhou < > zhou...@gmail.com> > *Date: *Thursday, May 2, 2019 at 7:12 PM > *To: *"Zhang, Jing C. (Nokia - CA/Ottawa)" <jing.c.zh...@nokia.com> > *Cc: *"ovs-discuss@openvswitch.org" <ovs-discuss@openvswitch.org> > *Subject: *Re: [ovs-discuss] OVS 2.9.0 native firewall drops empty > payload TCP packets continued > > > > > > On Thu, May 2, 2019 at 6:04 PM Zhang, Jing C. (Nokia - CA/Ottawa) < > jing.c.zh...@nokia.com> wrote: > > > > We (our VNFs) continue to observe the same empty payload TCP (ACK) > packet drop with native firewall (see original post below) after upgrading > to Centos 7.6. This packet drop results in unacceptable TCP performance, by > that native firewall still can not be enabled in product. > > > > > https://mail.openvswitch.org/pipermail/ovs-discuss/2018-August/047263.html > <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.openvswitch.org%2Fpipermail%2Fovs-discuss%2F2018-August%2F047263.html&data=02%7C01%7Cdball%40vmware.com%7Cd358d5a23b2640ebd28708d6cf6cc524%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C636924463524642583&sdata=ihTE%2BcOA9d8yNflCbqJYHXOJWhFuqvq4yJmu7H9lGwo%3D&reserved=0> > > > > $ uname -a > > Linux overcloud-sriovperformancecompute-0 3.10.0-957.10.1.el7.x86_64 #1 > SMP Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux > > > > $ ovs-vswitchd --version > > ovs-vswitchd (Open vSwitch) 2.9.0 > > DPDK 17.11.0 > > > > The scenario: OVS provider VLAN network is used > > > > > > in physical interface of ovs compute zero length tcp payload packet > arrives as padded to 64 bytes (and vlan tag is included in ethernet header) > > same packet does not appear anymore in the tcpdump taken from tap-xyz > interface (once vlan tag is removed and packet is cut by 4 bytes to 60 > bytes) > > > > > > Tcpdump on physical port: > > > > 00:25:24.468423 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 802.1Q > (0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id > 6893, offset 0, flags [DF], proto TCP (6), length 2656) > > 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 > (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP > > 00:25:24.468593 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 802.1Q > (0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id > 56318, offset 0, flags [DF], proto TCP (6), length 40) > > 192.168.10.60.57576 > 192.168.10.52.80: Flags [.], cksum 0x1d34 > (correct), seq 78, ack 11577, win 391, length 0 > > 00:25:24.475848 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 802.1Q > (0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id > 56319, offset 0, flags [DF], proto TCP (6), length 40) > > 192.168.10.60.57576 > 192.168.10.52.80: Flags [F.], cksum 0x1d33 > (correct), seq 78, ack 11577, win 391, length 0 > > 00:25:24.480337 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 802.1Q > (0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id > 6894, offset 0, flags [DF], proto TCP (6), length 2656) > > 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 > (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP > > > > Tcpdump on vm tap interface: > > > > 00:25:24.468419 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype IPv4 > (0x0800), length 2670: (tos 0x0, ttl 64, id 6893, offset 0, flags [DF], > proto TCP (6), length 2656) > > 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 > (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP > > 00:25:24.480331 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype IPv4 > (0x0800), length 2670: (tos 0x0, ttl 64, id 6894, offset 0, flags [DF], > proto TCP (6), length 2656) > > 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 > (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP > > > > Very straightforward to see the issue: > > > > > > Configure neutron OVS agent to use native firewall > > Create a pair of VMs on separate computes on provider vLAN > > Disable TCP timestamp inside the VMs > > Exchange TCP traffic between the VMs, e.g. http download. > > Tcpdump on the physical and vm port, and compare. > > > > > > I wonder why such obvious issue is not widely discussed? > > > > Jing > > > > > > Maybe it's fixed by: > > > https://github.com/openvswitch/ovs/commit/9171c63532ee9cbc63bb8cfae364ab071f44389b > <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenvswitch%2Fovs%2Fcommit%2F9171c63532ee9cbc63bb8cfae364ab071f44389b&data=02%7C01%7Cdball%40vmware.com%7Cd358d5a23b2640ebd28708d6cf6cc524%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C636924463524652585&sdata=OLr263fKPdgKA5nga2UlGNF0WtB6srXSIg9a7aHkf44%3D&reserved=0> > > > >
_______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss