Re: [ovs-discuss] FW: OVS 2.9.0 native firewall drops empty payload TCP packets continued

[Darrell Ball]

Thanks for reconfirming Jing

Darrell

On Fri, May 3, 2019 at 3:02 PM Zhang, Jing C. (Nokia - CA/Ottawa) <
jing.c.zh...@nokia.com> wrote:

> The thing is, I don’t see empty TCP packet drops on DPDK computes, I
> nevertheless applied the patch HAN mentioned on DPDK computes, no
> difference.
>
>
>
> The issues we see is on OVS computes.
>
>
>
> Jing
>
>
>
> *From:* Darrell Ball 
> *Sent:* Friday, May 03, 2019 3:34 PM
> *To:* Zhang, Jing C. (Nokia - CA/Ottawa) 
> *Cc:* Han Zhou ; ovs-discuss@openvswitch.org
> *Subject:* Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty
> payload TCP packets continued
>
>
>
>
>
>
>
> On Fri, May 3, 2019 at 10:44 AM Zhang, Jing C. (Nokia - CA/Ottawa) <
> jing.c.zh...@nokia.com> wrote:
>
>
>1. The hybrid firewall refers to Linux bridge based firewall. To debug
>the issue, we switch the neutron OVS agent to use native firewall.
>
>
>
> [securitygroup]
>
> #firewall_driver=iptables_hybrid
>
> firewall_driver=openvswitch
>
>
>
> # ovs-ofctl dump-flows br-int | grep ct_state
>
> cookie=0xddb977285e2ba9b6, duration=185.322s, table=71, n_packets=0,
> n_bytes=0, idle_age=185, priority=110,ct_state=+trk
> actions=ct_clear,resubmit(,71)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0,
> n_bytes=0, idle_age=184, priority=75,ct_state=+est-rel-rpl,icmp,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=204,
> n_bytes=16642, idle_age=18, priority=77,ct_state=+est-rel-rpl,udp,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0,
> n_bytes=0, idle_age=184, priority=77,ct_state=+est-rel-rpl,tcp,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0,
> n_bytes=0, idle_age=184, priority=75,ct_state=+new-est,icmp,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=204,
> n_bytes=16642, idle_age=18, priority=77,ct_state=+new-est,udp,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0,
> n_bytes=0, idle_age=184, priority=77,ct_state=+new-est,tcp,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0,
> n_bytes=0, idle_age=184, priority=74,ct_state=+est-rel-rpl,ipv6,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0,
> n_bytes=0, idle_age=184, priority=74,ct_state=+est-rel-rpl,ip,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0,
> n_bytes=0, idle_age=184, priority=74,ct_state=+new-est,ipv6,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0,
> n_bytes=0, idle_age=184, priority=74,ct_state=+new-est,ip,reg5=0x1
> actions=resubmit(,73)
>
>
>
> My understanding is Centos 7 packs the OVS tree, that how conntrack is
> supported before kernel 4.3.
>
>
>
> https://cbs.centos.org/koji/buildinfo?buildID=24381
>
>
>
> I assumed you are using Linux tree OVS kernel module
>
>
>
>
>
>
>
>
>
>1. I back-ported the patch pointed by Han to OVS v2.9.0, it does not
>solve the packet drop on the OVS compute.
>
>
>
> Thanks for confirming
>
> We don't know your full topology, but if you want to send packets
> following a path that goes thru an OVS userspace datapath then
>
> that patch would be applicable.
>
> Did you apply the patch to ALL userspace dataspath instances that could be
> in you packet path ?
>
> What is the path followed in the problem case ?
>
>
>
>
>
>
>1.
>
>
>
> The nature of the issue is same, OVS native firewall drops packets less
> than 60 bytes.
>
>
>
> Pls correct me and advise if the issue on OVS compute is fixable.
>
>
>
> You could check the OVS tree kernel module.
>
>
>
>
>
>
>
> JIng
>
>
>
> *From:* Darrell Ball 
> *Sent:* Friday, May 3, 2019 11:55 AM
> *To:* Zhang, Jing C. (Nokia - CA/Ottawa) 
> *Cc:* Han Zhou ; ovs-discuss@openvswitch.org
> *Subject:* Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty
> payload TCP packets continued
>
>
>
> couple corrections inline
>
>
>
> On Fri, May 3, 2019 at 8:52 AM Darrell Ball  wrote:
>
>
>
>
>
> On Fri, May 3, 2019 at 8:29 AM Zhang, Jing C. (Nokia - CA/Ottawa) <
> jing.c.zh...@nokia.com> wrote:
>
>
>1. This issue is with native OVS firewall where the data flows are
>subject to conntrack rules, there is no issue for hybrid firewall
>
>
>
> 1/ Does 'native OVS firewall' mean either kernel datapath or userpace
> datapath ?
>
> 2/ Pls define 'hybrid datapath' in your context ?
>
>
>
> 2/ Pls define 'hybrid firewall' in your context ?
>
>
>
>
>
>
>1.
>
>
>
>1. Below is from DPDK compute:
>
>
>
> # ovs-vsctl --no-wait get Open_vSwitch . other_config
>
>
>
> 3/ dpdk is not initialized
>
>
>
> An info log is also present when dpdk is initialized: "DPDK Enabled -
> initialized"
>
>
>
> btw: '--no-wait' is needed for 

Re: [ovs-discuss] FW: OVS 2.9.0 native firewall drops empty payload TCP packets continued

[Zhang, Jing C. (Nokia - CA/Ottawa)]

The thing is, I don’t see empty TCP packet drops on DPDK computes, I 
nevertheless applied the patch HAN mentioned on DPDK computes, no difference.

The issues we see is on OVS computes.

Jing

From: Darrell Ball 
Sent: Friday, May 03, 2019 3:34 PM
To: Zhang, Jing C. (Nokia - CA/Ottawa) 
Cc: Han Zhou ; ovs-discuss@openvswitch.org
Subject: Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload 
TCP packets continued



On Fri, May 3, 2019 at 10:44 AM Zhang, Jing C. (Nokia - CA/Ottawa) 
mailto:jing.c.zh...@nokia.com>> wrote:

  1.  The hybrid firewall refers to Linux bridge based firewall. To debug the 
issue, we switch the neutron OVS agent to use native firewall.

[securitygroup]
#firewall_driver=iptables_hybrid
firewall_driver=openvswitch

# ovs-ofctl dump-flows br-int | grep ct_state
cookie=0xddb977285e2ba9b6, duration=185.322s, table=71, n_packets=0, n_bytes=0, 
idle_age=185, priority=110,ct_state=+trk actions=ct_clear,resubmit(,71)
cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, n_bytes=0, 
idle_age=184, priority=75,ct_state=+est-rel-rpl,icmp,reg5=0x1 
actions=resubmit(,73)
cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=204, 
n_bytes=16642, idle_age=18, priority=77,ct_state=+est-rel-rpl,udp,reg5=0x1 
actions=resubmit(,73)
cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, n_bytes=0, 
idle_age=184, priority=77,ct_state=+est-rel-rpl,tcp,reg5=0x1 
actions=resubmit(,73)
cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, n_bytes=0, 
idle_age=184, priority=75,ct_state=+new-est,icmp,reg5=0x1 actions=resubmit(,73)
cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=204, 
n_bytes=16642, idle_age=18, priority=77,ct_state=+new-est,udp,reg5=0x1 
actions=resubmit(,73)
cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, n_bytes=0, 
idle_age=184, priority=77,ct_state=+new-est,tcp,reg5=0x1 actions=resubmit(,73)
cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, n_bytes=0, 
idle_age=184, priority=74,ct_state=+est-rel-rpl,ipv6,reg5=0x1 
actions=resubmit(,73)
cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, n_bytes=0, 
idle_age=184, priority=74,ct_state=+est-rel-rpl,ip,reg5=0x1 
actions=resubmit(,73)
cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, n_bytes=0, 
idle_age=184, priority=74,ct_state=+new-est,ipv6,reg5=0x1 actions=resubmit(,73)
cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, n_bytes=0, 
idle_age=184, priority=74,ct_state=+new-est,ip,reg5=0x1 actions=resubmit(,73)

My understanding is Centos 7 packs the OVS tree, that how conntrack is 
supported before kernel 4.3.

https://cbs.centos.org/koji/buildinfo?buildID=24381

I assumed you are using Linux tree OVS kernel module





  1.  I back-ported the patch pointed by Han to OVS v2.9.0, it does not solve 
the packet drop on the OVS compute.

Thanks for confirming
We don't know your full topology, but if you want to send packets following a 
path that goes thru an OVS userspace datapath then
that patch would be applicable.
Did you apply the patch to ALL userspace dataspath instances that could be in 
you packet path ?
What is the path followed in the problem case ?



  1.

The nature of the issue is same, OVS native firewall drops packets less than 60 
bytes.

Pls correct me and advise if the issue on OVS compute is fixable.

You could check the OVS tree kernel module.



JIng

From: Darrell Ball mailto:dlu...@gmail.com>>
Sent: Friday, May 3, 2019 11:55 AM
To: Zhang, Jing C. (Nokia - CA/Ottawa) 
mailto:jing.c.zh...@nokia.com>>
Cc: Han Zhou mailto:zhou...@gmail.com>>; 
ovs-discuss@openvswitch.org
Subject: Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload 
TCP packets continued

couple corrections inline

On Fri, May 3, 2019 at 8:52 AM Darrell Ball 
mailto:dlu...@gmail.com>> wrote:


On Fri, May 3, 2019 at 8:29 AM Zhang, Jing C. (Nokia - CA/Ottawa) 
mailto:jing.c.zh...@nokia.com>> wrote:

  1.  This issue is with native OVS firewall where the data flows are subject 
to conntrack rules, there is no issue for hybrid firewall

1/ Does 'native OVS firewall' mean either kernel datapath or userpace datapath ?
2/ Pls define 'hybrid datapath' in your context ?

2/ Pls define 'hybrid firewall' in your context ?



  1.


  1.  Below is from DPDK compute:

# ovs-vsctl --no-wait get Open_vSwitch . other_config

3/ dpdk is not initialized

An info log is also present when dpdk is initialized: "DPDK Enabled - 
initialized"

btw: '--no-wait' is needed for get commands

btw: '--no-wait' is NOT needed for get commands



# ovs-vsctl -- list bridge br-int | grep datapath
datapath_id : "1a9b5b9ec94e"
datapath_type   : netdev
datapath_version: ""

4/ You are using userspace datapath on this particular node without dpdk support
Is that intentional ?



From: Darrell Ball mailto:dlu...@gmail.com>>
Sent: Friday, May 

Re: [ovs-discuss] OVS GRE port selecting egress interface for tunnelled traffic

[Gregory Rose]

On 4/29/2019 12:21 AM, Aitor Zabala Orive wrote:


Hi,

We are currently working with OVS in order to be capable of routing 
traffic through different interfaces connecting to the internet. In 
order to maintain private ip addressing we are using gre tunnel OVS 
implementation. However, we are incapable of binding GRE tunnels to 
specific network interfaces, where tunneled traffic should be 
forwarded and finally egress. Once traffic egress the gre Openflow 
port in the OVS, it seems it uses Linux routing table to forward the 
traffic to the corresponding network interface. Under this situation, 
we are incapable of using gre ports in OVS to load balance traffic 
matched in the OVS to different interfaces, as traffic will normally 
use the single default route in Linux routing table.


Can anyone shine some light to this problem?



Questions like these come up often and I always try to remind and/or 
inform people that IP addresses in
Linux belong to the system, not interfaces.  Same goes for the routing 
table.  Using gre ports to do load

balancing is something I've never heard of.

- Greg


Thank you in advance,

**

Aitor


___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss


___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] FW: OVS 2.9.0 native firewall drops empty payload TCP packets continued

[Darrell Ball]

On Fri, May 3, 2019 at 10:44 AM Zhang, Jing C. (Nokia - CA/Ottawa) <
jing.c.zh...@nokia.com> wrote:

>
>1. The hybrid firewall refers to Linux bridge based firewall. To debug
>the issue, we switch the neutron OVS agent to use native firewall.
>
>
>
> [securitygroup]
>
> #firewall_driver=iptables_hybrid
>
> firewall_driver=openvswitch
>
>
>
> # ovs-ofctl dump-flows br-int | grep ct_state
>
> cookie=0xddb977285e2ba9b6, duration=185.322s, table=71, n_packets=0,
> n_bytes=0, idle_age=185, priority=110,ct_state=+trk
> actions=ct_clear,resubmit(,71)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0,
> n_bytes=0, idle_age=184, priority=75,ct_state=+est-rel-rpl,icmp,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=204,
> n_bytes=16642, idle_age=18, priority=77,ct_state=+est-rel-rpl,udp,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0,
> n_bytes=0, idle_age=184, priority=77,ct_state=+est-rel-rpl,tcp,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0,
> n_bytes=0, idle_age=184, priority=75,ct_state=+new-est,icmp,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=204,
> n_bytes=16642, idle_age=18, priority=77,ct_state=+new-est,udp,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0,
> n_bytes=0, idle_age=184, priority=77,ct_state=+new-est,tcp,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0,
> n_bytes=0, idle_age=184, priority=74,ct_state=+est-rel-rpl,ipv6,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0,
> n_bytes=0, idle_age=184, priority=74,ct_state=+est-rel-rpl,ip,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0,
> n_bytes=0, idle_age=184, priority=74,ct_state=+new-est,ipv6,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0,
> n_bytes=0, idle_age=184, priority=74,ct_state=+new-est,ip,reg5=0x1
> actions=resubmit(,73)
>
>
>
> My understanding is Centos 7 packs the OVS tree, that how conntrack is
> supported before kernel 4.3.
>
>
>
> https://cbs.centos.org/koji/buildinfo?buildID=24381
>

I assumed you are using Linux tree OVS kernel module



>
>
>
>
>1. I back-ported the patch pointed by Han to OVS v2.9.0, it does not
>solve the packet drop on the OVS compute.
>
>
Thanks for confirming
We don't know your full topology, but if you want to send packets following
a path that goes thru an OVS userspace datapath then
that patch would be applicable.
Did you apply the patch to ALL userspace dataspath instances that could be
in you packet path ?
What is the path followed in the problem case ?



>
>1.
>
>
>
> The nature of the issue is same, OVS native firewall drops packets less
> than 60 bytes.
>
>
>
> Pls correct me and advise if the issue on OVS compute is fixable.
>

You could check the OVS tree kernel module.



>
>
> JIng
>
>
>
> *From:* Darrell Ball 
> *Sent:* Friday, May 3, 2019 11:55 AM
> *To:* Zhang, Jing C. (Nokia - CA/Ottawa) 
> *Cc:* Han Zhou ; ovs-discuss@openvswitch.org
> *Subject:* Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty
> payload TCP packets continued
>
>
>
> couple corrections inline
>
>
>
> On Fri, May 3, 2019 at 8:52 AM Darrell Ball  wrote:
>
>
>
>
>
> On Fri, May 3, 2019 at 8:29 AM Zhang, Jing C. (Nokia - CA/Ottawa) <
> jing.c.zh...@nokia.com> wrote:
>
>
>1. This issue is with native OVS firewall where the data flows are
>subject to conntrack rules, there is no issue for hybrid firewall
>
>
>
> 1/ Does 'native OVS firewall' mean either kernel datapath or userpace
> datapath ?
>
> 2/ Pls define 'hybrid datapath' in your context ?
>
>
>
> 2/ Pls define 'hybrid firewall' in your context ?
>
>
>
>
>
>
>1.
>
>
>
>1. Below is from DPDK compute:
>
>
>
> # ovs-vsctl --no-wait get Open_vSwitch . other_config
>
>
>
> 3/ dpdk is not initialized
>
>
>
> An info log is also present when dpdk is initialized: "DPDK Enabled -
> initialized"
>
>
>
> btw: '--no-wait' is needed for get commands
>
>
>
> btw: '--no-wait' is NOT needed for get commands
>
>
>
>
>
>
>
> # ovs-vsctl -- list bridge br-int | grep datapath
>
> datapath_id : "1a9b5b9ec94e"
>
> datapath_type   : netdev
>
> datapath_version: ""
>
>
>
> 4/ You are using userspace datapath on this particular node without dpdk
> support
>
> Is that intentional ?
>
>
>
>
>
>
>
> *From:* Darrell Ball 
> *Sent:* Friday, May 3, 2019 11:24 AM
> *To:* Zhang, Jing C. (Nokia - CA/Ottawa) 
> *Cc:* Han Zhou ; ovs-discuss@openvswitch.org
> *Subject:* Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty
> payload TCP packets continued
>
>
>
> The node you are displaying below is running 

Re: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload TCP packets continued

[Gregory Rose]

On 5/2/2019 6:03 PM, Zhang, Jing C. (Nokia - CA/Ottawa) wrote:
We (our VNFs) continue to observe the same empty payload TCP (ACK) 
packet drop with native firewall (see original post below) after 
upgrading to Centos 7.6. This packet drop results in unacceptable TCP 
performance, by that native firewall still can not be enabled in product.
_https://mail.openvswitch.org/pipermail/ovs-discuss/2018-August/047263.html_ 


$ uname -a
Linux overcloud-sriovperformancecompute-0 3.10.0-957.10.1.el7.x86_64 
#1 SMP Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ ovs-vswitchd --version
ovs-vswitchd (Open vSwitch) 2.9.0
DPDK 17.11.0
The scenario: OVS provider VLAN network is used

 1. in physical interface of ovs compute zero length tcp payload
packet arrives as padded to 64 bytes (and vlan tag is included in
ethernet header)
 2. same packet does not appear anymore in the tcpdump taken from
tap-xyz interface (once vlan tag is removed and packet is cut by 4
bytes to 60 bytes)

Tcpdump on physical port:
00:25:24.468423 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 
802.1Q (0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 
0x0, ttl 64, id 6893, offset 0, flags [DF], proto TCP (6), length 2656)
    192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 
(incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
00:25:24.468593 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 
802.1Q (0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, 
ttl 64, id 56318, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.10.60.57576 > 192.168.10.52.80: Flags [.], cksum 0x1d34 
(correct), seq 78, ack 11577, win 391, length 0
00:25:24.475848 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 
802.1Q (0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, 
ttl 64, id 56319, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.10.60.57576 > 192.168.10.52.80: Flags [F.], cksum 0x1d33 
(correct), seq 78, ack 11577, win 391, length 0
00:25:24.480337 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 
802.1Q (0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 
0x0, ttl 64, id 6894, offset 0, flags [DF], proto TCP (6), length 2656)
    192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 
(incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP

Tcpdump on vm tap interface:
00:25:24.468419 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype IPv4 
(0x0800), length 2670: (tos 0x0, ttl 64, id 6893, offset 0, flags 
[DF], proto TCP (6), length 2656)
    192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 
(incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
00:25:24.480331 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype IPv4 
(0x0800), length 2670: (tos 0x0, ttl 64, id 6894, offset 0, flags 
[DF], proto TCP (6), length 2656)
    192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 
(incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP

Very straightforward to see the issue:

 1. Configure neutron OVS agent to use native firewall
 2. Create a pair of VMs on separate computes on provider vLAN
 3. Disable TCP timestamp inside the VMs
 4. Exchange TCP traffic between the VMs, e.g. http download.
 5. Tcpdump on the physical and vm port, and compare.

I wonder why such obvious issue is not widely discussed?


I wonder that myself.  I would indicate to me that there is some rare or 
unique circumstance to  your setup or

configuration.

In any case, we'll just focus on why you're seeing the issue and try to 
determine the root cause.  I'll probably want
to set up a remote debugging situation so I can interactively view and 
troubleshoot the problem.  Unfortunately,
that will probably not be doable until I return to work after May 27 
because the next week and a half before I go

on PTO is pretty booked up for me.

Thanks,

- Greg


Jing

___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss


___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] FW: OVS 2.9.0 native firewall drops empty payload TCP packets continued

[Zhang, Jing C. (Nokia - CA/Ottawa)]

Sorry, I overlooked your questions on dpdk computes. DPDK itself is find, VMs 
are up and running.

Jing

From: Darrell Ball 
Sent: Friday, May 3, 2019 11:55 AM
To: Zhang, Jing C. (Nokia - CA/Ottawa) 
Cc: Han Zhou ; ovs-discuss@openvswitch.org
Subject: Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload 
TCP packets continued

couple corrections inline

On Fri, May 3, 2019 at 8:52 AM Darrell Ball 
mailto:dlu...@gmail.com>> wrote:


On Fri, May 3, 2019 at 8:29 AM Zhang, Jing C. (Nokia - CA/Ottawa) 
mailto:jing.c.zh...@nokia.com>> wrote:

  1.  This issue is with native OVS firewall where the data flows are subject 
to conntrack rules, there is no issue for hybrid firewall

1/ Does 'native OVS firewall' mean either kernel datapath or userpace datapath ?
2/ Pls define 'hybrid datapath' in your context ?

2/ Pls define 'hybrid firewall' in your context ?



  1.


  1.  Below is from DPDK compute:

# ovs-vsctl --no-wait get Open_vSwitch . other_config

3/ dpdk is not initialized

An info log is also present when dpdk is initialized: "DPDK Enabled - 
initialized"

btw: '--no-wait' is needed for get commands

btw: '--no-wait' is NOT needed for get commands



# ovs-vsctl -- list bridge br-int | grep datapath
datapath_id : "1a9b5b9ec94e"
datapath_type   : netdev
datapath_version: ""

4/ You are using userspace datapath on this particular node without dpdk support
Is that intentional ?



From: Darrell Ball mailto:dlu...@gmail.com>>
Sent: Friday, May 3, 2019 11:24 AM
To: Zhang, Jing C. (Nokia - CA/Ottawa) 
mailto:jing.c.zh...@nokia.com>>
Cc: Han Zhou mailto:zhou...@gmail.com>>; 
ovs-discuss@openvswitch.org
Subject: Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload 
TCP packets continued

The node you are displaying below is running kernel datapath

fyi: The fix Han pointed you to is for userspace datapath/conntrack



On Fri, May 3, 2019 at 8:14 AM Zhang, Jing C. (Nokia - CA/Ottawa) 
mailto:jing.c.zh...@nokia.com>> wrote:
We have both OVS and OVS-dpdk computes.

Below is from OVS compute:

# ovs-vsctl --no-wait get Open_vSwitch . other_config
{}
# ovs-vsctl -- list bridge br-int | grep datapath
datapath_id : "aaf62aaf3546"
datapath_type   : system
datapath_version: ""

From: Darrell Ball mailto:dlu...@gmail.com>>
Sent: Friday, May 3, 2019 12:19 AM
To: Han Zhou mailto:zhou...@gmail.com>>; Zhang, Jing C. 
(Nokia - CA/Ottawa) mailto:jing.c.zh...@nokia.com>>; 
ovs-discuss@openvswitch.org
Subject: Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload 
TCP packets continued

What do the following commands yield ?

sudo ovs-vsctl -- get bridge  datapath_type

sudo ovs-vsctl --no-wait get Open_vSwitch . other_config



From: 
mailto:ovs-discuss-boun...@openvswitch.org>>
 on behalf of Han Zhou mailto:zhou...@gmail.com>>
Date: Thursday, May 2, 2019 at 7:12 PM
To: "Zhang, Jing C. (Nokia - CA/Ottawa)" 
mailto:jing.c.zh...@nokia.com>>
Cc: "ovs-discuss@openvswitch.org" 
mailto:ovs-discuss@openvswitch.org>>
Subject: Re: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload TCP 
packets continued



On Thu, May 2, 2019 at 6:04 PM Zhang, Jing C. (Nokia - CA/Ottawa) 
mailto:jing.c.zh...@nokia.com>> wrote:
>
> We (our VNFs) continue to observe the same empty payload TCP (ACK) packet 
> drop with native firewall (see original post below) after upgrading to Centos 
> 7.6. This packet drop results in unacceptable TCP performance, by that native 
> firewall still can not be enabled in product.
>
> https://mail.openvswitch.org/pipermail/ovs-discuss/2018-August/047263.html
>
> $ uname -a
> Linux overcloud-sriovperformancecompute-0 3.10.0-957.10.1.el7.x86_64 #1 SMP 
> Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
>
> $ ovs-vswitchd --version
> ovs-vswitchd (Open vSwitch) 2.9.0
> DPDK 17.11.0
>
> The scenario: OVS provider VLAN network is used
>
>
> in physical interface of ovs compute zero length tcp payload packet arrives 
> as padded to 64 bytes (and vlan tag is included in ethernet header)
> same packet does not appear anymore in the tcpdump taken from tap-xyz 
> interface (once vlan tag is removed and packet is cut by 4 bytes to 60 bytes)
>
>
> Tcpdump on physical port:
>
> 00:25:24.468423 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 802.1Q 
> (0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 
> 6893, offset 0, flags [DF], proto TCP (6), length 2656)
> 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 
> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 

Re: [ovs-discuss] FW: OVS 2.9.0 native firewall drops empty payload TCP packets continued

[Zhang, Jing C. (Nokia - CA/Ottawa)]

  1.  The hybrid firewall refers to Linux bridge based firewall. To debug the 
issue, we switch the neutron OVS agent to use native firewall.

[securitygroup]
#firewall_driver=iptables_hybrid
firewall_driver=openvswitch

# ovs-ofctl dump-flows br-int | grep ct_state
cookie=0xddb977285e2ba9b6, duration=185.322s, table=71, n_packets=0, n_bytes=0, 
idle_age=185, priority=110,ct_state=+trk actions=ct_clear,resubmit(,71)
cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, n_bytes=0, 
idle_age=184, priority=75,ct_state=+est-rel-rpl,icmp,reg5=0x1 
actions=resubmit(,73)
cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=204, 
n_bytes=16642, idle_age=18, priority=77,ct_state=+est-rel-rpl,udp,reg5=0x1 
actions=resubmit(,73)
cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, n_bytes=0, 
idle_age=184, priority=77,ct_state=+est-rel-rpl,tcp,reg5=0x1 
actions=resubmit(,73)
cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, n_bytes=0, 
idle_age=184, priority=75,ct_state=+new-est,icmp,reg5=0x1 actions=resubmit(,73)
cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=204, 
n_bytes=16642, idle_age=18, priority=77,ct_state=+new-est,udp,reg5=0x1 
actions=resubmit(,73)
cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, n_bytes=0, 
idle_age=184, priority=77,ct_state=+new-est,tcp,reg5=0x1 actions=resubmit(,73)
cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, n_bytes=0, 
idle_age=184, priority=74,ct_state=+est-rel-rpl,ipv6,reg5=0x1 
actions=resubmit(,73)
cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, n_bytes=0, 
idle_age=184, priority=74,ct_state=+est-rel-rpl,ip,reg5=0x1 
actions=resubmit(,73)
cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, n_bytes=0, 
idle_age=184, priority=74,ct_state=+new-est,ipv6,reg5=0x1 actions=resubmit(,73)
cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, n_bytes=0, 
idle_age=184, priority=74,ct_state=+new-est,ip,reg5=0x1 actions=resubmit(,73)

My understanding is Centos 7 packs the OVS tree, that how conntrack is 
supported before kernel 4.3.

https://cbs.centos.org/koji/buildinfo?buildID=24381



  1.  I back-ported the patch pointed by Han to OVS v2.9.0, it does not solve 
the packet drop on the OVS compute.

The nature of the issue is same, OVS native firewall drops packets less than 60 
bytes.

Pls correct me and advise if the issue on OVS compute is fixable.

JIng

From: Darrell Ball 
Sent: Friday, May 3, 2019 11:55 AM
To: Zhang, Jing C. (Nokia - CA/Ottawa) 
Cc: Han Zhou ; ovs-discuss@openvswitch.org
Subject: Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload 
TCP packets continued

couple corrections inline

On Fri, May 3, 2019 at 8:52 AM Darrell Ball 
mailto:dlu...@gmail.com>> wrote:


On Fri, May 3, 2019 at 8:29 AM Zhang, Jing C. (Nokia - CA/Ottawa) 
mailto:jing.c.zh...@nokia.com>> wrote:

  1.  This issue is with native OVS firewall where the data flows are subject 
to conntrack rules, there is no issue for hybrid firewall

1/ Does 'native OVS firewall' mean either kernel datapath or userpace datapath ?
2/ Pls define 'hybrid datapath' in your context ?

2/ Pls define 'hybrid firewall' in your context ?



  1.


  1.  Below is from DPDK compute:

# ovs-vsctl --no-wait get Open_vSwitch . other_config

3/ dpdk is not initialized

An info log is also present when dpdk is initialized: "DPDK Enabled - 
initialized"

btw: '--no-wait' is needed for get commands

btw: '--no-wait' is NOT needed for get commands



# ovs-vsctl -- list bridge br-int | grep datapath
datapath_id : "1a9b5b9ec94e"
datapath_type   : netdev
datapath_version: ""

4/ You are using userspace datapath on this particular node without dpdk support
Is that intentional ?



From: Darrell Ball mailto:dlu...@gmail.com>>
Sent: Friday, May 3, 2019 11:24 AM
To: Zhang, Jing C. (Nokia - CA/Ottawa) 
mailto:jing.c.zh...@nokia.com>>
Cc: Han Zhou mailto:zhou...@gmail.com>>; 
ovs-discuss@openvswitch.org
Subject: Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload 
TCP packets continued

The node you are displaying below is running kernel datapath

fyi: The fix Han pointed you to is for userspace datapath/conntrack



On Fri, May 3, 2019 at 8:14 AM Zhang, Jing C. (Nokia - CA/Ottawa) 
mailto:jing.c.zh...@nokia.com>> wrote:
We have both OVS and OVS-dpdk computes.

Below is from OVS compute:

# ovs-vsctl --no-wait get Open_vSwitch . other_config
{}
# ovs-vsctl -- list bridge br-int | grep datapath
datapath_id : "aaf62aaf3546"
datapath_type   : system
datapath_version: ""

From: Darrell Ball mailto:dlu...@gmail.com>>
Sent: Friday, May 3, 2019 12:19 AM
To: Han Zhou mailto:zhou...@gmail.com>>; Zhang, Jing C. 
(Nokia - CA/Ottawa) mailto:jing.c.zh...@nokia.com>>; 
ovs-discuss@openvswitch.org
Subject: Re: FW: [ovs-discuss] OVS 2.9.0 

Re: [ovs-discuss] FW: OVS 2.9.0 native firewall drops empty payload TCP packets continued

[Darrell Ball]

couple corrections inline

On Fri, May 3, 2019 at 8:52 AM Darrell Ball  wrote:

>
>
> On Fri, May 3, 2019 at 8:29 AM Zhang, Jing C. (Nokia - CA/Ottawa) <
> jing.c.zh...@nokia.com> wrote:
>
>>
>>1. This issue is with native OVS firewall where the data flows are
>>subject to conntrack rules, there is no issue for hybrid firewall
>>
>>
> 1/ Does 'native OVS firewall' mean either kernel datapath or userpace
> datapath ?
> 2/ Pls define 'hybrid datapath' in your context ?
>

2/ Pls define 'hybrid firewall' in your context ?


>
>
>>
>>1.
>>
>>
>>
>>1. Below is from DPDK compute:
>>
>>
>>
>> # ovs-vsctl --no-wait get Open_vSwitch . other_config
>>
>
> 3/ dpdk is not initialized
>
> An info log is also present when dpdk is initialized: "DPDK Enabled -
> initialized"
>
> btw: '--no-wait' is needed for get commands
>

btw: '--no-wait' is NOT needed for get commands


>
>
>
>> # ovs-vsctl -- list bridge br-int | grep datapath
>>
>> datapath_id : "1a9b5b9ec94e"
>>
>> datapath_type   : netdev
>>
>> datapath_version: ""
>>
>
> 4/ You are using userspace datapath on this particular node without dpdk
> support
> Is that intentional ?
>
>
>
>>
>>
>> *From:* Darrell Ball 
>> *Sent:* Friday, May 3, 2019 11:24 AM
>> *To:* Zhang, Jing C. (Nokia - CA/Ottawa) 
>> *Cc:* Han Zhou ; ovs-discuss@openvswitch.org
>> *Subject:* Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty
>> payload TCP packets continued
>>
>>
>>
>> The node you are displaying below is running kernel datapath
>>
>>
>>
>> fyi: The fix Han pointed you to is for userspace datapath/conntrack
>>
>>
>>
>>
>>
>>
>>
>> On Fri, May 3, 2019 at 8:14 AM Zhang, Jing C. (Nokia - CA/Ottawa) <
>> jing.c.zh...@nokia.com> wrote:
>>
>> We have both OVS and OVS-dpdk computes.
>>
>>
>>
>> Below is from OVS compute:
>>
>>
>>
>> # ovs-vsctl --no-wait get Open_vSwitch . other_config
>>
>> {}
>>
>> # ovs-vsctl -- list bridge br-int | grep datapath
>>
>> datapath_id : "aaf62aaf3546"
>>
>> datapath_type   : system
>>
>> datapath_version: ""
>>
>>
>>
>> *From:* Darrell Ball 
>> *Sent:* Friday, May 3, 2019 12:19 AM
>> *To:* Han Zhou ; Zhang, Jing C. (Nokia - CA/Ottawa) <
>> jing.c.zh...@nokia.com>; ovs-discuss@openvswitch.org
>> *Subject:* Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty
>> payload TCP packets continued
>>
>>
>>
>> What do the following commands yield ?
>>
>>
>>
>> sudo ovs-vsctl -- get bridge  datapath_type
>>
>>
>>
>> sudo ovs-vsctl --no-wait get Open_vSwitch . other_config
>>
>>
>>
>>
>>
>>
>>
>> *From: * on behalf of Han Zhou <
>> zhou...@gmail.com>
>> *Date: *Thursday, May 2, 2019 at 7:12 PM
>> *To: *"Zhang, Jing C. (Nokia - CA/Ottawa)" 
>> *Cc: *"ovs-discuss@openvswitch.org" 
>> *Subject: *Re: [ovs-discuss] OVS 2.9.0 native firewall drops empty
>> payload TCP packets continued
>>
>>
>>
>>
>>
>> On Thu, May 2, 2019 at 6:04 PM Zhang, Jing C. (Nokia - CA/Ottawa) <
>> jing.c.zh...@nokia.com> wrote:
>> >
>> > We (our VNFs) continue to observe the same empty payload TCP (ACK)
>> packet drop with native firewall (see original post below) after upgrading
>> to Centos 7.6. This packet drop results in unacceptable TCP performance, by
>> that native firewall still can not be enabled in product.
>> >
>> >
>> https://mail.openvswitch.org/pipermail/ovs-discuss/2018-August/047263.html
>> 
>> >
>> > $ uname -a
>> > Linux overcloud-sriovperformancecompute-0 3.10.0-957.10.1.el7.x86_64 #1
>> SMP Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
>> >
>> > $ ovs-vswitchd --version
>> > ovs-vswitchd (Open vSwitch) 2.9.0
>> > DPDK 17.11.0
>> >
>> > The scenario: OVS provider VLAN network is used
>> >
>> >
>> > in physical interface of ovs compute zero length tcp payload packet
>> arrives as padded to 64 bytes (and vlan tag is included in ethernet header)
>> > same packet does not appear anymore in the tcpdump taken from tap-xyz
>> interface (once vlan tag is removed and packet is cut by 4 bytes to 60
>> bytes)
>> >
>> >
>> > Tcpdump on physical port:
>> >
>> > 00:25:24.468423 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 802.1Q
>> (0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id
>> 6893, offset 0, flags [DF], proto TCP (6), length 2656)
>> > 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013
>> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
>> > 00:25:24.468593 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 802.1Q
>> (0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id
>> 56318, offset 0, flags [DF], proto TCP (6), length 40)
>> > 192.168.10.60.57576 > 192.168.10.52.80: Flags [.], cksum 

Re: [ovs-discuss] FW: OVS 2.9.0 native firewall drops empty payload TCP packets continued

[Darrell Ball]

On Fri, May 3, 2019 at 8:29 AM Zhang, Jing C. (Nokia - CA/Ottawa) <
jing.c.zh...@nokia.com> wrote:

>
>1. This issue is with native OVS firewall where the data flows are
>subject to conntrack rules, there is no issue for hybrid firewall
>
>
1/ Does 'native OVS firewall' mean either kernel datapath or userpace
datapath ?
2/ Pls define 'hybrid datapath' in your context ?


>
>1.
>
>
>
>1. Below is from DPDK compute:
>
>
>
> # ovs-vsctl --no-wait get Open_vSwitch . other_config
>

3/ dpdk is not initialized

An info log is also present when dpdk is initialized: "DPDK Enabled -
initialized"

btw: '--no-wait' is needed for get commands



> # ovs-vsctl -- list bridge br-int | grep datapath
>
> datapath_id : "1a9b5b9ec94e"
>
> datapath_type   : netdev
>
> datapath_version: ""
>

4/ You are using userspace datapath on this particular node without dpdk
support
Is that intentional ?



>
>
> *From:* Darrell Ball 
> *Sent:* Friday, May 3, 2019 11:24 AM
> *To:* Zhang, Jing C. (Nokia - CA/Ottawa) 
> *Cc:* Han Zhou ; ovs-discuss@openvswitch.org
> *Subject:* Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty
> payload TCP packets continued
>
>
>
> The node you are displaying below is running kernel datapath
>
>
>
> fyi: The fix Han pointed you to is for userspace datapath/conntrack
>
>
>
>
>
>
>
> On Fri, May 3, 2019 at 8:14 AM Zhang, Jing C. (Nokia - CA/Ottawa) <
> jing.c.zh...@nokia.com> wrote:
>
> We have both OVS and OVS-dpdk computes.
>
>
>
> Below is from OVS compute:
>
>
>
> # ovs-vsctl --no-wait get Open_vSwitch . other_config
>
> {}
>
> # ovs-vsctl -- list bridge br-int | grep datapath
>
> datapath_id : "aaf62aaf3546"
>
> datapath_type   : system
>
> datapath_version: ""
>
>
>
> *From:* Darrell Ball 
> *Sent:* Friday, May 3, 2019 12:19 AM
> *To:* Han Zhou ; Zhang, Jing C. (Nokia - CA/Ottawa) <
> jing.c.zh...@nokia.com>; ovs-discuss@openvswitch.org
> *Subject:* Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty
> payload TCP packets continued
>
>
>
> What do the following commands yield ?
>
>
>
> sudo ovs-vsctl -- get bridge  datapath_type
>
>
>
> sudo ovs-vsctl --no-wait get Open_vSwitch . other_config
>
>
>
>
>
>
>
> *From: * on behalf of Han Zhou <
> zhou...@gmail.com>
> *Date: *Thursday, May 2, 2019 at 7:12 PM
> *To: *"Zhang, Jing C. (Nokia - CA/Ottawa)" 
> *Cc: *"ovs-discuss@openvswitch.org" 
> *Subject: *Re: [ovs-discuss] OVS 2.9.0 native firewall drops empty
> payload TCP packets continued
>
>
>
>
>
> On Thu, May 2, 2019 at 6:04 PM Zhang, Jing C. (Nokia - CA/Ottawa) <
> jing.c.zh...@nokia.com> wrote:
> >
> > We (our VNFs) continue to observe the same empty payload TCP (ACK)
> packet drop with native firewall (see original post below) after upgrading
> to Centos 7.6. This packet drop results in unacceptable TCP performance, by
> that native firewall still can not be enabled in product.
> >
> >
> https://mail.openvswitch.org/pipermail/ovs-discuss/2018-August/047263.html
> 
> >
> > $ uname -a
> > Linux overcloud-sriovperformancecompute-0 3.10.0-957.10.1.el7.x86_64 #1
> SMP Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
> >
> > $ ovs-vswitchd --version
> > ovs-vswitchd (Open vSwitch) 2.9.0
> > DPDK 17.11.0
> >
> > The scenario: OVS provider VLAN network is used
> >
> >
> > in physical interface of ovs compute zero length tcp payload packet
> arrives as padded to 64 bytes (and vlan tag is included in ethernet header)
> > same packet does not appear anymore in the tcpdump taken from tap-xyz
> interface (once vlan tag is removed and packet is cut by 4 bytes to 60
> bytes)
> >
> >
> > Tcpdump on physical port:
> >
> > 00:25:24.468423 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 802.1Q
> (0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id
> 6893, offset 0, flags [DF], proto TCP (6), length 2656)
> > 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013
> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
> > 00:25:24.468593 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 802.1Q
> (0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id
> 56318, offset 0, flags [DF], proto TCP (6), length 40)
> > 192.168.10.60.57576 > 192.168.10.52.80: Flags [.], cksum 0x1d34
> (correct), seq 78, ack 11577, win 391, length 0
> > 00:25:24.475848 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 802.1Q
> (0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id
> 56319, offset 0, flags [DF], proto TCP (6), length 40)
> > 192.168.10.60.57576 > 192.168.10.52.80: Flags [F.], cksum 0x1d33
> (correct), seq 78, ack 11577, win 

Re: [ovs-discuss] FW: OVS 2.9.0 native firewall drops empty payload TCP packets continued

[Zhang, Jing C. (Nokia - CA/Ottawa)]

  1.  This issue is with native OVS firewall where the data flows are subject 
to conntrack rules, there is no issue for hybrid firewall


  1.  Below is from DPDK compute:

# ovs-vsctl --no-wait get Open_vSwitch . other_config
# ovs-vsctl -- list bridge br-int | grep datapath
datapath_id : "1a9b5b9ec94e"
datapath_type   : netdev
datapath_version: ""

From: Darrell Ball 
Sent: Friday, May 3, 2019 11:24 AM
To: Zhang, Jing C. (Nokia - CA/Ottawa) 
Cc: Han Zhou ; ovs-discuss@openvswitch.org
Subject: Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload 
TCP packets continued

The node you are displaying below is running kernel datapath

fyi: The fix Han pointed you to is for userspace datapath/conntrack



On Fri, May 3, 2019 at 8:14 AM Zhang, Jing C. (Nokia - CA/Ottawa) 
mailto:jing.c.zh...@nokia.com>> wrote:
We have both OVS and OVS-dpdk computes.

Below is from OVS compute:

# ovs-vsctl --no-wait get Open_vSwitch . other_config
{}
# ovs-vsctl -- list bridge br-int | grep datapath
datapath_id : "aaf62aaf3546"
datapath_type   : system
datapath_version: ""

From: Darrell Ball mailto:dlu...@gmail.com>>
Sent: Friday, May 3, 2019 12:19 AM
To: Han Zhou mailto:zhou...@gmail.com>>; Zhang, Jing C. 
(Nokia - CA/Ottawa) mailto:jing.c.zh...@nokia.com>>; 
ovs-discuss@openvswitch.org
Subject: Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload 
TCP packets continued

What do the following commands yield ?

sudo ovs-vsctl -- get bridge  datapath_type

sudo ovs-vsctl --no-wait get Open_vSwitch . other_config



From: 
mailto:ovs-discuss-boun...@openvswitch.org>>
 on behalf of Han Zhou mailto:zhou...@gmail.com>>
Date: Thursday, May 2, 2019 at 7:12 PM
To: "Zhang, Jing C. (Nokia - CA/Ottawa)" 
mailto:jing.c.zh...@nokia.com>>
Cc: "ovs-discuss@openvswitch.org" 
mailto:ovs-discuss@openvswitch.org>>
Subject: Re: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload TCP 
packets continued



On Thu, May 2, 2019 at 6:04 PM Zhang, Jing C. (Nokia - CA/Ottawa) 
mailto:jing.c.zh...@nokia.com>> wrote:
>
> We (our VNFs) continue to observe the same empty payload TCP (ACK) packet 
> drop with native firewall (see original post below) after upgrading to Centos 
> 7.6. This packet drop results in unacceptable TCP performance, by that native 
> firewall still can not be enabled in product.
>
> https://mail.openvswitch.org/pipermail/ovs-discuss/2018-August/047263.html
>
> $ uname -a
> Linux overcloud-sriovperformancecompute-0 3.10.0-957.10.1.el7.x86_64 #1 SMP 
> Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
>
> $ ovs-vswitchd --version
> ovs-vswitchd (Open vSwitch) 2.9.0
> DPDK 17.11.0
>
> The scenario: OVS provider VLAN network is used
>
>
> in physical interface of ovs compute zero length tcp payload packet arrives 
> as padded to 64 bytes (and vlan tag is included in ethernet header)
> same packet does not appear anymore in the tcpdump taken from tap-xyz 
> interface (once vlan tag is removed and packet is cut by 4 bytes to 60 bytes)
>
>
> Tcpdump on physical port:
>
> 00:25:24.468423 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 802.1Q 
> (0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 
> 6893, offset 0, flags [DF], proto TCP (6), length 2656)
> 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 
> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
> 00:25:24.468593 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 802.1Q 
> (0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 
> 56318, offset 0, flags [DF], proto TCP (6), length 40)
> 192.168.10.60.57576 > 192.168.10.52.80: Flags [.], cksum 0x1d34 
> (correct), seq 78, ack 11577, win 391, length 0
> 00:25:24.475848 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 802.1Q 
> (0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 
> 56319, offset 0, flags [DF], proto TCP (6), length 40)
> 192.168.10.60.57576 > 192.168.10.52.80: Flags [F.], cksum 0x1d33 
> (correct), seq 78, ack 11577, win 391, length 0
> 00:25:24.480337 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 802.1Q 
> (0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 
> 6894, offset 0, flags [DF], proto TCP (6), length 2656)
> 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 
> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
>
> Tcpdump on vm tap interface:
>
> 00:25:24.468419 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype IPv4 
> (0x0800), length 2670: (tos 0x0, ttl 

Re: [ovs-discuss] FW: OVS 2.9.0 native firewall drops empty payload TCP packets continued

[Darrell Ball]

and jtbc, ovs-dpdk uses the userspace datapath

On Fri, May 3, 2019 at 8:24 AM Darrell Ball  wrote:

> The node you are displaying below is running kernel datapath
>
> fyi: The fix Han pointed you to is for userspace datapath/conntrack
>
>
>
> On Fri, May 3, 2019 at 8:14 AM Zhang, Jing C. (Nokia - CA/Ottawa) <
> jing.c.zh...@nokia.com> wrote:
>
>> We have both OVS and OVS-dpdk computes.
>>
>>
>>
>> Below is from OVS compute:
>>
>>
>>
>> # ovs-vsctl --no-wait get Open_vSwitch . other_config
>>
>> {}
>>
>> # ovs-vsctl -- list bridge br-int | grep datapath
>>
>> datapath_id : "aaf62aaf3546"
>>
>> datapath_type   : system
>>
>> datapath_version: ""
>>
>>
>>
>> *From:* Darrell Ball 
>> *Sent:* Friday, May 3, 2019 12:19 AM
>> *To:* Han Zhou ; Zhang, Jing C. (Nokia - CA/Ottawa) <
>> jing.c.zh...@nokia.com>; ovs-discuss@openvswitch.org
>> *Subject:* Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty
>> payload TCP packets continued
>>
>>
>>
>> What do the following commands yield ?
>>
>>
>>
>> sudo ovs-vsctl -- get bridge  datapath_type
>>
>>
>>
>> sudo ovs-vsctl --no-wait get Open_vSwitch . other_config
>>
>>
>>
>>
>>
>>
>>
>> *From: * on behalf of Han Zhou <
>> zhou...@gmail.com>
>> *Date: *Thursday, May 2, 2019 at 7:12 PM
>> *To: *"Zhang, Jing C. (Nokia - CA/Ottawa)" 
>> *Cc: *"ovs-discuss@openvswitch.org" 
>> *Subject: *Re: [ovs-discuss] OVS 2.9.0 native firewall drops empty
>> payload TCP packets continued
>>
>>
>>
>>
>>
>> On Thu, May 2, 2019 at 6:04 PM Zhang, Jing C. (Nokia - CA/Ottawa) <
>> jing.c.zh...@nokia.com> wrote:
>> >
>> > We (our VNFs) continue to observe the same empty payload TCP (ACK)
>> packet drop with native firewall (see original post below) after upgrading
>> to Centos 7.6. This packet drop results in unacceptable TCP performance, by
>> that native firewall still can not be enabled in product.
>> >
>> >
>> https://mail.openvswitch.org/pipermail/ovs-discuss/2018-August/047263.html
>> 
>> >
>> > $ uname -a
>> > Linux overcloud-sriovperformancecompute-0 3.10.0-957.10.1.el7.x86_64 #1
>> SMP Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
>> >
>> > $ ovs-vswitchd --version
>> > ovs-vswitchd (Open vSwitch) 2.9.0
>> > DPDK 17.11.0
>> >
>> > The scenario: OVS provider VLAN network is used
>> >
>> >
>> > in physical interface of ovs compute zero length tcp payload packet
>> arrives as padded to 64 bytes (and vlan tag is included in ethernet header)
>> > same packet does not appear anymore in the tcpdump taken from tap-xyz
>> interface (once vlan tag is removed and packet is cut by 4 bytes to 60
>> bytes)
>> >
>> >
>> > Tcpdump on physical port:
>> >
>> > 00:25:24.468423 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 802.1Q
>> (0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id
>> 6893, offset 0, flags [DF], proto TCP (6), length 2656)
>> > 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013
>> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
>> > 00:25:24.468593 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 802.1Q
>> (0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id
>> 56318, offset 0, flags [DF], proto TCP (6), length 40)
>> > 192.168.10.60.57576 > 192.168.10.52.80: Flags [.], cksum 0x1d34
>> (correct), seq 78, ack 11577, win 391, length 0
>> > 00:25:24.475848 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 802.1Q
>> (0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id
>> 56319, offset 0, flags [DF], proto TCP (6), length 40)
>> > 192.168.10.60.57576 > 192.168.10.52.80: Flags [F.], cksum 0x1d33
>> (correct), seq 78, ack 11577, win 391, length 0
>> > 00:25:24.480337 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 802.1Q
>> (0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id
>> 6894, offset 0, flags [DF], proto TCP (6), length 2656)
>> > 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013
>> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
>> >
>> > Tcpdump on vm tap interface:
>> >
>> > 00:25:24.468419 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype IPv4
>> (0x0800), length 2670: (tos 0x0, ttl 64, id 6893, offset 0, flags [DF],
>> proto TCP (6), length 2656)
>> > 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013
>> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
>> > 00:25:24.480331 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype IPv4
>> (0x0800), length 2670: (tos 0x0, ttl 64, id 6894, offset 0, flags [DF],
>> proto TCP (6), length 2656)
>> > 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 

Re: [ovs-discuss] FW: OVS 2.9.0 native firewall drops empty payload TCP packets continued

[Darrell Ball]

The node you are displaying below is running kernel datapath

fyi: The fix Han pointed you to is for userspace datapath/conntrack



On Fri, May 3, 2019 at 8:14 AM Zhang, Jing C. (Nokia - CA/Ottawa) <
jing.c.zh...@nokia.com> wrote:

> We have both OVS and OVS-dpdk computes.
>
>
>
> Below is from OVS compute:
>
>
>
> # ovs-vsctl --no-wait get Open_vSwitch . other_config
>
> {}
>
> # ovs-vsctl -- list bridge br-int | grep datapath
>
> datapath_id : "aaf62aaf3546"
>
> datapath_type   : system
>
> datapath_version: ""
>
>
>
> *From:* Darrell Ball 
> *Sent:* Friday, May 3, 2019 12:19 AM
> *To:* Han Zhou ; Zhang, Jing C. (Nokia - CA/Ottawa) <
> jing.c.zh...@nokia.com>; ovs-discuss@openvswitch.org
> *Subject:* Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty
> payload TCP packets continued
>
>
>
> What do the following commands yield ?
>
>
>
> sudo ovs-vsctl -- get bridge  datapath_type
>
>
>
> sudo ovs-vsctl --no-wait get Open_vSwitch . other_config
>
>
>
>
>
>
>
> *From: * on behalf of Han Zhou <
> zhou...@gmail.com>
> *Date: *Thursday, May 2, 2019 at 7:12 PM
> *To: *"Zhang, Jing C. (Nokia - CA/Ottawa)" 
> *Cc: *"ovs-discuss@openvswitch.org" 
> *Subject: *Re: [ovs-discuss] OVS 2.9.0 native firewall drops empty
> payload TCP packets continued
>
>
>
>
>
> On Thu, May 2, 2019 at 6:04 PM Zhang, Jing C. (Nokia - CA/Ottawa) <
> jing.c.zh...@nokia.com> wrote:
> >
> > We (our VNFs) continue to observe the same empty payload TCP (ACK)
> packet drop with native firewall (see original post below) after upgrading
> to Centos 7.6. This packet drop results in unacceptable TCP performance, by
> that native firewall still can not be enabled in product.
> >
> >
> https://mail.openvswitch.org/pipermail/ovs-discuss/2018-August/047263.html
> 
> >
> > $ uname -a
> > Linux overcloud-sriovperformancecompute-0 3.10.0-957.10.1.el7.x86_64 #1
> SMP Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
> >
> > $ ovs-vswitchd --version
> > ovs-vswitchd (Open vSwitch) 2.9.0
> > DPDK 17.11.0
> >
> > The scenario: OVS provider VLAN network is used
> >
> >
> > in physical interface of ovs compute zero length tcp payload packet
> arrives as padded to 64 bytes (and vlan tag is included in ethernet header)
> > same packet does not appear anymore in the tcpdump taken from tap-xyz
> interface (once vlan tag is removed and packet is cut by 4 bytes to 60
> bytes)
> >
> >
> > Tcpdump on physical port:
> >
> > 00:25:24.468423 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 802.1Q
> (0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id
> 6893, offset 0, flags [DF], proto TCP (6), length 2656)
> > 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013
> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
> > 00:25:24.468593 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 802.1Q
> (0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id
> 56318, offset 0, flags [DF], proto TCP (6), length 40)
> > 192.168.10.60.57576 > 192.168.10.52.80: Flags [.], cksum 0x1d34
> (correct), seq 78, ack 11577, win 391, length 0
> > 00:25:24.475848 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 802.1Q
> (0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id
> 56319, offset 0, flags [DF], proto TCP (6), length 40)
> > 192.168.10.60.57576 > 192.168.10.52.80: Flags [F.], cksum 0x1d33
> (correct), seq 78, ack 11577, win 391, length 0
> > 00:25:24.480337 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 802.1Q
> (0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id
> 6894, offset 0, flags [DF], proto TCP (6), length 2656)
> > 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013
> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
> >
> > Tcpdump on vm tap interface:
> >
> > 00:25:24.468419 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype IPv4
> (0x0800), length 2670: (tos 0x0, ttl 64, id 6893, offset 0, flags [DF],
> proto TCP (6), length 2656)
> > 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013
> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
> > 00:25:24.480331 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype IPv4
> (0x0800), length 2670: (tos 0x0, ttl 64, id 6894, offset 0, flags [DF],
> proto TCP (6), length 2656)
> > 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013
> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
> >
> > Very straightforward to see the issue:
> >
> >
> > Configure neutron OVS agent to use native firewall
> > Create a pair of VMs on separate computes 

Re: [ovs-discuss] FW: OVS 2.9.0 native firewall drops empty payload TCP packets continued

[Zhang, Jing C. (Nokia - CA/Ottawa)]

We have both OVS and OVS-dpdk computes.

Below is from OVS compute:

# ovs-vsctl --no-wait get Open_vSwitch . other_config
{}
# ovs-vsctl -- list bridge br-int | grep datapath
datapath_id : "aaf62aaf3546"
datapath_type   : system
datapath_version: ""

From: Darrell Ball 
Sent: Friday, May 3, 2019 12:19 AM
To: Han Zhou ; Zhang, Jing C. (Nokia - CA/Ottawa) 
; ovs-discuss@openvswitch.org
Subject: Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload 
TCP packets continued

What do the following commands yield ?

sudo ovs-vsctl -- get bridge  datapath_type

sudo ovs-vsctl --no-wait get Open_vSwitch . other_config



From: 
mailto:ovs-discuss-boun...@openvswitch.org>>
 on behalf of Han Zhou mailto:zhou...@gmail.com>>
Date: Thursday, May 2, 2019 at 7:12 PM
To: "Zhang, Jing C. (Nokia - CA/Ottawa)" 
mailto:jing.c.zh...@nokia.com>>
Cc: "ovs-discuss@openvswitch.org" 
mailto:ovs-discuss@openvswitch.org>>
Subject: Re: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload TCP 
packets continued



On Thu, May 2, 2019 at 6:04 PM Zhang, Jing C. (Nokia - CA/Ottawa) 
mailto:jing.c.zh...@nokia.com>> wrote:
>
> We (our VNFs) continue to observe the same empty payload TCP (ACK) packet 
> drop with native firewall (see original post below) after upgrading to Centos 
> 7.6. This packet drop results in unacceptable TCP performance, by that native 
> firewall still can not be enabled in product.
>
> https://mail.openvswitch.org/pipermail/ovs-discuss/2018-August/047263.html
>
> $ uname -a
> Linux overcloud-sriovperformancecompute-0 3.10.0-957.10.1.el7.x86_64 #1 SMP 
> Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
>
> $ ovs-vswitchd --version
> ovs-vswitchd (Open vSwitch) 2.9.0
> DPDK 17.11.0
>
> The scenario: OVS provider VLAN network is used
>
>
> in physical interface of ovs compute zero length tcp payload packet arrives 
> as padded to 64 bytes (and vlan tag is included in ethernet header)
> same packet does not appear anymore in the tcpdump taken from tap-xyz 
> interface (once vlan tag is removed and packet is cut by 4 bytes to 60 bytes)
>
>
> Tcpdump on physical port:
>
> 00:25:24.468423 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 802.1Q 
> (0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 
> 6893, offset 0, flags [DF], proto TCP (6), length 2656)
> 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 
> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
> 00:25:24.468593 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 802.1Q 
> (0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 
> 56318, offset 0, flags [DF], proto TCP (6), length 40)
> 192.168.10.60.57576 > 192.168.10.52.80: Flags [.], cksum 0x1d34 
> (correct), seq 78, ack 11577, win 391, length 0
> 00:25:24.475848 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 802.1Q 
> (0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 
> 56319, offset 0, flags [DF], proto TCP (6), length 40)
> 192.168.10.60.57576 > 192.168.10.52.80: Flags [F.], cksum 0x1d33 
> (correct), seq 78, ack 11577, win 391, length 0
> 00:25:24.480337 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 802.1Q 
> (0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 
> 6894, offset 0, flags [DF], proto TCP (6), length 2656)
> 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 
> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
>
> Tcpdump on vm tap interface:
>
> 00:25:24.468419 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype IPv4 
> (0x0800), length 2670: (tos 0x0, ttl 64, id 6893, offset 0, flags [DF], proto 
> TCP (6), length 2656)
> 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 
> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
> 00:25:24.480331 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype IPv4 
> (0x0800), length 2670: (tos 0x0, ttl 64, id 6894, offset 0, flags [DF], proto 
> TCP (6), length 2656)
> 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 
> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
>
> Very straightforward to see the issue:
>
>
> Configure neutron OVS agent to use native firewall
> Create a pair of VMs on separate computes on provider vLAN
> Disable TCP timestamp inside the VMs
> Exchange TCP traffic between the VMs, e.g. http download.
> Tcpdump on the physical and vm port, and compare.
>
>
> I wonder why such obvious issue is not widely discussed?
>
> Jing
>

Maybe it's fixed by:

Re: [ovs-discuss] TSO in OVS-DPDK 2.11.0

[Harsh Gondaliya]

Basically, I want to run a VM2VM (same host) iperf test and thus want to
enable TSO for the vhostuser backend. But, I am not able to figure out how
can I turn ON this feature in guest VMs. ethtool is unable to change these
fixed features. Either some change in domain XML or some change in OVS-DPDK
config need to be done. Any suggestions?

On Fri, May 3, 2019 at 1:12 PM Harsh Gondaliya <
harshgondaliya_vinodb...@srmuniv.edu.in> wrote:

> Does OVS 2.11.0 support TSO in OVS-DPDK? Or are there any separate
> configurations to be done or patches to be applied?
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload TCP packets continued

[Zhang, Jing C. (Nokia - CA/Ottawa)]

Thank you Han, I will check out this fix.

From: Han Zhou 
Sent: Thursday, May 2, 2019 10:11 PM
To: Zhang, Jing C. (Nokia - CA/Ottawa) 
Cc: ovs-discuss@openvswitch.org
Subject: Re: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload TCP 
packets continued



On Thu, May 2, 2019 at 6:04 PM Zhang, Jing C. (Nokia - CA/Ottawa) 
mailto:jing.c.zh...@nokia.com>> wrote:
>
> We (our VNFs) continue to observe the same empty payload TCP (ACK) packet 
> drop with native firewall (see original post below) after upgrading to Centos 
> 7.6. This packet drop results in unacceptable TCP performance, by that native 
> firewall still can not be enabled in product.
>
> https://mail.openvswitch.org/pipermail/ovs-discuss/2018-August/047263.html
>
> $ uname -a
> Linux overcloud-sriovperformancecompute-0 3.10.0-957.10.1.el7.x86_64 #1 SMP 
> Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
>
> $ ovs-vswitchd --version
> ovs-vswitchd (Open vSwitch) 2.9.0
> DPDK 17.11.0
>
> The scenario: OVS provider VLAN network is used
>
>
> in physical interface of ovs compute zero length tcp payload packet arrives 
> as padded to 64 bytes (and vlan tag is included in ethernet header)
> same packet does not appear anymore in the tcpdump taken from tap-xyz 
> interface (once vlan tag is removed and packet is cut by 4 bytes to 60 bytes)
>
>
> Tcpdump on physical port:
>
> 00:25:24.468423 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 802.1Q 
> (0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 
> 6893, offset 0, flags [DF], proto TCP (6), length 2656)
> 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 
> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
> 00:25:24.468593 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 802.1Q 
> (0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 
> 56318, offset 0, flags [DF], proto TCP (6), length 40)
> 192.168.10.60.57576 > 192.168.10.52.80: Flags [.], cksum 0x1d34 
> (correct), seq 78, ack 11577, win 391, length 0
> 00:25:24.475848 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 802.1Q 
> (0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 
> 56319, offset 0, flags [DF], proto TCP (6), length 40)
> 192.168.10.60.57576 > 192.168.10.52.80: Flags [F.], cksum 0x1d33 
> (correct), seq 78, ack 11577, win 391, length 0
> 00:25:24.480337 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 802.1Q 
> (0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 
> 6894, offset 0, flags [DF], proto TCP (6), length 2656)
> 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 
> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
>
> Tcpdump on vm tap interface:
>
> 00:25:24.468419 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype IPv4 
> (0x0800), length 2670: (tos 0x0, ttl 64, id 6893, offset 0, flags [DF], proto 
> TCP (6), length 2656)
> 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 
> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
> 00:25:24.480331 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype IPv4 
> (0x0800), length 2670: (tos 0x0, ttl 64, id 6894, offset 0, flags [DF], proto 
> TCP (6), length 2656)
> 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 
> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
>
> Very straightforward to see the issue:
>
>
> Configure neutron OVS agent to use native firewall
> Create a pair of VMs on separate computes on provider vLAN
> Disable TCP timestamp inside the VMs
> Exchange TCP traffic between the VMs, e.g. http download.
> Tcpdump on the physical and vm port, and compare.
>
>
> I wonder why such obvious issue is not widely discussed?
>
> Jing
>

Maybe it's fixed by:
https://github.com/openvswitch/ovs/commit/9171c63532ee9cbc63bb8cfae364ab071f44389b

___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

[ovs-discuss] TSO in OVS-DPDK 2.11.0

[Harsh Gondaliya]

Does OVS 2.11.0 support TSO in OVS-DPDK? Or are there any separate
configurations to be done or patches to be applied?
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss