Re: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload TCP packets continued

2019-05-03 Thread Gregory Rose 


On 5/2/2019 6:03 PM, Zhang, Jing C. (Nokia - CA/Ottawa) wrote:
We (our VNFs) continue to observe the same empty payload TCP (ACK) 
packet drop with native firewall (see original post below) after 
upgrading to Centos 7.6. This packet drop results in unacceptable TCP 
performance, by that native firewall still can not be enabled in product.
_https://mail.openvswitch.org/pipermail/ovs-discuss/2018-August/047263.html_ 


$ uname -a
Linux overcloud-sriovperformancecompute-0 3.10.0-957.10.1.el7.x86_64 
#1 SMP Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ ovs-vswitchd --version
ovs-vswitchd (Open vSwitch) 2.9.0
DPDK 17.11.0
The scenario: OVS provider VLAN network is used

 1. in physical interface of ovs compute zero length tcp payload
packet arrives as padded to 64 bytes (and vlan tag is included in
ethernet header)
 2. same packet does not appear anymore in the tcpdump taken from
tap-xyz interface (once vlan tag is removed and packet is cut by 4
bytes to 60 bytes)

Tcpdump on physical port:
00:25:24.468423 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 
802.1Q (0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 
0x0, ttl 64, id 6893, offset 0, flags [DF], proto TCP (6), length 2656)
    192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 
(incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
00:25:24.468593 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 
802.1Q (0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, 
ttl 64, id 56318, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.10.60.57576 > 192.168.10.52.80: Flags [.], cksum 0x1d34 
(correct), seq 78, ack 11577, win 391, length 0
00:25:24.475848 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 
802.1Q (0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, 
ttl 64, id 56319, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.10.60.57576 > 192.168.10.52.80: Flags [F.], cksum 0x1d33 
(correct), seq 78, ack 11577, win 391, length 0
00:25:24.480337 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 
802.1Q (0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 
0x0, ttl 64, id 6894, offset 0, flags [DF], proto TCP (6), length 2656)
    192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 
(incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP

Tcpdump on vm tap interface:
00:25:24.468419 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype IPv4 
(0x0800), length 2670: (tos 0x0, ttl 64, id 6893, offset 0, flags 
[DF], proto TCP (6), length 2656)
    192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 
(incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
00:25:24.480331 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype IPv4 
(0x0800), length 2670: (tos 0x0, ttl 64, id 6894, offset 0, flags 
[DF], proto TCP (6), length 2656)
    192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 
(incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP

Very straightforward to see the issue:

 1. Configure neutron OVS agent to use native firewall
 2. Create a pair of VMs on separate computes on provider vLAN
 3. Disable TCP timestamp inside the VMs
 4. Exchange TCP traffic between the VMs, e.g. http download.
 5. Tcpdump on the physical and vm port, and compare.

I wonder why such obvious issue is not widely discussed?


I wonder that myself.  I would indicate to me that there is some rare or 
unique circumstance to  your setup or

configuration.

In any case, we'll just focus on why you're seeing the issue and try to 
determine the root cause.  I'll probably want
to set up a remote debugging situation so I can interactively view and 
troubleshoot the problem.  Unfortunately,
that will probably not be doable until I return to work after May 27 
because the next week and a half before I go

on PTO is pretty booked up for me.

Thanks,

- Greg


Jing

___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss


___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload TCP packets continued

2019-05-03 Thread Zhang, Jing C. (Nokia - CA/Ottawa) 
Thank you Han, I will check out this fix.

From: Han Zhou 
Sent: Thursday, May 2, 2019 10:11 PM
To: Zhang, Jing C. (Nokia - CA/Ottawa) 
Cc: ovs-discuss@openvswitch.org
Subject: Re: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload TCP 
packets continued



On Thu, May 2, 2019 at 6:04 PM Zhang, Jing C. (Nokia - CA/Ottawa) 
mailto:jing.c.zh...@nokia.com>> wrote:
>
> We (our VNFs) continue to observe the same empty payload TCP (ACK) packet 
> drop with native firewall (see original post below) after upgrading to Centos 
> 7.6. This packet drop results in unacceptable TCP performance, by that native 
> firewall still can not be enabled in product.
>
> https://mail.openvswitch.org/pipermail/ovs-discuss/2018-August/047263.html
>
> $ uname -a
> Linux overcloud-sriovperformancecompute-0 3.10.0-957.10.1.el7.x86_64 #1 SMP 
> Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
>
> $ ovs-vswitchd --version
> ovs-vswitchd (Open vSwitch) 2.9.0
> DPDK 17.11.0
>
> The scenario: OVS provider VLAN network is used
>
>
> in physical interface of ovs compute zero length tcp payload packet arrives 
> as padded to 64 bytes (and vlan tag is included in ethernet header)
> same packet does not appear anymore in the tcpdump taken from tap-xyz 
> interface (once vlan tag is removed and packet is cut by 4 bytes to 60 bytes)
>
>
> Tcpdump on physical port:
>
> 00:25:24.468423 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 802.1Q 
> (0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 
> 6893, offset 0, flags [DF], proto TCP (6), length 2656)
> 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 
> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
> 00:25:24.468593 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 802.1Q 
> (0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 
> 56318, offset 0, flags [DF], proto TCP (6), length 40)
> 192.168.10.60.57576 > 192.168.10.52.80: Flags [.], cksum 0x1d34 
> (correct), seq 78, ack 11577, win 391, length 0
> 00:25:24.475848 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 802.1Q 
> (0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 
> 56319, offset 0, flags [DF], proto TCP (6), length 40)
> 192.168.10.60.57576 > 192.168.10.52.80: Flags [F.], cksum 0x1d33 
> (correct), seq 78, ack 11577, win 391, length 0
> 00:25:24.480337 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 802.1Q 
> (0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 
> 6894, offset 0, flags [DF], proto TCP (6), length 2656)
> 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 
> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
>
> Tcpdump on vm tap interface:
>
> 00:25:24.468419 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype IPv4 
> (0x0800), length 2670: (tos 0x0, ttl 64, id 6893, offset 0, flags [DF], proto 
> TCP (6), length 2656)
> 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 
> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
> 00:25:24.480331 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype IPv4 
> (0x0800), length 2670: (tos 0x0, ttl 64, id 6894, offset 0, flags [DF], proto 
> TCP (6), length 2656)
> 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 
> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
>
> Very straightforward to see the issue:
>
>
> Configure neutron OVS agent to use native firewall
> Create a pair of VMs on separate computes on provider vLAN
> Disable TCP timestamp inside the VMs
> Exchange TCP traffic between the VMs, e.g. http download.
> Tcpdump on the physical and vm port, and compare.
>
>
> I wonder why such obvious issue is not widely discussed?
>
> Jing
>

Maybe it's fixed by:
https://github.com/openvswitch/ovs/commit/9171c63532ee9cbc63bb8cfae364ab071f44389b

___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload TCP packets continued

2019-05-02 Thread Han Zhou 
On Thu, May 2, 2019 at 6:04 PM Zhang, Jing C. (Nokia - CA/Ottawa) <
jing.c.zh...@nokia.com> wrote:
>
> We (our VNFs) continue to observe the same empty payload TCP (ACK) packet
drop with native firewall (see original post below) after upgrading to
Centos 7.6. This packet drop results in unacceptable TCP performance, by
that native firewall still can not be enabled in product.
>
> https://mail.openvswitch.org/pipermail/ovs-discuss/2018-August/047263.html
>
> $ uname -a
> Linux overcloud-sriovperformancecompute-0 3.10.0-957.10.1.el7.x86_64 #1
SMP Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
>
> $ ovs-vswitchd --version
> ovs-vswitchd (Open vSwitch) 2.9.0
> DPDK 17.11.0
>
> The scenario: OVS provider VLAN network is used
>
>
> in physical interface of ovs compute zero length tcp payload packet
arrives as padded to 64 bytes (and vlan tag is included in ethernet header)
> same packet does not appear anymore in the tcpdump taken from tap-xyz
interface (once vlan tag is removed and packet is cut by 4 bytes to 60
bytes)
>
>
> Tcpdump on physical port:
>
> 00:25:24.468423 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 802.1Q
(0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id
6893, offset 0, flags [DF], proto TCP (6), length 2656)
> 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013
(incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
> 00:25:24.468593 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 802.1Q
(0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id
56318, offset 0, flags [DF], proto TCP (6), length 40)
> 192.168.10.60.57576 > 192.168.10.52.80: Flags [.], cksum 0x1d34
(correct), seq 78, ack 11577, win 391, length 0
> 00:25:24.475848 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 802.1Q
(0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id
56319, offset 0, flags [DF], proto TCP (6), length 40)
> 192.168.10.60.57576 > 192.168.10.52.80: Flags [F.], cksum 0x1d33
(correct), seq 78, ack 11577, win 391, length 0
> 00:25:24.480337 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 802.1Q
(0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id
6894, offset 0, flags [DF], proto TCP (6), length 2656)
> 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013
(incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
>
> Tcpdump on vm tap interface:
>
> 00:25:24.468419 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype IPv4
(0x0800), length 2670: (tos 0x0, ttl 64, id 6893, offset 0, flags [DF],
proto TCP (6), length 2656)
> 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013
(incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
> 00:25:24.480331 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype IPv4
(0x0800), length 2670: (tos 0x0, ttl 64, id 6894, offset 0, flags [DF],
proto TCP (6), length 2656)
> 192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013
(incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
>
> Very straightforward to see the issue:
>
>
> Configure neutron OVS agent to use native firewall
> Create a pair of VMs on separate computes on provider vLAN
> Disable TCP timestamp inside the VMs
> Exchange TCP traffic between the VMs, e.g. http download.
> Tcpdump on the physical and vm port, and compare.
>
>
> I wonder why such obvious issue is not widely discussed?
>
> Jing
>

Maybe it's fixed by:
https://github.com/openvswitch/ovs/commit/9171c63532ee9cbc63bb8cfae364ab071f44389b
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

[ovs-discuss] OVS 2.9.0 native firewall drops empty payload TCP packets continued

2019-05-02 Thread Zhang, Jing C. (Nokia - CA/Ottawa) 
We (our VNFs) continue to observe the same empty payload TCP (ACK) packet drop 
with native firewall (see original post below) after upgrading to Centos 7.6. 
This packet drop results in unacceptable TCP performance, by that native 
firewall still can not be enabled in product.

https://mail.openvswitch.org/pipermail/ovs-discuss/2018-August/047263.html

$ uname -a
Linux overcloud-sriovperformancecompute-0 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon 
Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ ovs-vswitchd --version
ovs-vswitchd (Open vSwitch) 2.9.0
DPDK 17.11.0

The scenario: OVS provider VLAN network is used

1)  in physical interface of ovs compute zero length tcp payload packet 
arrives as padded to 64 bytes (and vlan tag is included in ethernet header)
2)  same packet does not appear anymore in the tcpdump taken from tap-xyz 
interface (once vlan tag is removed and packet is cut by 4 bytes to 60 bytes)

Tcpdump on physical port:

00:25:24.468423 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 802.1Q 
(0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 
6893, offset 0, flags [DF], proto TCP (6), length 2656)
192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 (incorrect 
-> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
00:25:24.468593 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 802.1Q 
(0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 
56318, offset 0, flags [DF], proto TCP (6), length 40)
192.168.10.60.57576 > 192.168.10.52.80: Flags [.], cksum 0x1d34 (correct), 
seq 78, ack 11577, win 391, length 0
00:25:24.475848 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 802.1Q 
(0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 
56319, offset 0, flags [DF], proto TCP (6), length 40)
192.168.10.60.57576 > 192.168.10.52.80: Flags [F.], cksum 0x1d33 (correct), 
seq 78, ack 11577, win 391, length 0
00:25:24.480337 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 802.1Q 
(0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 
6894, offset 0, flags [DF], proto TCP (6), length 2656)
192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 (incorrect 
-> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP

Tcpdump on vm tap interface:

00:25:24.468419 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype IPv4 (0x0800), 
length 2670: (tos 0x0, ttl 64, id 6893, offset 0, flags [DF], proto TCP (6), 
length 2656)
192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 (incorrect 
-> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
00:25:24.480331 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype IPv4 (0x0800), 
length 2670: (tos 0x0, ttl 64, id 6894, offset 0, flags [DF], proto TCP (6), 
length 2656)
192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013 (incorrect 
-> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP

Very straightforward to see the issue:

(1) Configure neutron OVS agent to use native firewall
(2) Create a pair of VMs on separate computes on provider vLAN
(3) Disable TCP timestamp inside the VMs
(4) Exchange TCP traffic between the VMs, e.g. http download.
(5) Tcpdump on the physical and vm port, and compare.

I wonder why such obvious issue is not widely discussed?

Jing



___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload TCP packets

2018-08-27 Thread Zhang, Jing C. (Nokia - CA/Ottawa) 
It is a fresh install, not an upgrade.

uname -a
Linux overcloud-ovscompute-1 3.10.0-862.9.1.el7.x86_64 #1 SMP Mon Jul 16 
16:29:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux


We have wanted to switch to native firewall since 2.8 but can not do that 
because of multiple reports of TCP packet drop with the native firewall, 
observed by VNF either in the form of control plane message retransmission or 
in the form of TCP throughput degradataion .

But this is the first time we have nailed down a specific packet drop scenario 
that we can easily reproduce in our own lab, this time the customer is using 
our latest lineup which is ovs 2.9.

Jing

From: Gregory Rose 
Sent: Monday, August 27, 2018 4:08 PM
To: Zhang, Jing C. (Nokia - CA/Ottawa) ; 
b...@openvswitch.org
Subject: Re: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload TCP 
packets


On 8/27/2018 6:20 AM, Zhang, Jing C. (Nokia - CA/Ottawa) wrote:
We have customers reporting sluggish HTTP download with OVS 2.9.0. After 
debugging, we find the issue is OVS 2.9.0 native firewall drops TCP control 
packets (TCP ACK) with empty payload. The issue can be avoided by either 
reverting back to the legacy Linux bridge firewall or enabling TCP timestamp at 
both HTTP server and client sides.
$ ovs-vswitchd --version
ovs-vswitchd (Open vSwitch) 2.9.0
DPDK 17.11.0
openvswitch-2.9.0-3.el7.x86_64

While I'm setting up to reproduce this could you let me know if this is 
something that just started happening
on a system without any changes to the underlying system configuration or 
operating system?  Or was there
a recent upgrade of the kernel or OVS?

What is the output of uname -a?

Thanks,

- Greg
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload TCP packets

2018-08-27 Thread Gregory Rose 


On 8/27/2018 6:20 AM, Zhang, Jing C. (Nokia - CA/Ottawa) wrote:
We have customers reporting sluggish HTTP download with OVS 2.9.0. 
After debugging, we find the issue is OVS 2.9.0 native firewall drops 
TCP control packets (TCP ACK) with empty payload. The issue can be 
avoided by either reverting back to the legacy Linux bridge firewall 
or enabling TCP timestamp at both HTTP server and client sides.

$ ovs-vswitchd --version
ovs-vswitchd (Open vSwitch) 2.9.0
DPDK 17.11.0
openvswitch-2.9.0-3.el7.x86_64


While I'm setting up to reproduce this could you let me know if this is 
something that just started happening
on a system without any changes to the underlying system configuration 
or operating system?  Or was there

a recent upgrade of the kernel or OVS?

What is the output of uname -a?

Thanks,

- Greg

___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload TCP packets

2018-08-27 Thread Darrell Ball 
This seems to be in the kernel datapath and maybe recent issue
Greg, do you want to take a look ?


On 8/27/18, 9:50 AM, "ovs-discuss-boun...@openvswitch.org on behalf of Darrell 
Ball"  wrote:

I can take a look.

On 8/27/18, 9:47 AM, "ovs-discuss-boun...@openvswitch.org on behalf of Ben 
Pfaff"  wrote:

On Mon, Aug 27, 2018 at 01:20:15PM +, Zhang, Jing C. (Nokia - 
CA/Ottawa) wrote:
> We have customers reporting sluggish HTTP download with OVS 2.9.0. 
After debugging, we find the issue is OVS 2.9.0 native firewall drops TCP 
control packets (TCP ACK) with empty payload. The issue can be avoided by 
either reverting back to the legacy Linux bridge firewall or enabling TCP 
timestamp at both HTTP server and client sides.
> $ ovs-vswitchd --version
> ovs-vswitchd (Open vSwitch) 2.9.0
> DPDK 17.11.0

Darrell, this seems like it's in your area.  Will you take a look?

Thanks,

Ben.
___
discuss mailing list
disc...@openvswitch.org

https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.openvswitch.org%2Fmailman%2Flistinfo%2Fovs-discussdata=02%7C01%7Cdball%40vmware.com%7C2f9b0b93dbd241c6c41008d60c3d2c68%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C1%7C0%7C636709854215126566sdata=LPjqpxatlO4lvu5cA4ejqF6PcBWoPbmdjvQuAwLcnH4%3Dreserved=0


___
discuss mailing list
disc...@openvswitch.org

https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.openvswitch.org%2Fmailman%2Flistinfo%2Fovs-discussdata=02%7C01%7Cdball%40vmware.com%7C2f9b0b93dbd241c6c41008d60c3d2c68%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C1%7C0%7C636709854215126566sdata=LPjqpxatlO4lvu5cA4ejqF6PcBWoPbmdjvQuAwLcnH4%3Dreserved=0


___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload TCP packets

2018-08-27 Thread Darrell Ball 
I can take a look.

On 8/27/18, 9:47 AM, "ovs-discuss-boun...@openvswitch.org on behalf of Ben 
Pfaff"  wrote:

On Mon, Aug 27, 2018 at 01:20:15PM +, Zhang, Jing C. (Nokia - 
CA/Ottawa) wrote:
> We have customers reporting sluggish HTTP download with OVS 2.9.0. After 
debugging, we find the issue is OVS 2.9.0 native firewall drops TCP control 
packets (TCP ACK) with empty payload. The issue can be avoided by either 
reverting back to the legacy Linux bridge firewall or enabling TCP timestamp at 
both HTTP server and client sides.
> $ ovs-vswitchd --version
> ovs-vswitchd (Open vSwitch) 2.9.0
> DPDK 17.11.0

Darrell, this seems like it's in your area.  Will you take a look?

Thanks,

Ben.
___
discuss mailing list
disc...@openvswitch.org

https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.openvswitch.org%2Fmailman%2Flistinfo%2Fovs-discussdata=02%7C01%7Cdball%40vmware.com%7C1f27abb785824caad36a08d60c3cb330%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C1%7C0%7C636709852193530364sdata=ivs6qeK%2BSpD1tLmPUq9VnwsBpPaf6sEnw7emt5N4iFE%3Dreserved=0


___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload TCP packets

2018-08-27 Thread Zhang, Jing C. (Nokia - CA/Ottawa) 
Yes, yes, pls help!!! I am desperate now ;-(

Jing

-Original Message-
From: Ben Pfaff  
Sent: Monday, August 27, 2018 12:47 PM
To: Zhang, Jing C. (Nokia - CA/Ottawa) ; Darrell Ball 

Cc: b...@openvswitch.org
Subject: Re: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload TCP 
packets

On Mon, Aug 27, 2018 at 01:20:15PM +, Zhang, Jing C. (Nokia - CA/Ottawa) 
wrote:
> We have customers reporting sluggish HTTP download with OVS 2.9.0. After 
> debugging, we find the issue is OVS 2.9.0 native firewall drops TCP control 
> packets (TCP ACK) with empty payload. The issue can be avoided by either 
> reverting back to the legacy Linux bridge firewall or enabling TCP timestamp 
> at both HTTP server and client sides.
> $ ovs-vswitchd --version
> ovs-vswitchd (Open vSwitch) 2.9.0
> DPDK 17.11.0

Darrell, this seems like it's in your area.  Will you take a look?

Thanks,

Ben.
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload TCP packets

2018-08-27 Thread Ben Pfaff 
On Mon, Aug 27, 2018 at 01:20:15PM +, Zhang, Jing C. (Nokia - CA/Ottawa) 
wrote:
> We have customers reporting sluggish HTTP download with OVS 2.9.0. After 
> debugging, we find the issue is OVS 2.9.0 native firewall drops TCP control 
> packets (TCP ACK) with empty payload. The issue can be avoided by either 
> reverting back to the legacy Linux bridge firewall or enabling TCP timestamp 
> at both HTTP server and client sides.
> $ ovs-vswitchd --version
> ovs-vswitchd (Open vSwitch) 2.9.0
> DPDK 17.11.0

Darrell, this seems like it's in your area.  Will you take a look?

Thanks,

Ben.
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

[ovs-discuss] OVS 2.9.0 native firewall drops empty payload TCP packets

2018-08-27 Thread Zhang, Jing C. (Nokia - CA/Ottawa) 
We have customers reporting sluggish HTTP download with OVS 2.9.0. After 
debugging, we find the issue is OVS 2.9.0 native firewall drops TCP control 
packets (TCP ACK) with empty payload. The issue can be avoided by either 
reverting back to the legacy Linux bridge firewall or enabling TCP timestamp at 
both HTTP server and client sides.
$ ovs-vswitchd --version
ovs-vswitchd (Open vSwitch) 2.9.0
DPDK 17.11.0

openvswitch-2.9.0-3.el7.x86_64
The issue can be easily reproduced with scapy.
This is an example empty payload TCP packet received on the physical NIC port:

# tcpdump tcp -i ens1f0 -nevvXX
tcpdump: listening on ens1f0, link-type EN10MB (Ethernet), capture size 262144 
bytes

19:05:52.354860 fa:16:3e:37:a1:4f > fa:16:3e:f8:db:64, ethertype 802.1Q 
(0x8100), length 64: vlan 628, p 0, ethertype IPv4, (tos 0x0, ttl 63, id 1, 
offset 0, flags [none], proto TCP (6), length 40)
135.255.159.203.64766 > 10.0.1.7.http: Flags [S], cksum 0x5f5e (correct), 
seq 100, win 8192, length 0
0x:  fa16 3ef8 db64 fa16 3e37 a14f 8100 0274  ..>..d..>7.O...t
0x0010:  0800 4500 0028 0001  3f06 48fe 87ff  ..E..(?.H...
0x0020:  9fcb 0a00 0107 fcfe 0050  0064   .P...d..
0x0030:   5002 2000 5f5e      ..P..._^

Debugging shows those packets can arrive on the tap interface if the firewall 
is skipped like the following:

ovs-ofctl add-flow br-int "table=81, priority=101, ct_state=-trk,reg5=0x3 
actions=output:3"

# tcpdump tcp -i tapeda6e769-ea -nevvXX
tcpdump: listening on tapeda6e769-ea, link-type EN10MB (Ethernet), capture size 
262144 bytes

17:26:47.091233 fa:16:3e:37:a1:4f > fa:16:3e:f8:db:64, ethertype IPv4 (0x0800), 
length 60: (tos 0x0, ttl 63, id 1, offset 0, flags [none], proto TCP (6), 
length 40)


I have been digging the related code areas but unable to find the obvious 
packet drop due to empty payload. Please advise of the root cause, or give 
pointers, I am looking for a quick local fix first.

No issue with firewall rules themselves, I copy paste here the rules leading to 
the packet drop on a lab I use for debugging, to help explain the context:
...
cookie=0x3671c32b581ad607, duration=211770.989s, table=0, n_packets=619026, 
n_bytes=168758644, idle_age=14, hard_age=65534, 
priority=3,in_port=3,dl_vlan=697 actions=mod_vlan_vid:1,resubmit(,60)
...
cookie=0x3671c32b581ad607, duration=211769.188s, table=81, n_packets=611880, 
n_bytes=168298703, idle_age=19, hard_age=65534, 
priority=90,ct_state=-trk,ip,reg5=0x1 
actions=ct(table=82,zone=NXM_NX_REG6[0..15])
...
cookie=0x3671c32b581ad607, duration=211769.188s, table=60, n_packets=619006, 
n_bytes=168757352, idle_age=14, hard_age=65534, 
priority=90,dl_vlan=1,dl_dst=fa:16:3e:9a:b1:5c 
actions=load:0x1->NXM_NX_REG5[],load:0x1->NXM_NX_REG6[],strip_vlan,resubmit(,81)



Thanks a lot.

Jing Zhang




___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss