We have customers reporting sluggish HTTP download with OVS 2.9.0. After
debugging, we find the issue is OVS 2.9.0 native firewall drops TCP control
packets (TCP ACK) with empty payload. The issue can be avoided by either
reverting back to the legacy Linux bridge firewall or enabling TCP timestamp at
both HTTP server and client sides.
$ ovs-vswitchd --version
ovs-vswitchd (Open vSwitch) 2.9.0
DPDK 17.11.0
openvswitch-2.9.0-3.el7.x86_64
The issue can be easily reproduced with scapy.
This is an example empty payload TCP packet received on the physical NIC port:
# tcpdump tcp -i ens1f0 -nevvXX
tcpdump: listening on ens1f0, link-type EN10MB (Ethernet), capture size 262144
bytes
19:05:52.354860 fa:16:3e:37:a1:4f > fa:16:3e:f8:db:64, ethertype 802.1Q
(0x8100), length 64: vlan 628, p 0, ethertype IPv4, (tos 0x0, ttl 63, id 1,
offset 0, flags [none], proto TCP (6), length 40)
135.255.159.203.64766 > 10.0.1.7.http: Flags [S], cksum 0x5f5e (correct),
seq 100, win 8192, length 0
0x0000: fa16 3ef8 db64 fa16 3e37 a14f 8100 0274 ..>..d..>7.O...t
0x0010: 0800 4500 0028 0001 0000 3f06 48fe 87ff ..E..(....?.H...
0x0020: 9fcb 0a00 0107 fcfe 0050 0000 0064 0000 .........P...d..
0x0030: 0000 5002 2000 5f5e 0000 0000 0000 0000 ..P..._^........
Debugging shows those packets can arrive on the tap interface if the firewall
is skipped like the following:
ovs-ofctl add-flow br-int "table=81, priority=101, ct_state=-trk,reg5=0x3
actions=output:3"
# tcpdump tcp -i tapeda6e769-ea -nevvXX
tcpdump: listening on tapeda6e769-ea, link-type EN10MB (Ethernet), capture size
262144 bytes
17:26:47.091233 fa:16:3e:37:a1:4f > fa:16:3e:f8:db:64, ethertype IPv4 (0x0800),
length 60: (tos 0x0, ttl 63, id 1, offset 0, flags [none], proto TCP (6),
length 40)
I have been digging the related code areas but unable to find the obvious
packet drop due to empty payload. Please advise of the root cause, or give
pointers, I am looking for a quick local fix first.
No issue with firewall rules themselves, I copy paste here the rules leading to
the packet drop on a lab I use for debugging, to help explain the context:
...
cookie=0x3671c32b581ad607, duration=211770.989s, table=0, n_packets=619026,
n_bytes=168758644, idle_age=14, hard_age=65534,
priority=3,in_port=3,dl_vlan=697 actions=mod_vlan_vid:1,resubmit(,60)
...
cookie=0x3671c32b581ad607, duration=211769.188s, table=81, n_packets=611880,
n_bytes=168298703, idle_age=19, hard_age=65534,
priority=90,ct_state=-trk,ip,reg5=0x1
actions=ct(table=82,zone=NXM_NX_REG6[0..15])
...
cookie=0x3671c32b581ad607, duration=211769.188s, table=60, n_packets=619006,
n_bytes=168757352, idle_age=14, hard_age=65534,
priority=90,dl_vlan=1,dl_dst=fa:16:3e:9a:b1:5c
actions=load:0x1->NXM_NX_REG5[],load:0x1->NXM_NX_REG6[],strip_vlan,resubmit(,81)
Thanks a lot.
Jing Zhang
_______________________________________________
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
