Problem:
---------
We are running centos 7.2 (ovs-kernel module from distribution) with OVS
2.3.1.
Large tcp frames (> 30Kbytes gro'd) are coming into a vxlan-tunnel port
and switched by an ovs-bridge onto a veth port
vxlan-tunnel--(gro'd tcp frames sized > 30K)-->OVS-bridge--(frames sized
1448)-->veth port
As the ovs transmits onto the veth port, the frames are getting
segmented into 1448 byte sized packets. All the offload settings are at
default on the veth port (gso on, gro on, tso on)
We expect that the OVS can transmit the frame without segmentation onto
the veth port. But, due to segmentation, the performance suffers.
Root Cause
-----------
The following trace with debugs shows the root cause that the gso_type
is not correctly set on
the skb as the packet comes in, and as a result, the dev_xmit_skb is
segmenting the frame as ovs transmits it.
ksoftirqd/25-231 [025] ..s. 319.964120: vxlan_rcv:
gso_size=1448,gso_type=1001
The above is debug in vport-vxlan.c:vxlan_rcv:
trace_printk("gso_size=%d,gso_type=%x\n",skb_shinfo(skb)->gso_size,skb_shinfo(skb)->gso_type);
Here, gso_type seem to have an invalid value.
ksoftirqd/25-231 [025] ..s. 319.964120: ovs_vport_receive
<-vxlan_rcv
ksoftirqd/25-231 [025] ..s. 319.964121: ovs_flow_key_extract
<-ovs_vport_receive
ksoftirqd/25-231 [025] ..s. 319.964121: key_extract
<-ovs_flow_key_extract
ksoftirqd/25-231 [025] ..s. 319.964121: check_header
<-key_extract
ksoftirqd/25-231 [025] ..s. 319.964121: tcphdr_ok <-key_extract
ksoftirqd/25-231 [025] ..s. 319.964121: ovs_dp_process_packet
<-ovs_vport_receive
ksoftirqd/25-231 [025] ..s. 319.964121:
ovs_flow_tbl_lookup_stats <-ovs_dp_process_packet
ksoftirqd/25-231 [025] ..s. 319.964121: masked_flow_lookup
<-ovs_flow_tbl_lookup_stats
ksoftirqd/25-231 [025] ..s. 319.964121: ovs_flow_mask_key
<-masked_flow_lookup
ksoftirqd/25-231 [025] ..s. 319.964122: find_bucket.isra.2
<-masked_flow_lookup
ksoftirqd/25-231 [025] ..s. 319.964122: masked_flow_lookup
<-ovs_flow_tbl_lookup_stats
ksoftirqd/25-231 [025] ..s. 319.964122: ovs_flow_mask_key
<-masked_flow_lookup
ksoftirqd/25-231 [025] ..s. 319.964122: find_bucket.isra.2
<-masked_flow_lookup
ksoftirqd/25-231 [025] ..s. 319.964122: ovs_flow_stats_update
<-ovs_dp_process_packet
ksoftirqd/25-231 [025] ..s. 319.964122: _raw_spin_loca
<-ovs_flow_stats_update
ksoftirqd/25-231 [025] ..s. 319.964122: _raw_spin_unlock
<-ovs_flow_stats_update
ksoftirqd/25-231 [025] ..s. 319.964122: ovs_execute_actions
<-ovs_dp_process_packet
ksoftirqd/25-231 [025] ..s. 319.964123: do_execute_actions
<-ovs_execute_actions
ksoftirqd/25-231 [025] ..s. 319.964123: do_output
<-do_execute_actions
ksoftirqd/25-231 [025] ..s. 319.964123: ovs_lookup_vport
<-do_output
ksoftirqd/25-231 [025] ..s. 319.964123: ovs_vport_send
<-do_output
ksoftirqd/25-231 [025] ..s. 319.964123: netdev_send
<-ovs_vport_send
ksoftirqd/25-231 [025] ..s. 319.964123: netdev_send: devmtu=8950
ksoftirqd/25-231 [025] ..s. 319.964123: netdev_send:
devname=ovs-veth0
ksoftirqd/25-231 [025] ..s. 319.964124: netdev_send: gso_segs=45
ksoftirqd/25-231 [025] ..s. 319.964124: netif_skb_features
<-netdev_send
ksoftirqd/25-231 [025] ..s. 319.964124: skb_network_protocol
<-netif_skb_features
ksoftirqd/25-231 [025] ..s. 319.964124: netdev_send: ip_summed=3
ksoftirqd/25-231 [025] ..s. 319.964124: netdev_send:
gso_segs=45,len=65226,feat=6103db59e9
ksoftirqd/25-231 [025] ..s. 319.964124: netdev_send:
net_gso_ok=0,has_frag_list=1,frag_feat=1
ksoftirqd/25-231 [025] ..s. 319.964125: netdev_send:
gso_size=1448,gso_type=4097,ok=0
The above are debugs in vport-netdev.c:netdev_send:
trace_printk("devmtu=%d\n",mtu);
trace_printk("devname=%s\n",netdev_vport->dev->name);
trace_printk("gso_segs=%d\n",skb_shinfo(skb)->gso_segs);
features = netif_skb_features(skb);
feature = skb_shinfo(skb)->gso_type << NETIF_F_GSO_SHIFT;
net_gso_ok = ((features & feature) == feature);
gso_ok = skb_gso_ok(skb, features);
has_frag_list = skb_has_frag_list(skb);
frag_feat = features & NETIF_F_FRAGLIST ? 1 : 0;
trace_printk("ip_summed=%d\n",skb->ip_summed);
trace_printk("gso_segs=%d,len=%d,feat=%llx\n",skb_shinfo(skb)->gso_segs,len,features);
trace_printk("net_gso_ok=%d,has_frag_list=%d,frag_feat=%d\n",net_gso_ok,has_frag_list,frag_feat);
trace_printk("gso_size=%d,gso_type=%d,ok=%d\n",skb_shinfo(skb)->gso_size,skb_shinfo(skb)->gso_type,gso_ok);
ksoftirqd/25-231 [025] ..s. 319.964125: dev_queue_xmit
<-netdev_send
ksoftirqd/25-231 [025] ..s. 319.964125: local_bh_disable
<-dev_queue_xmit
ksoftirqd/25-231 [025] ..s. 319.964125: netdev_pick_tx
<-dev_queue_xmit
ksoftirqd/25-231 [025] ..s. 319.964125: _raw_spin_lock
<-dev_queue_xmit
ksoftirqd/25-231 [025] ..s. 319.964125:
validate_xmit_skb.part.86 <-dev_queue_xmit
ksoftirqd/25-231 [025] ..s. 319.964125: netif_skb_features
<-validate_xmit_skb.part.86
ksoftirqd/25-231 [025] ..s. 319.964125: skb_network_protocol
<-netif_skb_features
validate_xmit_skb() calls netif_needs_gso()
netif_needs_gso() is true because:
skb_gso_ok is FALSE !! because net_gso_ok() is FALSE
because
skb_shinfo(skb)->gso_type has a bad value of 0x1001 !!
ksoftirqd/25-231 [025] ..s. 319.964125: __skb_gso_segment
<-validate_xmit_skb.part.86
ksoftirqd/25-231 [025] ..s. 319.964126: skb_mac_gso_segment
<-__skb_gso_segment
ksoftirqd/25-231 [025] ..s. 319.964126: skb_network_protocol
<-skb_mac_gso_segment
ksoftirqd/25-231 [025] ..s. 319.964126: inet_gso_segment
<-skb_mac_gso_segment
ksoftirqd/25-231 [025] ..s. 319.964126: tcp4_gso_segment
<-inet_gso_segment
ksoftirqd/25-231 [025] ..s. 319.964126: tcp_gso_segment
<-tcp4_gso_segment
ksoftirqd/25-231 [025] ..s. 319.964126: skb_segment
<-tcp_gso_segment
Frame is getting segmented
I tried the following experiment: in vport-vxlan.c:vxlan_rcv() set the
value of gso_type as:
skb_shinfo(skb)->gso_type = skb_shinfo(skb)->gso_type&0xff;
With this change, frames dont segment onto the veth.
Has this sort of problem been fixed already in more recent versions ?
_______________________________________________
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev
