Hi Ilya,
The whole output of 'ethtool -k ens6' is here:
$ ethtool -k ens6
Features for ens6:
rx-checksumming: on [fixed]
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: on
tx-tcp-mangleid-segmentation: on
tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: on [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
yfs@ubuntu:~$ ethtool -k ens6 | grep rx
rx-checksumming: on [fixed]
rx-vlan-offload: off [fixed]
rx-vlan-filter: on [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
yfs@ubuntu:~$ ethtool -k ens6
Features for ens6:
rx-checksumming: on [fixed]
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: on
tx-tcp-mangleid-segmentation: on
tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: on [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
Thanks,
Yifeng
On Wed, Jan 29, 2020 at 4:07 AM Ilya Maximets <i.maxim...@ovn.org> wrote:
>
> On 29.01.2020 12:25, Flavio Leitner wrote:
> > On Tue, Jan 28, 2020 at 03:23:02PM -0800, Yifeng Sun wrote:
> >> Sure.
> >>
> >> Firstly, make sure userspace-tso-enable is true
> >> # ovs-vsctl get Open_vSwitch . other_config
> >> {dpdk-init="true", enable-statistics="true", pmd-cpu-mask="0xf",
> >> userspace-tso-enable="true"}
> >>
> >> Next, create 2 VMs with vhostuser-type interface on the same KVM host:
> >> <interface type='vhostuser'>
> >> <mac address='88:69:00:00:00:11'/>
> >> <source type='unix'
> >> path='/tmp/041afca0-6e11-4eab-a62f-1ccf5cd318fd' mode='server'/>
> >> <model type='virtio'/>
> >> <driver queues='2' rx_queue_size='512'>
> >> <host csum='on' tso4='on' tso6='on'/>
> >> <guest csum='on' tso4='on' tso6='on'/>
> >
> > I have other options set, but I don't think they are related:
> > <host csum='on' gso='off' tso4='on' tso6='on' ecn='off'
> > ufo='off' mrg_rxbuf='on'/>> <guest csum='on' tso4='on' tso6='on'
> > ecn='off' ufo='off'/>
> >
> >
> >> </driver>
> >> <alias name='net2'/>
> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> >> function='0x0'/>
> >> </interface>
> >>
> >> When VM boots up, turn on tx, tso and sg
> >> # ethtool -K ens6 tx on
> >> # ethtool -K ens6 tso on
> >> # ethtool -K ens6 sg on
>
> Could you, please, provide the output of 'ethtool -k ens6'?
> If for some reason rx offloading is not enabled by default, you need to
> enable it too.
>
> >
> > All the needed offloading features are turned on by default,
> > so I don't change anything in my testbed.
> >
> >> Then run 'iperf -s' on one VM and 'iperf -c xx.xx.xx.xx' on another VM.
> >> Iperf doesn't work if there is no chage to VM's kernel. `tcpdump` shows
> >> that iperf server received packets with invalid TCP checksum.
> >> `nstat -a` shows that TcpInCsumErr number is accumulating.
> >>
> >> After adding changes to VM's kernel as below, iperf works properly.
> >> in tcp_v4_rcv()
> >> - if (skb_checksum_init(skb, IPPROTO_TCP, inet_compute_pseudo))
> >> + if (skb_checksum_init(skb, IPPROTO_TCP, inet_compute_pseudo))
> >>
> >> static inline bool tcp_checksum_complete(struct sk_buff *skb)
> >> {
> >> return 0;
> >> }
> >
> > That's odd. Which kernel is that? Maybe I can try the same version.
> > I am using 5.2.14-200.fc30.x86_64.
> >
> > Looks like somehow the packet lost its offloading flags, then kernel
> > has to check the csum and since it wasn't calculated before, it's
> > just random garbage.
> >
> > fbl
> >
> >
> >>
> >>
> >>
> >> Best,
> >> Yifeng
> >>
> >> On Tue, Jan 28, 2020 at 2:52 PM Flavio Leitner <f...@sysclose.org> wrote:
> >>>
> >>> On Tue, Jan 28, 2020 at 02:21:30PM -0800, Yifeng Sun wrote:
> >>>> Hi Flavio,
> >>>>
> >>>> Thanks for the explanation. I followed the steps in the document but
> >>>> TCP connection still failed to build between 2 VMs.
> >>>>
> >>>> I finally modified VM's kernel directly to disable TCP checksum
> >>>> validation
> >>>> to get it working properly. I got 30.0Gbps for 'iperf' between 2 VMs.
> >>>
> >>> Could you provide more details on how you did that? What's running
> >>> inside the VM?
> >>>
> >>> I don't change anything inside of the VMs (Linux) in my testbed.
> >>>
> >>> fbl
> >>>
> >>>
> >>>>
> >>>> Best,
> >>>> Yifeng
> >>>>
> >>>>
> >>>> On Tue, Jan 28, 2020 at 4:00 AM Flavio Leitner <f...@sysclose.org> wrote:
> >>>>>
> >>>>> On Mon, Jan 27, 2020 at 05:17:01PM -0800, Yifeng Sun wrote:
> >>>>>> Hi Ilya,
> >>>>>>
> >>>>>> Thanks for your reply.
> >>>>>>
> >>>>>> The thing is, if checksum offloading is enabled in both VMs, then
> >>>>>> sender VM will send
> >>>>>> a packet with invalid TCP checksum, and later OVS will send this
> >>>>>> packet to receiver
> >>>>>> VM directly without calculating a valid checksum. As a result,
> >>>>>> receiver VM will drop
> >>>>>> this packet because it contains invalid checksum. This is what
> >>>>>> happened when I tried
> >>>>>> this patch.
> >>>>>>
> >>>>>
> >>>>> When TSO is enabled, the TX checksumming offloading is required,
> >>>>> then you will see invalid checksum. This is well documented here:
> >>>>>
> >>>>> https://github.com/openvswitch/ovs/blob/master/Documentation/topics/userspace-tso.rst#userspace-datapath---tso
> >>>>>
> >>>>> "Additionally, if the traffic is headed to a VM within the same host
> >>>>> further optimization can be expected. As the traffic never leaves
> >>>>> the machine, no MTU needs to be accounted for, and thus no
> >>>>> segmentation and checksum calculations are required, which saves yet
> >>>>> more cycles."
> >>>>>
> >>>>> Therefore, it's expected to see bad csum in the traffic dumps.
> >>>>>
> >>>>> To use the feature, you need few steps: enable the feature in OvS
> >>>>> enable in qemu and inside the VM. The linux guest usually enable
> >>>>> the feature by default if qemu offers it.
> >>>>>
> >>>>> HTH,
> >>>>> fbl
> >>>>>
> >>>>>
> >>>>>> Best,
> >>>>>> Yifeng
> >>>>>>
> >>>>>> On Mon, Jan 27, 2020 at 12:09 PM Ilya Maximets <i.maxim...@ovn.org>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> On 27.01.2020 18:24, Yifeng Sun wrote:
> >>>>>>>> Hi Flavio,
> >>>>>>>>
> >>>>>>>> I am testing your patch using iperf between 2 VMs on the same host.
> >>>>>>>> But it seems that TCP connection can't be created between these 2
> >>>>>>>> VMs.
> >>>>>>>> When inspecting further, I found that TCP packets have invalid
> >>>>>>>> checksums.
> >>>>>>>> This might be the reason.
> >>>>>>>>
> >>>>>>>> I am wondering if I missed something in the setup? Thanks a lot.
> >>>>>>>
> >>>>>>> I didn't test myself, but according to current design, checksum
> >>>>>>> offloading
> >>>>>>> (rx and tx) shuld be enabled in both VMs. Otherwise all the packets
> >>>>>>> will
> >>>>>>> be dropped by the guest kernel.
> >>>>>>>
> >>>>>>> Best regards, Ilya Maximets.
> >>>>>
> >>>>> --
> >>>>> fbl
> >>>
> >>> --
> >>> fbl
> >
_______________________________________________
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
