[dpdk-dev] Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)
Hi Huawei, On 09/04/14 04:54, Xie, Huawei wrote: > Hi Franck: > I checked your original thread. > root at linux-native:~# tcpdump -vnei eth0 > tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 > bytes > 17:27:09.262926 00:1b:21:b9:9b:2c > 52:54:00:51:51:12, ethertype IPv4 > (0x0800), length 74: (tos 0x0, ttl 64, id 47743, offset 0, flags [DF], proto > TCP (6), length 60) > 1.1.1.3.34272 > 1.1.1.2.: Flags [S], cksum 0x0435 (incorrect -> > 0xb2dd), seq 1963818356, win 14600, options [mss 1460,sackOK,TS val 49530635 > ecr 0,nop,wscale 7], length 0 > 17:27:09.263220 52:54:00:51:51:12 > 00:1b:21:b9:9b:2c, ethertype IPv4 > (0x0800), length 60: (tos 0x0, ttl 64, id 3367, offset 0, flags [DF], proto > TCP (6), length 40) > 1.1.1.2. > 1.1.1.3.34272: Flags [R.], cksum 0x1db4 (correct), seq 0, > ack 1963818357, win 0, length 0 > > The packet from the guest, received on the native machine, has "correct" > checksum 0x1db4. The TCP connection is refused is due to this packet has R > flag. Could you check if it is the firewall that blocks the > connection? > > Besides, I did ssh tcp test between two guest VMs, the ssh connection could > be established correctly with tx checksum off. I didn't use OVDK, but set the > arp table and route table manually. > The packet flow is > Guest A(virtio)<->vhost example->Guest B(virtio) I re-made a test between two identical VMs, UDP and ICMP are fine. Some observations: on both client & server: ethtool -k eth1 | grep ": on" generic-receive-offload: on highdma: on [fixed] rx-vlan-filter: on [fixed] ethtool -k eth1 Features for eth1: rx-checksumming: off [fixed] tx-checksumming: off tx-checksum-ipv4: off [fixed] tx-checksum-unneeded: off [fixed] tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: off [fixed] tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: off tx-scatter-gather: off [fixed] tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: off tx-tcp-segmentation: off [fixed] tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: off [fixed] udp-fragmentation-offload: off [fixed] generic-segmentation-offload: off [requested on] generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: off [fixed] tx-vlan-offload: off [fixed] ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: on [fixed] rx-vlan-filter: on [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] On the client side: 11:00:23.296592 IP (tos 0x0, ttl 64, id 25239, offset 0, flags [DF], proto TCP (6), length 60) 1.1.1.2.47192 > 1.1.1.1.dsf: Flags [S], cksum 0x0433 (incorrect -> 0xe3bd), seq 3885781956, win 14600, options [mss 1460,sackOK,TS val 4294914387 ecr 0,nop,wscale 5], length 0 0x: 4500 003c 6297 4000 4006 d420 0101 0102 E.. 1.1.1.2.47192: Flags [R.], cksum 0xb5e6 (correct), seq 0, ack 3885781957, win 0, length 0 0x: 4500 0028 198f 4000 4006 1d3d 0101 0101 E..(.. at .@..= 0x0010: 0101 0102 022b b858 e79c 53c5 .+.X..S. 0x0020: 5014 b5e6 P... On the server side: 11:00:23.631991 IP (tos 0x0, ttl 64, id 25239, offset 0, flags [DF], proto TCP (6), length 60) 1.1.1.2.47192 > 1.1.1.1.dsf: Flags [S], cksum 0xe3bd (correct), seq 3885781956, win 14600, options [mss 1460,sackOK,TS val 4294914387 ecr 0,nop,wscale 5], length 0 0x: 4500 003c 6297 4000 4006 d420 0101 0102 E.. 1.1.1.2.47192: Flags [R.], cksum 0xb5e6 (correct), seq 0, ack 3885781957, win 0, length 0 0x: 4500 0028 198f 4000 4006 1d3d 0101 0101 E..(.. at .@..= 0x0010: 0101 0102 022b b858 e79c 53c5 .+.X..S. 0x0020: 5014 b5e6 P... On the client, ethtool announce that TX checksum offload is disabled, but tcpdump show it's not. On the server side, all packets looks fine from a checksum perspective but the SYN is answered by a RST. Please note that the same images behave correctly with a regular openvswitch, as long as TX checksuming offloading is disabled. If anyone has an equivalent setup, functioning properly, thanks for sharing its configurations parameters. if not, I might be able to wait for vhost backend PMD driver: I will then with testpmd instead of dpdk-ovs, to see where is the problem (maybe in my configuration...). When so you plan to release the vhost backend PMD driver? Thanks! Franck
[dpdk-dev] Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)
> -Original Message- > From: Franck Baudin [mailto:franck.baudin at qosmos.com] > Sent: Thursday, September 04, 2014 5:24 PM > To: Xie, Huawei; Gray, Mark D; Thomas Monjalon > Cc: dev at dpdk.org; dpdk-ovs at lists.01.org > Subject: Re: [dpdk-dev] Wrong TCP checksum of packets sent by Linux guest > (virtIO/vhost) > > Hi Huawei, > > On 09/04/14 04:54, Xie, Huawei wrote: > > Hi Franck: > > I checked your original thread. > > root at linux-native:~# tcpdump -vnei eth0 > > tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 > bytes > > 17:27:09.262926 00:1b:21:b9:9b:2c > 52:54:00:51:51:12, ethertype IPv4 > (0x0800), length 74: (tos 0x0, ttl 64, id 47743, offset 0, flags [DF], proto > TCP (6), > length 60) > > 1.1.1.3.34272 > 1.1.1.2.: Flags [S], cksum 0x0435 (incorrect -> > > 0xb2dd), > seq 1963818356, win 14600, options [mss 1460,sackOK,TS val 49530635 ecr > 0,nop,wscale 7], length 0 > > 17:27:09.263220 52:54:00:51:51:12 > 00:1b:21:b9:9b:2c, ethertype IPv4 > (0x0800), length 60: (tos 0x0, ttl 64, id 3367, offset 0, flags [DF], proto > TCP (6), > length 40) > > 1.1.1.2. > 1.1.1.3.34272: Flags [R.], cksum 0x1db4 (correct), seq > > 0, ack > 1963818357, win 0, length 0 > > > > The packet from the guest, received on the native machine, has "correct" > checksum 0x1db4. The TCP connection is refused is due to this packet has R > flag. > Could you check if it is the firewall that blocks the > > connection? > > > > Besides, I did ssh tcp test between two guest VMs, the ssh connection could > be established correctly with tx checksum off. I didn't use OVDK, but set the > arp > table and route table manually. > > The packet flow is > > Guest A(virtio)<->vhost example->Guest B(virtio) > I re-made a test between two identical VMs, UDP and ICMP are fine. Some > observations: > on both client & server: > > ethtool -k eth1 | grep ": on" > generic-receive-offload: on > highdma: on [fixed] > rx-vlan-filter: on [fixed] > > ethtool -k eth1 > Features for eth1: > rx-checksumming: off [fixed] > tx-checksumming: off > tx-checksum-ipv4: off [fixed] > tx-checksum-unneeded: off [fixed] > tx-checksum-ip-generic: off [fixed] > tx-checksum-ipv6: off [fixed] > tx-checksum-fcoe-crc: off [fixed] > tx-checksum-sctp: off [fixed] > scatter-gather: off > tx-scatter-gather: off [fixed] > tx-scatter-gather-fraglist: off [fixed] > tcp-segmentation-offload: off > tx-tcp-segmentation: off [fixed] > tx-tcp-ecn-segmentation: off [fixed] > tx-tcp6-segmentation: off [fixed] > udp-fragmentation-offload: off [fixed] > generic-segmentation-offload: off [requested on] > generic-receive-offload: on > large-receive-offload: off [fixed] > rx-vlan-offload: off [fixed] > tx-vlan-offload: off [fixed] > ntuple-filters: off [fixed] > receive-hashing: off [fixed] > highdma: on [fixed] > rx-vlan-filter: on [fixed] > vlan-challenged: off [fixed] > tx-lockless: off [fixed] > netns-local: off [fixed] > tx-gso-robust: off [fixed] > tx-fcoe-segmentation: off [fixed] > fcoe-mtu: off [fixed] > tx-nocache-copy: off > loopback: off [fixed] > > On the client side: > 11:00:23.296592 IP (tos 0x0, ttl 64, id 25239, offset 0, flags [DF], > proto TCP > (6), length 60) > 1.1.1.2.47192 > 1.1.1.1.dsf: Flags [S], cksum 0x0433 (incorrect -> > 0xe3bd), seq 3885781956, win 14600, options [mss 1460,sackOK,TS val > 4294914387 > ecr 0,nop,wscale 5], length 0 > 0x: 4500 003c 6297 4000 4006 d420 0101 0102 E.. 0x0010: 0101 0101 b858 022b e79c 53c4 .X.+..S. > 0x0020: a002 3908 0433 0204 05b4 0402 080a ..9..3.. > 0x0030: 3153 0103 0305 ..1S > 11:00:23.296881 IP (tos 0x0, ttl 64, id 6543, offset 0, flags [DF], > proto TCP > (6), length 40) > 1.1.1.1.dsf > 1.1.1.2.47192: Flags [R.], cksum 0xb5e6 (correct), seq 0, > ack 3885781957, win 0, length 0 > 0x: 4500 0028 198f 4000 4006 1d3d 0101 0101 E..(.. at .@..= > 0x0010: 0101 0102 022b b858 e79c 53c5 .+.X..S. > 0x0020: 5014 b5e6 P... > > On the server side: > 11:00:23.631991 IP (tos 0x0, ttl 64, id 25239, offset 0, flags [DF], > proto TCP > (6), length 60) > 1.1.1.2.47192 > 1.1.1.1.dsf: Flags [S], cksum 0xe3bd (correct), seq > 3885781956, win 14600, options [mss 1460,sackOK,TS val 4294914387 ecr > 0,nop,wscale 5], length 0 >
[dpdk-dev] Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)
> -Original Message-
> From: Franck Baudin [mailto:franck.baudin at qosmos.com]
> Sent: Wednesday, September 03, 2014 10:13 PM
> To: Xie, Huawei; Gray, Mark D; Thomas Monjalon
> Cc: dev at dpdk.org; dpdk-ovs at lists.01.org
> Subject: Re: [dpdk-dev] Wrong TCP checksum of packets sent by Linux guest
> (virtIO/vhost)
>
> Hi,
>
> On 09/03/14 13:13, Xie, Huawei wrote:
> > Looping in the dpdk-ovs list.
> >
> > * Does the new vhost API allow a user to know if all the relevant offloads
> > have
> > been
> > turned on/off for that interface? It seems that this is possible through the
> > virtio_net
> > structure but it would be good to get some feedback from the relevant person
> > working on DPDK (Huawei?).
> >
> > * If this is the case, then it is probably in the realm of the vswitch do
> > the actual
> > checksum (for VM-VM) or correctly configure the NIC when sending out
> through
> > the physical interface.
> >
> > Comments?
> >
> > Mark:
> > So far not supported. This is important as well in VxLan case. For the
> > packet
> flow
> > Guest A-> virtio -> ..->OVDK->.. -> Guest B.
> > 1) If guest A and B are on different host machines, say A and B
> > respectively,
> and if the nic on A supports
> > vxlan checksum offload, then both guest and host needn't generate checksum,
> the nic will
> > generate checksum for both inner and outer packet.
> > 2) In VM2VM case, as it is trusted communication channel, could we negotiate
> with the guest tcp stack not to verify checksum
> > for received packet?
> The problem is that any TCP packet send by a vanilla Linux guest through
> vhost is incorrect (VM to anything, including other colocalied VMs). In
> other words, the VM cannot use TCP. QEMU options and ethtool -K csum off
> tso off ("TCP stack negociation") have no effect, maybe because the
> vhost backend is misbehaving.
>
> Franck
Hi Franck:
I checked your original thread.
root at linux-native:~# tcpdump -vnei eth0
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535
bytes
17:27:09.262926 00:1b:21:b9:9b:2c > 52:54:00:51:51:12, ethertype IPv4 (0x0800),
length 74: (tos 0x0, ttl 64, id 47743, offset 0, flags [DF], proto TCP (6),
length 60)
1.1.1.3.34272 > 1.1.1.2.: Flags [S], cksum 0x0435 (incorrect -> 0xb2dd),
seq 1963818356, win 14600, options [mss 1460,sackOK,TS val 49530635 ecr
0,nop,wscale 7], length 0
17:27:09.263220 52:54:00:51:51:12 > 00:1b:21:b9:9b:2c, ethertype IPv4 (0x0800),
length 60: (tos 0x0, ttl 64, id 3367, offset 0, flags [DF], proto TCP (6),
length 40)
1.1.1.2. > 1.1.1.3.34272: Flags [R.], cksum 0x1db4 (correct), seq 0,
ack 1963818357, win 0, length 0
The packet from the guest, received on the native machine, has "correct"
checksum 0x1db4. The TCP connection is refused is due to this packet has R
flag. Could you check if it is the firewall that blocks the
connection?
Besides, I did ssh tcp test between two guest VMs, the ssh connection could be
established correctly with tx checksum off. I didn't use OVDK, but set the arp
table and route table manually.
The packet flow is
Guest A(virtio)<->vhost example->Guest B(virtio)
[dpdk-dev] Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)
Hi,
On 09/03/14 13:13, Xie, Huawei wrote:
> Looping in the dpdk-ovs list.
>
> * Does the new vhost API allow a user to know if all the relevant offloads
> have
> been
> turned on/off for that interface? It seems that this is possible through the
> virtio_net
> structure but it would be good to get some feedback from the relevant person
> working on DPDK (Huawei?).
>
> * If this is the case, then it is probably in the realm of the vswitch do the
> actual
> checksum (for VM-VM) or correctly configure the NIC when sending out through
> the physical interface.
>
> Comments?
>
> Mark:
> So far not supported. This is important as well in VxLan case. For the packet
> flow
> Guest A-> virtio -> ..->OVDK->.. -> Guest B.
> 1) If guest A and B are on different host machines, say A and B respectively,
> and if the nic on A supports
> vxlan checksum offload, then both guest and host needn't generate checksum,
> the nic will
> generate checksum for both inner and outer packet.
> 2) In VM2VM case, as it is trusted communication channel, could we negotiate
> with the guest tcp stack not to verify checksum
> for received packet?
The problem is that any TCP packet send by a vanilla Linux guest through
vhost is incorrect (VM to anything, including other colocalied VMs). In
other words, the VM cannot use TCP. QEMU options and ethtool -K csum off
tso off ("TCP stack negociation") have no effect, maybe because the
vhost backend is misbehaving.
Franck
[dpdk-dev] Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)
>
> Hi,
>
> On 09/03/14 13:13, Xie, Huawei wrote:
> > Looping in the dpdk-ovs list.
> >
> > * Does the new vhost API allow a user to know if all the relevant
> > offloads have been turned on/off for that interface? It seems that
> > this is possible through the virtio_net structure but it would be good
> > to get some feedback from the relevant person working on DPDK
> > (Huawei?).
> >
> > * If this is the case, then it is probably in the realm of the vswitch
> > do the actual checksum (for VM-VM) or correctly configure the NIC when
> > sending out through the physical interface.
> >
> > Comments?
> >
> > Mark:
> > So far not supported. This is important as well in VxLan case. For the
> > packet flow Guest A-> virtio -> ..->OVDK->.. -> Guest B.
> > 1) If guest A and B are on different host machines, say A and B
> > respectively, and if the nic on A supports vxlan checksum offload,
> > then both guest and host needn't generate checksum, the nic will generate
> checksum for both inner and outer packet.
> > 2) In VM2VM case, as it is trusted communication channel, could we
> > negotiate with the guest tcp stack not to verify checksum for received
> packet?
> The problem is that any TCP packet send by a vanilla Linux guest through
> vhost is incorrect (VM to anything, including other colocalied VMs). In other
> words, the VM cannot use TCP. QEMU options and ethtool -K csum off tso
> off ("TCP stack negociation") have no effect, maybe because the vhost
> backend is misbehaving.
This sounds like a DPDK issue. However, we did "clone and own" the vhost
code as it is a sample app. It may have changed since we integrated it.
We plan to use the vhost library interface when it is available on dpdk.org. So
that would pick up the latest version of the vhost code.
>
> Franck
[dpdk-dev] Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)
> -Original Message- > From: Gray, Mark D > Sent: Wednesday, September 03, 2014 6:01 PM > To: Thomas Monjalon; Franck BAUDIN; Xie, Huawei > Cc: dev at dpdk.org; dpdk-ovs at lists.01.org > Subject: RE: [dpdk-dev] Wrong TCP checksum of packets sent by Linux guest > (virtIO/vhost) > > > > > Hi Franck, > > > > 2014-09-02 13:20, Franck BAUDIN: > > > I am using dpdk-ovs 1.1.0 (latest release) as follow : > > > > > > linux-guest (no DPKD) <--- virtIO ---> OVDK 1.1.0 (with latest DPDK [*]) > > > < -- > > - Niantic --- > linux-native > > > > > [...] > > > UDP/ICMP connectivity is fine, but TCP checksum of packet sent by the > > guest are incorrect, as showed with tcpdump on linux-native. > > > > Thanks for reporting. > > Could you try virtio without vhost? > > > > > > Looping in the dpdk-ovs list. > > * Does the new vhost API allow a user to know if all the relevant offloads > have > been > turned on/off for that interface? It seems that this is possible through the > virtio_net > structure but it would be good to get some feedback from the relevant person > working on DPDK (Huawei?). > > * If this is the case, then it is probably in the realm of the vswitch do the > actual > checksum (for VM-VM) or correctly configure the NIC when sending out through > the physical interface. > > Comments? Mark: So far not supported. This is important as well in VxLan case. For the packet flow Guest A-> virtio -> ..->OVDK->.. -> Guest B. 1) If guest A and B are on different host machines, say A and B respectively, and if the nic on A supports vxlan checksum offload, then both guest and host needn't generate checksum, the nic will generate checksum for both inner and outer packet. 2) In VM2VM case, as it is trusted communication channel, could we negotiate with the guest tcp stack not to verify checksum for received packet? > > Mark
[dpdk-dev] Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)
> > Hi Franck, > > 2014-09-02 13:20, Franck BAUDIN: > > I am using dpdk-ovs 1.1.0 (latest release) as follow : > > > > linux-guest (no DPKD) <--- virtIO ---> OVDK 1.1.0 (with latest DPDK [*]) < > > -- > - Niantic --- > linux-native > > > [...] > > UDP/ICMP connectivity is fine, but TCP checksum of packet sent by the > guest are incorrect, as showed with tcpdump on linux-native. > > Thanks for reporting. > Could you try virtio without vhost? > > Looping in the dpdk-ovs list. * Does the new vhost API allow a user to know if all the relevant offloads have been turned on/off for that interface? It seems that this is possible through the virtio_net structure but it would be good to get some feedback from the relevant person working on DPDK (Huawei?). * If this is the case, then it is probably in the realm of the vswitch do the actual checksum (for VM-VM) or correctly configure the NIC when sending out through the physical interface. Comments? Mark
[dpdk-dev] Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)
Hi Thomas, On 09/02/14 15:29, Thomas Monjalon wrote: > UDP/ICMP connectivity is fine, but TCP checksum of packet sent by the guest > are incorrect, as showed with tcpdump on linux-native. > Thanks for reporting. > Could you try virtio without vhost? Nope, dpdk-ovs supports only vhost. Thanks! Franck
[dpdk-dev] Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)
Hi Franck, 2014-09-02 13:20, Franck BAUDIN: > I am using dpdk-ovs 1.1.0 (latest release) as follow : > > linux-guest (no DPKD) <--- virtIO ---> OVDK 1.1.0 (with latest DPDK [*]) < > --- Niantic --- > linux-native > [...] > UDP/ICMP connectivity is fine, but TCP checksum of packet sent by the guest > are incorrect, as showed with tcpdump on linux-native. Thanks for reporting. Could you try virtio without vhost? > This message and any attachments (the "message") are confidential, intended > solely for the addressees. If you are not the intended recipient, please > notify the sender immediately by e-mail and delete this message from your > system. In this case, you are not authorized to use, copy this message and/or > disclose the content to any other person. E-mails are susceptible to > alteration. Neither Qosmos nor any of its subsidiaries or affiliates shall be > liable for the message if altered, changed or falsified. > > Ce message et toutes ses pi?ces jointes (ci-apr?s le "message")sont > confidentiels et ?tablis ? l'intention exclusive de ses destinataires. Si > vous avez re?u ce message par erreur, merci d?en informer imm?diatement son > ?metteur par courrier ?lectronique et d?effacer ce message de votre syst?me. > Dans cette hypoth?se, vous n??tes pas autoris? ? utiliser, copier ce message > et/ou en divulguer le contenu ? un tiers. Tout message ?lectronique est > susceptible d'alt?ration. Qosmos et ses filiales d?clinent toute > responsabilit? au titre de ce message s'il a ?t? alt?r?, d?form? ou falsifi?. This big disclaimer block has no sense on a public mailing list. Please try to remove it. Thanks -- Thomas
[dpdk-dev] Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)
Hello, This is a repost of https://lists.01.org/pipermail/dpdk-ovs/2014-September/001788.html as the issue might be related to dpdk.org vhost code. I am using dpdk-ovs 1.1.0 (latest release) as follow : linux-guest (no DPKD) <--- virtIO ---> OVDK 1.1.0 (with latest DPDK [*]) < --- Niantic --- > linux-native [*] commit 8fd8bebc051704d7caecfed8d8a065a79c022329 Author: Adrien Mazarguil Date: Mon Sep 1 12:31:11 2014 +0200 UDP/ICMP connectivity is fine, but TCP checksum of packet sent by the guest are incorrect, as showed with tcpdump on linux-native. Is it supposed to work? If yes, what is the proper qemu version to use and its parameters? I am using qemu version provided by dpdk-ovs (same issue with PDDK 1.7.0 and latest one) and I followed this documentation: https://github.com/01org/dpdk-ovs/blob/development/docs/04_Sample_Configurations/02_Userspace-vHost.md Thanks! Franck This message and any attachments (the "message") are confidential, intended solely for the addressees. If you are not the intended recipient, please notify the sender immediately by e-mail and delete this message from your system. In this case, you are not authorized to use, copy this message and/or disclose the content to any other person. E-mails are susceptible to alteration. Neither Qosmos nor any of its subsidiaries or affiliates shall be liable for the message if altered, changed or falsified. Ce message et toutes ses pi?ces jointes (ci-apr?s le "message")sont confidentiels et ?tablis ? l'intention exclusive de ses destinataires. Si vous avez re?u ce message par erreur, merci d?en informer imm?diatement son ?metteur par courrier ?lectronique et d?effacer ce message de votre syst?me. Dans cette hypoth?se, vous n??tes pas autoris? ? utiliser, copier ce message et/ou en divulguer le contenu ? un tiers. Tout message ?lectronique est susceptible d'alt?ration. Qosmos et ses filiales d?clinent toute responsabilit? au titre de ce message s'il a ?t? alt?r?, d?form? ou falsifi?.
