Re: [vpp-dev] Issue forwarding TCP packets
This is a system problem; vpp can’t solve it all by itself. When forwarding packets at L2, vpp doesn’t look past the ethernet header. It’s simply delivering packets generated by the Linux kernel on one interface to another Linux kernel interface. The kernel cheats by not generating L4 checksums. When we deliver such packets to a Linux kernel interface which isn’t part of a Linux bridge, it throws them on the floor. Linux bridging winks and nods, knowing that the L4 checksum won’t be set. Two options: compute the checksums in software, or ignore them across the board. Neither option is perfect. The checksum computation is slow. I’m not a big believer in checksums, to be honest, but ignoring the L4 checksum across the board is probably not acceptable. Thanks… Dave From: vpp-dev-boun...@lists.fd.io [mailto:vpp-dev-boun...@lists.fd.io] On Behalf Of Prabhjot Singh Sethi Sent: Wednesday, August 30, 2017 9:45 AM To: Florin Coras ; Prabhjot Singh Sethi Cc: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Issue forwarding TCP packets Hi Florin, For now we can proceed with a requirement to disable offload on VMs. but still keeping performance in view i think VPP should have option similar to linux bridge to be able to do forwarding without correct checksums, while offloading is enable the real checksum calculations should be done only when the packet is egressing out of the server to other compute nodes or gateway. i think we can leave the topic open for now, and it can be considered later for performance improvements for TCP Regards, Prabhjot - Original Message - From: "Florin Coras" mailto:fcoras.li...@gmail.com>> To: "Prabhjot Singh Sethi" mailto:prabh...@techtrueup.com>>. Cc: mailto:vpp-dev@lists.fd.io>> Sent: Tue, 29 Aug 2017 11:13:14 -0700 Subject: Re: [vpp-dev] Issue forwarding TCP packets Hi Prabhjot, In your setup, VPP just switches packets from one interface to the other, irrespective of their L4 checksum. That is, it does not read nor update them. Therefore, since the source Linux interface does not provide correct checksums, the destination rejects all packets. As you’ve noted lower, although the packets are ultimately delivered to the app, even with the Linux bridge, the checksum is not correctly computed when offloading is on. I suspect the reason for this is an underlying optimization in the Linux kernel whereby local interfaces can exchange L4 packets without the use of checksums. In fact, the approach makes sense because packets never leave the kernel between the sndmsg() and recvmsg() calls so there’s no need to protect data integrity. However, this condition does not hold anymore when vpp does the switching, so the checksums must be computed before the packets leave the kernel (hence the need for checksum offload disabling). Hope this helps, Florin > On Aug 28, 2017, at 11:46 PM, > prabh...@techtrueup.com<mailto:prabh...@techtrueup.com> wrote: > > Thanks Florin, > it works with offload disabled, earlier when i tried changing offload > settings i missed doing it on one machine. > > However my question is still about why VPP is unable to handle it, is it a > missing configuration or missing functionality in VPP? > > i can ssh from one vm to another without turning off offload while using > linux bridge, and i don't see any change in the packet as well after being > forwarded. Though tcpdump complains about checksum, but everything works fine. > > packet ingressing into linux bridge from vm-1 > 06:41:58.626869 02:53:71:ef:2f:2a > 02:91:fb:46:9e:43, ethertype IPv4 > (0x0800), length 74: (tos 0x0, ttl 64, id 6341, offset 0, flags [DF], proto > TCP (6), length 60) > 1.1.1.3.51912 > 1.1.1.4.22: Flags [S], cksum 0x0437 (incorrect -> 0xdeae), > seq 2306393024, win 29200, options [mss 1460,sackOK,TS val 1825541 ecr > 0,nop,wscale 7], length 0 > 0x: 0291 fb46 9e43 0253 71ef 2f2a 0800 4500 > 0x0010: 003c 18c5 4000 4006 1def 0101 0103 0101 > 0x0020: 0104 cac8 0016 8978 c3c0 a002 > 0x0030: 7210 0437 0204 05b4 0402 080a 001b > 0x0040: db05 0103 0307 > > packet egressing from linux bridge to vm-2 > 06:41:58.627130 02:53:71:ef:2f:2a > 02:91:fb:46:9e:43, ethertype IPv4 > (0x0800), length 74: (tos 0x0, ttl 64, id 6341, offset 0, flags [DF], proto > TCP (6), length 60) > 1.1.1.3.51912 > 1.1.1.4.22: Flags [S], cksum 0x0437 (incorrect -> 0xdeae), > seq 2306393024, win 29200, options [mss 1460,sackOK,TS val 1825541 ecr > 0,nop,wscale 7], length 0 > 0x: 0291 fb46 9e43 0253 71ef 2f2a 0800 4500 > 0x0010: 003c 18c5 4000 4006 1def 0101 0103 0101 > 0x0020: 0104 cac8 0016 8978 c3c0 a002 > 0x0030: 7210 0437 0204 05b4 0402 080a 001b > 0x0040: db05 0103 0307 > > > Regards, > Prabhjot > > Quoting Florin Coras mai
Re: [vpp-dev] Issue forwarding TCP packets
Hi Florin, For now we can proceed with a requirement to disable offload on VMs. but still keeping performance in view i think VPP should have option similar to linux bridge to be able to do forwarding without correct checksums, while offloading is enable the real checksum calculations should be done only when the packet is egressing out of the server to other compute nodes or gateway. i think we can leave the topic open for now, and it can be considered later for performance improvements for TCP Regards, Prabhjot - Original Message - From: "Florin Coras" To: "Prabhjot Singh Sethi" . Cc: Sent: Tue, 29 Aug 2017 11:13:14 -0700 Subject: Re: [vpp-dev] Issue forwarding TCP packets Hi Prabhjot, In your setup, VPP just switches packets from one interface to the other, irrespective of their L4 checksum. That is, it does not read nor update them. Therefore, since the source Linux interface does not provide correct checksums, the destination rejects all packets. As you’ve noted lower, although the packets are ultimately delivered to the app, even with the Linux bridge, the checksum is not correctly computed when offloading is on. I suspect the reason for this is an underlying optimization in the Linux kernel whereby local interfaces can exchange L4 packets without the use of checksums. In fact, the approach makes sense because packets never leave the kernel between the sndmsg() and recvmsg() calls so there’s no need to protect data integrity. However, this condition does not hold anymore when vpp does the switching, so the checksums must be computed before the packets leave the kernel (hence the need for checksum offload disabling). Hope this helps, Florin > On Aug 28, 2017, at 11:46 PM, prabh...@techtrueup.com wrote: > > Thanks Florin, > it works with offload disabled, earlier when i tried changing offload settings i missed doing it on one machine. > > However my question is still about why VPP is unable to handle it, is it a missing configuration or missing functionality in VPP? > > i can ssh from one vm to another without turning off offload while using linux bridge, and i don't see any change in the packet as well after being forwarded. Though tcpdump complains about checksum, but everything works fine. > > packet ingressing into linux bridge from vm-1 > 06:41:58.626869 02:53:71:ef:2f:2a > 02:91:fb:46:9e:43, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 6341, offset 0, flags [DF], proto TCP (6), length 60) > 1.1.1.3.51912 > 1.1.1.4.22: Flags [S], cksum 0x0437 (incorrect -> 0xdeae), seq 2306393024, win 29200, options [mss 1460,sackOK,TS val 1825541 ecr 0,nop,wscale 7], length 0 > 0x: 0291 fb46 9e43 0253 71ef 2f2a 0800 4500 > 0x0010: 003c 18c5 4000 4006 1def 0101 0103 0101 > 0x0020: 0104 cac8 0016 8978 c3c0 a002 > 0x0030: 7210 0437 0204 05b4 0402 080a 001b > 0x0040: db05 0103 0307 > > packet egressing from linux bridge to vm-2 > 06:41:58.627130 02:53:71:ef:2f:2a > 02:91:fb:46:9e:43, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 6341, offset 0, flags [DF], proto TCP (6), length 60) > 1.1.1.3.51912 > 1.1.1.4.22: Flags [S], cksum 0x0437 (incorrect -> 0xdeae), seq 2306393024, win 29200, options [mss 1460,sackOK,TS val 1825541 ecr 0,nop,wscale 7], length 0 > 0x: 0291 fb46 9e43 0253 71ef 2f2a 0800 4500 > 0x0010: 003c 18c5 4000 4006 1def 0101 0103 0101 > 0x0020: 0104 cac8 0016 8978 c3c0 a002 > 0x0030: 7210 0437 0204 05b4 0402 080a 001b > 0x0040: db05 0103 0307 > > > Regards, > Prabhjot > > Quoting Florin Coras : > >> Hi Prabhjot, >> >> From your description, I suspect it may be a linux tcp checksum offload issue. Could you try disabling it for all interfaces with: >> >> ethtool --offload rx off tx off >> >> Hope this helps, >> Florin >> >>> On Aug 28, 2017, at 10:56 AM, Prabhjot Singh Sethi wrote: >>> >>> We have been trying to use VPP as a L2/L3 forwarding data path between VMs. >>> >>> where we have connected tap interfaces from VM to VPP using "create host-interface name " >>> after adding both the interfaces to same bridge-domain (bd_id = 2), we are able to ping from >>> one VM to another (both in same subnet). However when we try to do ssh we observe that all the >>> tcp packets are transmitted/forwarded by VPP to the other end but they are dropped as >>> "bad segments received". >>> >>> Can anyone help with what could be wrong here, are we missing any other required config? >>> >>> Note:- same VMs works well when connected to linux bridge >>> >>> root@vm-2:~# netstat -s | grep -A11 -i
Re: [vpp-dev] Issue forwarding TCP packets
Hi Prabhjot, In your setup, VPP just switches packets from one interface to the other, irrespective of their L4 checksum. That is, it does not read nor update them. Therefore, since the source Linux interface does not provide correct checksums, the destination rejects all packets. As you’ve noted lower, although the packets are ultimately delivered to the app, even with the Linux bridge, the checksum is not correctly computed when offloading is on. I suspect the reason for this is an underlying optimization in the Linux kernel whereby local interfaces can exchange L4 packets without the use of checksums. In fact, the approach makes sense because packets never leave the kernel between the sndmsg() and recvmsg() calls so there’s no need to protect data integrity. However, this condition does not hold anymore when vpp does the switching, so the checksums must be computed before the packets leave the kernel (hence the need for checksum offload disabling). Hope this helps, Florin > On Aug 28, 2017, at 11:46 PM, prabh...@techtrueup.com wrote: > > Thanks Florin, >it works with offload disabled, earlier when i tried changing offload > settings i missed doing it on one machine. > > However my question is still about why VPP is unable to handle it, is it a > missing configuration or missing functionality in VPP? > > i can ssh from one vm to another without turning off offload while using > linux bridge, and i don't see any change in the packet as well after being > forwarded. Though tcpdump complains about checksum, but everything works fine. > > packet ingressing into linux bridge from vm-1 > 06:41:58.626869 02:53:71:ef:2f:2a > 02:91:fb:46:9e:43, ethertype IPv4 > (0x0800), length 74: (tos 0x0, ttl 64, id 6341, offset 0, flags [DF], proto > TCP (6), length 60) >1.1.1.3.51912 > 1.1.1.4.22: Flags [S], cksum 0x0437 (incorrect -> 0xdeae), > seq 2306393024, win 29200, options [mss 1460,sackOK,TS val 1825541 ecr > 0,nop,wscale 7], length 0 >0x: 0291 fb46 9e43 0253 71ef 2f2a 0800 4500 >0x0010: 003c 18c5 4000 4006 1def 0101 0103 0101 >0x0020: 0104 cac8 0016 8978 c3c0 a002 >0x0030: 7210 0437 0204 05b4 0402 080a 001b >0x0040: db05 0103 0307 > > packet egressing from linux bridge to vm-2 > 06:41:58.627130 02:53:71:ef:2f:2a > 02:91:fb:46:9e:43, ethertype IPv4 > (0x0800), length 74: (tos 0x0, ttl 64, id 6341, offset 0, flags [DF], proto > TCP (6), length 60) >1.1.1.3.51912 > 1.1.1.4.22: Flags [S], cksum 0x0437 (incorrect -> 0xdeae), > seq 2306393024, win 29200, options [mss 1460,sackOK,TS val 1825541 ecr > 0,nop,wscale 7], length 0 >0x: 0291 fb46 9e43 0253 71ef 2f2a 0800 4500 >0x0010: 003c 18c5 4000 4006 1def 0101 0103 0101 >0x0020: 0104 cac8 0016 8978 c3c0 a002 >0x0030: 7210 0437 0204 05b4 0402 080a 001b >0x0040: db05 0103 0307 > > > Regards, > Prabhjot > > Quoting Florin Coras : > >> Hi Prabhjot, >> >> From your description, I suspect it may be a linux tcp checksum offload >> issue. Could you try disabling it for all interfaces with: >> >> ethtool --offload rx off tx off >> >> Hope this helps, >> Florin >> >>> On Aug 28, 2017, at 10:56 AM, Prabhjot Singh Sethi >>> wrote: >>> >>> We have been trying to use VPP as a L2/L3 forwarding data path between VMs. >>> >>> where we have connected tap interfaces from VM to VPP using "create >>> host-interface name " >>> after adding both the interfaces to same bridge-domain (bd_id = 2), we are >>> able to ping from >>> one VM to another (both in same subnet). However when we try to do ssh we >>> observe that all the >>> tcp packets are transmitted/forwarded by VPP to the other end but they are >>> dropped as >>> "bad segments received". >>> >>> Can anyone help with what could be wrong here, are we missing any other >>> required config? >>> >>> Note:- same VMs works well when connected to linux bridge >>> >>> root@vm-2:~# netstat -s | grep -A11 -i tcp: >>> Tcp: >>>33 active connections openings >>>4 passive connection openings >>>0 failed connection attempts >>>1 connection resets received >>>1 connections established >>>885 segments received >>>650 segments send out >>>8 segments retransmited >>>19 bad segments received. Keeps increasing for >>> every tcp packet >>>2 resets sent >>>InCsumErrors: 19 Keeps increasing for >>> every tcp packet >>> root@vm-2:~# >>> >>> Regards, >>> Prabhjot >>> ___ >>> vpp-dev mailing list >>> vpp-dev@lists.fd.io >>> https://lists.fd.io/mailman/listinfo/vpp-dev > > ___ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev
Re: [vpp-dev] Issue forwarding TCP packets
Thanks Florin, it works with offload disabled, earlier when i tried changing offload settings i missed doing it on one machine. However my question is still about why VPP is unable to handle it, is it a missing configuration or missing functionality in VPP? i can ssh from one vm to another without turning off offload while using linux bridge, and i don't see any change in the packet as well after being forwarded. Though tcpdump complains about checksum, but everything works fine. packet ingressing into linux bridge from vm-1 06:41:58.626869 02:53:71:ef:2f:2a > 02:91:fb:46:9e:43, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 6341, offset 0, flags [DF], proto TCP (6), length 60) 1.1.1.3.51912 > 1.1.1.4.22: Flags [S], cksum 0x0437 (incorrect -> 0xdeae), seq 2306393024, win 29200, options [mss 1460,sackOK,TS val 1825541 ecr 0,nop,wscale 7], length 0 0x: 0291 fb46 9e43 0253 71ef 2f2a 0800 4500 0x0010: 003c 18c5 4000 4006 1def 0101 0103 0101 0x0020: 0104 cac8 0016 8978 c3c0 a002 0x0030: 7210 0437 0204 05b4 0402 080a 001b 0x0040: db05 0103 0307 packet egressing from linux bridge to vm-2 06:41:58.627130 02:53:71:ef:2f:2a > 02:91:fb:46:9e:43, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 6341, offset 0, flags [DF], proto TCP (6), length 60) 1.1.1.3.51912 > 1.1.1.4.22: Flags [S], cksum 0x0437 (incorrect -> 0xdeae), seq 2306393024, win 29200, options [mss 1460,sackOK,TS val 1825541 ecr 0,nop,wscale 7], length 0 0x: 0291 fb46 9e43 0253 71ef 2f2a 0800 4500 0x0010: 003c 18c5 4000 4006 1def 0101 0103 0101 0x0020: 0104 cac8 0016 8978 c3c0 a002 0x0030: 7210 0437 0204 05b4 0402 080a 001b 0x0040: db05 0103 0307 Regards, Prabhjot Quoting Florin Coras : Hi Prabhjot, From your description, I suspect it may be a linux tcp checksum offload issue. Could you try disabling it for all interfaces with: ethtool --offload rx off tx off Hope this helps, Florin On Aug 28, 2017, at 10:56 AM, Prabhjot Singh Sethi wrote: We have been trying to use VPP as a L2/L3 forwarding data path between VMs. where we have connected tap interfaces from VM to VPP using "create host-interface name " after adding both the interfaces to same bridge-domain (bd_id = 2), we are able to ping from one VM to another (both in same subnet). However when we try to do ssh we observe that all the tcp packets are transmitted/forwarded by VPP to the other end but they are dropped as "bad segments received". Can anyone help with what could be wrong here, are we missing any other required config? Note:- same VMs works well when connected to linux bridge root@vm-2:~# netstat -s | grep -A11 -i tcp: Tcp: 33 active connections openings 4 passive connection openings 0 failed connection attempts 1 connection resets received 1 connections established 885 segments received 650 segments send out 8 segments retransmited 19 bad segments received. Keeps increasing for every tcp packet 2 resets sent InCsumErrors: 19 Keeps increasing for every tcp packet root@vm-2:~# Regards, Prabhjot ___ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev ___ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev
Re: [vpp-dev] Issue forwarding TCP packets
Hi Prabhjot, >From your description, I suspect it may be a linux tcp checksum offload issue. >Could you try disabling it for all interfaces with: ethtool --offload rx off tx off Hope this helps, Florin > On Aug 28, 2017, at 10:56 AM, Prabhjot Singh Sethi > wrote: > > We have been trying to use VPP as a L2/L3 forwarding data path between VMs. > > where we have connected tap interfaces from VM to VPP using "create > host-interface name " > after adding both the interfaces to same bridge-domain (bd_id = 2), we are > able to ping from > one VM to another (both in same subnet). However when we try to do ssh we > observe that all the > tcp packets are transmitted/forwarded by VPP to the other end but they are > dropped as > "bad segments received". > > Can anyone help with what could be wrong here, are we missing any other > required config? > > Note:- same VMs works well when connected to linux bridge > > root@vm-2:~# netstat -s | grep -A11 -i tcp: > Tcp: > 33 active connections openings > 4 passive connection openings > 0 failed connection attempts > 1 connection resets received > 1 connections established > 885 segments received > 650 segments send out > 8 segments retransmited > 19 bad segments received. Keeps increasing for > every tcp packet > 2 resets sent > InCsumErrors: 19 Keeps increasing for > every tcp packet > root@vm-2:~# > > Regards, > Prabhjot > ___ > vpp-dev mailing list > vpp-dev@lists.fd.io > https://lists.fd.io/mailman/listinfo/vpp-dev ___ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev
[vpp-dev] Issue forwarding TCP packets
We have been trying to use VPP as a L2/L3 forwarding data path between VMs. where we have connected tap interfaces from VM to VPP using "create host-interface name " after adding both the interfaces to same bridge-domain (bd_id = 2), we are able to ping from one VM to another (both in same subnet). However when we try to do ssh we observe that all the tcp packets are transmitted/forwarded by VPP to the other end but they are dropped as "bad segments received". Can anyone help with what could be wrong here, are we missing any other required config? Note:- same VMs works well when connected to linux bridge root@vm-2:~# netstat -s | grep -A11 -i tcp: Tcp: 33 active connections openings 4 passive connection openings 0 failed connection attempts 1 connection resets received 1 connections established 885 segments received 650 segments send out 8 segments retransmited 19 bad segments received. Keeps increasing for every tcp packet 2 resets sent InCsumErrors: 19 Keeps increasing for every tcp packet root@vm-2:~# Regards, Prabhjot ___ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev
