Hello On Thu, Mar 3, 2022 at 6:23 PM Harold Huang <baymaxhu...@gmail.com> wrote:
> On Fri, Mar 4, 2022 at 3:05 AM Amir Alimohammadifar > <amir.af...@gmail.com> wrote: > > > > Hello > > > > On Wed, Mar 2, 2022 at 6:45 PM Harold Huang <baymaxhu...@gmail.com> > wrote: > >> > >> On Thu, Mar 3, 2022 at 3:33 AM Amir Alimohammadifar > >> <amir.af...@gmail.com> wrote: > >> > > >> > Hello, > >> > > >> > I am having troubles with hardware vxlan offloading when the > tx_checksumming is turned on. > >> > Here is my environment: > >> > > >> > GuestVM1 ---> VM1 (running in ESXi 6.7 Hypervisor1) <--VxLAN > tunnel--> VM2 (running in ESXi 6.7 Hypervisor2) ---> GuestVM2 > >> > >> Do you use OVS kernel datapath to create VxLAN tunnels and forward > traffic? > > > > I cannot use the configure with the "--with-linux=/lib/modules/$(uname > -r)/build" option for kernels above > 5.8. And I think the vxlan offload is > not supporded in kernels with versions below < 5.8. > > for example, using Ubuntu Focal with Kernel 5.4, the options for vxlan > offload are disabled and turned off > > I have a server with kernel version 4.18. And the Vxlan offload is > supported. Maybe you should check more about your hypervisor and > vSwitch. > Thank you so much for your help. Yes I think it is the limitation of the Hypervisor. I am using ESXi 6.7, and 7.0. And in both of them there is the same issue. The NICs I am using are: - Intel(R) Ethernet Controller X710/X557-AT 10GBASE-T - Mellanox Technologies MT27710 Family [ConnectX-4 Lx] In both cases, when using ESXi 6.7 and ESXi 7.0 with kernel 5.15, when I turn on the SR-IOV in the hypervisor, in the VM1 and VM2 I see the driver as the following: For Mellanox: - driver: mlx5_core version: 5.15.0-18-generic For Intel: - driver: iavf version: 5.15.0-18-generic In the scenario above I have: - tx-udp_tnl-segmentation: off [fixed] - tx-udp_tnl-csum-segmentation: off [fixed] And when I turn off the SR-IOV, I have to use the VMXNET3 which uses the driver below: - driver: vmxnet3 version: 1.6.0.0-k-NAPI And with kernel 5.15 the parameters are: - tx-udp_tnl-segmentation: on - tx-udp_tnl-csum-segmentation: on In this case I face the issue that I mentioned earlier, i.e., when the tx-checksumming is turned on I cannot go more than 10Mb and when it is off not everything is done in hardware and I have high CPU usage. > > > > amir@focalvm2:~$ uname -r > > > > 5.4.0-100-generic > > > > amir@focalvm2:~$ ethtool -k ens224 | grep tx-udp > > > > tx-udp_tnl-segmentation: off [fixed] > > > > tx-udp_tnl-csum-segmentation: off [fixed] > > > > tx-udp-segmentation: off [fixed] > >> > >> > >> > >> > > >> > The VM1 and VM2 are responsible for creating openflow flows using > VxLAN tunnels to route the traffic between the GuestVMs. > >> > > >> > I have created all the required configurations to route the traffic > and everything works fine. However, when I try to enable VxLAN offloading, > using the parameters below: > >> > > >> > VM1 (transmitter): > >> > ... > >> > > >> > tx-checksumming: on > >> > > >> > tx-checksum-ipv4: off [fixed] > >> > > >> > tx-checksum-ip-generic: on > >> > > >> > tx-checksum-ipv6: off [fixed] > >> > > >> > tx-checksum-fcoe-crc: off [fixed] > >> > > >> > tx-checksum-sctp: off [fixed] > >> > > >> > ... > >> > > >> > tx-udp_tnl-segmentation: on > >> > > >> > tx-udp_tnl-csum-segmentation: on > >> > ... > >> > > >> > The throughput between GustVM1 ---> GuestVM2 is less than 5Mbps. > >> > After googling around, I turned off the tx_checksumming on the > transmitter side and everything works well. (I can see 10Gbps traffic going > through but the CPU usage is terrible) > >> > >> Could you show the CPU stats in your host and perf results? > > > > With tx-checksumming ON: > > VM2 Settings: > > > > amir@jammyvm2:~$ uname -r > > > > 5.15.0-18-generic > > > > amir@jammyvm2:~$ ethtool -k ens224 | grep "tx-udp\|tx_check" > > > > tx-checksumming: on > > > > tx-checksum-ipv4: off [fixed] > > > > tx-checksum-ip-generic: on > > > > tx-checksum-ipv6: off [fixed] > > > > tx-checksum-fcoe-crc: off [fixed] > > > > tx-checksum-sctp: off [fixed] > > > > tx-udp_tnl-segmentation: on > > > > tx-udp_tnl-csum-segmentation: on > > > > tx-udp-segmentation: off [fixed] > > > > amir@jammyvm2:~$ cat /etc/lsb-release > > > > DISTRIB_ID=Ubuntu > > > > DISTRIB_RELEASE=22.04 > > > > DISTRIB_CODENAME=jammy > > > > DISTRIB_DESCRIPTION="Ubuntu Jammy Jellyfish (development branch)" > > > > > > CPU in the VM1: > > > > Average: CPU %usr %nice %sys %iowait %irq %soft > %steal %guest %gnice %idle > > > > Average: all 0.03 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 99.97 > > > > Average: 0 0.05 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 99.95 > > > > Average: 1 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 100.00 > > > > > > Average: NODE %usr %nice %sys %iowait %irq %soft > %steal %guest %gnice %idle > > > > Average: all 0.03 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 99.97 > > > > Iperf in the GuestVM2: > > > > amir@guestvm2:~$ iperf3 -c 10.0.0.13 --bind 10.0.0.12 -V -t 20 -i 5 > > > > iperf 3.7 > > > > Linux guestvm2 5.4.0-100-generic #113-Ubuntu SMP Thu Feb 3 18:43:29 UTC > 2022 x86_64 > > > > Control connection MSS 1398 > > > > Time: Thu, 03 Mar 2022 18:57:24 GMT > > > > Connecting to host 10.0.0.13, port 5201 > > > > Cookie: 2dmnehmxcmeodeel5tdv3ojvvjeguokvzj22 > > > > TCP MSS: 1398 (default) > > > > [ 5] local 10.0.0.12 port 45497 connected to 10.0.0.13 port 5201 > > > > Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 > seconds, 20 second test, tos 0 > > > > [ ID] Interval Transfer Bitrate Retr Cwnd > > > > [ 5] 0.00-5.00 sec 1.45 MBytes 2.43 Mbits/sec 438 2.73 KBytes > > > > [ 5] 5.00-10.00 sec 2.07 MBytes 3.47 Mbits/sec 622 2.73 KBytes > > > > [ 5] 10.00-15.00 sec 2.15 MBytes 3.60 Mbits/sec 624 2.73 KBytes > > > > [ 5] 15.00-20.00 sec 2.07 MBytes 3.47 Mbits/sec 624 2.73 KBytes > > > > - - - - - - - - - - - - - - - - - - - - - - - - - > > > > Test Complete. Summary Results: > > > > [ ID] Interval Transfer Bitrate Retr > > > > [ 5] 0.00-20.00 sec 7.73 MBytes 3.24 Mbits/sec 2308 > sender > > > > [ 5] 0.00-20.00 sec 7.62 MBytes 3.20 Mbits/sec > receiver > > > > CPU Utilization: local/sender 0.0% (0.0%u/0.0%s), remote/receiver 0.2% > (0.0%u/0.1%s) > > > > snd_tcp_congestion cubic > > > > rcv_tcp_congestion cubic > > > > > > iperf Done. > > > > NOW with tx-checksumming off: > > > > amir@jammyvm2:~$ ethtool -k ens224 | grep "tx-udp\|tx-check" > > > > tx-checksumming: off > > > > tx-checksum-ipv4: off [fixed] > > > > tx-checksum-ip-generic: off > > > > tx-checksum-ipv6: off [fixed] > > > > tx-checksum-fcoe-crc: off [fixed] > > > > tx-checksum-sctp: off [fixed] > > > > tx-udp_tnl-segmentation: on > > > > tx-udp_tnl-csum-segmentation: on > > > > tx-udp-segmentation: off [fixed] > > > > CPU in the VM1: > > > > Average: CPU %usr %nice %sys %iowait %irq %soft > %steal %guest %gnice %idle > > > > Average: all 0.07 0.00 0.07 0.00 0.00 46.76 > 0.00 0.00 0.00 53.09 > > > > Average: 0 0.00 0.00 0.05 0.00 0.00 93.27 > 0.00 0.00 0.00 6.68 > > > > Average: 1 0.15 0.00 0.10 0.00 0.00 0.25 > 0.00 0.00 0.00 99.50 > > > > > > Average: NODE %usr %nice %sys %iowait %irq %soft > %steal %guest %gnice %idle > > > > Average: all 0.07 0.00 0.07 0.00 0.00 46.76 > 0.00 0.00 0.00 53.09 > > > > Iperf in the GuestVM2: > > > > amir@guestvm2:~$ iperf3 -c 10.0.0.13 --bind 10.0.0.12 -V -t 20 -i 5 > > > > iperf 3.7 > > > > Linux guestvm2 5.4.0-100-generic #113-Ubuntu SMP Thu Feb 3 18:43:29 UTC > 2022 x86_64 > > > > Control connection MSS 1398 > > > > Time: Thu, 03 Mar 2022 19:03:10 GMT > > > > Connecting to host 10.0.0.13, port 5201 > > > > Cookie: pvlmogudjq4vcxq5xvwxwmfuaq3upvf5xtba > > > > TCP MSS: 1398 (default) > > > > [ 5] local 10.0.0.12 port 59369 connected to 10.0.0.13 port 5201 > > > > Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 > seconds, 20 second test, tos 0 > > > > [ ID] Interval Transfer Bitrate Retr Cwnd > > > > [ 5] 0.00-5.00 sec 4.33 GBytes 7.44 Gbits/sec 89 2.12 MBytes > > > > [ 5] 5.00-10.00 sec 5.35 GBytes 9.19 Gbits/sec 152 1.67 MBytes > > > > [ 5] 10.00-15.00 sec 5.37 GBytes 9.23 Gbits/sec 228 2.00 MBytes > > > > [ 5] 15.00-20.00 sec 5.34 GBytes 9.18 Gbits/sec 227 1.55 MBytes > > > > - - - - - - - - - - - - - - - - - - - - - - - - - > > > > Test Complete. Summary Results: > > > > [ ID] Interval Transfer Bitrate Retr > > > > [ 5] 0.00-20.00 sec 20.4 GBytes 8.76 Gbits/sec 696 > sender > > > > [ 5] 0.00-20.00 sec 20.4 GBytes 8.76 Gbits/sec > receiver > > > > CPU Utilization: local/sender 15.3% (0.2%u/15.1%s), remote/receiver 2.3% > (0.1%u/2.2%s) > > > > snd_tcp_congestion cubic > > > > rcv_tcp_congestion cubic > > > > > > iperf Done. > > > > > >> > >> > >> > > >> > VM1 (transmitter): > >> > > >> > ... > >> > > >> > tx-checksumming: off > >> > > >> > tx-checksum-ipv4: off [fixed] > >> > > >> > tx-checksum-ip-generic: off > >> > > >> > tx-checksum-ipv6: off [fixed] > >> > > >> > tx-checksum-fcoe-crc: off [fixed] > >> > > >> > tx-checksum-sctp: off [fixed] > >> > > >> > ... > >> > > >> > tx-udp_tnl-segmentation: on > >> > > >> > tx-udp_tnl-csum-segmentation: on > >> > > >> > ... > >> > > >> > > >> > This issue doesn't exist when I use the Linux-bridge for connecting > the VM1 and VM2. > >> > > >> > I was wondering if there is an issue with openvswitch when I keep the > tx_checsumming on? > >> > > >> > And I am using the following NICs which are capable of VxLAN > offloading: > >> > Mellanox Technologies MT27710 Family [ConnectX-4 Lx] > >> > > >> > Driver Info: > >> > Bus Info: 0000:3b:00:0 > >> > Driver: nmlx5_core > >> > Firmware Version: 14.23.1020 > >> > Version: 4.15.10.3 > >> > > >> > > >> > Thank you, > >> > Amir > >> > > >> > _______________________________________________ > >> > discuss mailing list > >> > disc...@openvswitch.org > >> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss >
_______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
