Hello, As of today, with i40e-2.22.18 the problem with passing traffic over VLAN's is still there. What can I do to further investigate the issue?
-- Andrey On Wed, 4 Jan 2023 at 17:36, Andrey Kulikov <amde...@gmail.com> wrote: > Hello, > > Thanks for the prompt reply! > > > Are you loading the 8021q.ko module? > > Sure. lsmod shows it. > > > Once the network setup is done, what does 'ip link' show? > > Nothing suspicious to me... > Full output: > > root@ng-bond1:~# ip link > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode > DEFAULT group default qlen 1000 > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > 2: enp2s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP > mode DEFAULT group default qlen 1000 > link/ether 00:90:0b:b7:46:c0 brd ff:ff:ff:ff:ff:ff > 3: enp2s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state > DOWN mode DEFAULT group default qlen 1000 > link/ether 00:90:0b:b7:46:c1 brd ff:ff:ff:ff:ff:ff > 4: enp2s0f2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state > DOWN mode DEFAULT group default qlen 1000 > link/ether 00:90:0b:b7:46:c2 brd ff:ff:ff:ff:ff:ff > 5: enp132s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state > UP mode DEFAULT group default qlen 1000 > link/ether 00:90:0b:b6:10:7d brd ff:ff:ff:ff:ff:ff > 6: enp2s0f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP > mode DEFAULT group default qlen 1000 > link/ether 00:90:0b:b7:46:c3 brd ff:ff:ff:ff:ff:ff > 7: enp8s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state > DOWN mode DEFAULT group default qlen 1000 > link/ether 00:90:0b:b5:da:7c brd ff:ff:ff:ff:ff:ff > 8: enp9s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state > DOWN mode DEFAULT group default qlen 1000 > link/ether 00:90:0b:b5:da:7d brd ff:ff:ff:ff:ff:ff > 9: enp132s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state > UP mode DEFAULT group default qlen 1000 > link/ether 00:90:0b:b6:10:7e brd ff:ff:ff:ff:ff:ff > 10: enp132s0f2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq > state DOWN mode DEFAULT group default qlen 1000 > link/ether 00:90:0b:b6:10:7f brd ff:ff:ff:ff:ff:ff > 11: enp132s0f3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq > state DOWN mode DEFAULT group default qlen 1000 > link/ether 00:90:0b:b6:10:80 brd ff:ff:ff:ff:ff:ff > 12: enp132s0f0.545@enp132s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 > qdisc noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:90:0b:b6:10:7d brd ff:ff:ff:ff:ff:ff > 13: tunhub0: <POINTOPOINT,MULTICAST,NOARP> mtu 1500 qdisc noop state DOWN > mode DEFAULT group default qlen 500 > link/none > root@ng-bond1:~# > > > > Did test of ping over VLAN with some available versions of i40e driver. > > Setup is the same as before: > 1. Two 'identical twins' machines (ng-bond1 and ng-bond2). > 2. All manipulations listed below is on ng-bond1 machine. > 3. Before all manipulations both machines were rebooted. > 4. Prior to experiments on ng-bond2 tx-vlan-offload was switched off on > interface, connected to enp132s0f0 on ng-bond1 (see below.) > > > Results are: > > i40e-debian - Works (5.10.0-20-amd64) > i40e-2.18.9 - Works > i40e-2.19.3 - NOT work > i40e-2.20.12 - NOT work > i40e-2.21.12 - NOT work > i40e-2.22.8 - NOT work > > "Works" here means: > ping over VLAN is successful. > > "NOT work" here means: > ping over VLAN is NOT successful: > root@ng-bond1:~# ping -c 2 192.168.44.2 > PING 192.168.44.2 (192.168.44.2) 56(84) bytes of data. > From 192.168.44.1 icmp_seq=1 Destination Host Unreachable > From 192.168.44.1 icmp_seq=2 Destination Host Unreachable > > --- 192.168.44.2 ping statistics --- > 2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1028ms > pipe 2 > > while tcpdump on other side of the cable (ng-bond2) shows nothing. > root@ng-bond2:~# tcpdump -i enp132s0f0 -ne -vvvvv > (except some IPv6 stuff on HW interface) > > Previously I've mentioned that both tx-vlan-offload AND rx-vlan-offload > should be switched off in order to get ping over VLAN work again. > It appears that switching tx-vlan-offload is enough, and after switching > it off ping over VLAN works again. > > root@ng-bond1:~# ethtool --features enp132s0f0 tx-vlan-offload off > root@ng-bond1:~# ping -c 2 192.168.44.2 > PING 192.168.44.2 (192.168.44.2) 56(84) bytes of data. > 64 bytes from 192.168.44.2: icmp_seq=1 ttl=64 time=0.188 ms > 64 bytes from 192.168.44.2: icmp_seq=2 ttl=64 time=0.058 ms > > --- 192.168.44.2 ping statistics --- > 2 packets transmitted, 2 received, 0% packet loss, time 1024ms > rtt min/avg/max/mdev = 0.058/0.123/0.188/0.065 ms > > > > > When pinging, it would be useful to see what ethtool -S shows as changing > > arp -an output would be useful as well. > > Collected requested information with these commands (on ng-bond1): > ===== > ip neigh show > arp_before_ping.txt > nstat --reset > ethtool --statistics enp2s0f0 > ethtool_stats_before_ping.txt > ip -s link > ip_link_stat_before_ping.txt > > ping -c 2 192.168.44.2 > > ethtool --statistics enp2s0f0 > ethtool_stats_after_ping.txt > ip -s link > ip_link_stat_after_ping.txt > nstat > nstat_after_ping.txt > ip neigh show > arp_after_ping.txt > ===== > > ARP table after failed ping: > 192.168.44.2 dev enp132s0f0.545 FAILED > 192.168.85.129 dev enp2s0f0 lladdr d0:17:c2:92:eb:38 REACHABLE > 192.168.85.1 dev enp2s0f0 lladdr 00:00:5e:00:01:55 STALE > > Relevan diff in ethtool statistic (see full stats in files attached) > root@ng-bond1:~# diff -uN ethtool_stats_before_ping.txt > ethtool_stats_after_ping.txt > --- ethtool_stats_before_ping.txt > +++ ethtool_stats_after_ping.txt > @@ -1,9 +1,9 @@ > NIC statistics: > - rx_packets: 1708 > - tx_packets: 1822 > - rx_bytes: 406856 > - tx_bytes: 787992 > - rx_broadcast: 664 > + rx_packets: 1727 > + tx_packets: 1855 > + rx_bytes: 409662 > + tx_bytes: 801682 > + rx_broadcast: 666 > tx_broadcast: 3 > rx_multicast: 0 > tx_multicast: 12 > @@ -23,13 +23,13 @@ > rx_long_length_errors: 0 > rx_short_length_errors: 0 > rx_align_errors: 0 > - tx_tcp_seg_good: 180 > + tx_tcp_seg_good: 183 > tx_tcp_seg_failed: 0 > rx_flow_control_xon: 0 > rx_flow_control_xoff: 0 > tx_flow_control_xon: 0 > tx_flow_control_xoff: 0 > - rx_long_byte_count: 406856 > + rx_long_byte_count: 409662 > tx_dma_out_of_sync: 0 > tx_smbus: 0 > rx_smbus: 0 > @@ -71,11 +71,11 @@ > tx_queue_6_packets: 2 > tx_queue_6_bytes: 180 > tx_queue_6_restart: 0 > - tx_queue_7_packets: 1788 > - tx_queue_7_bytes: 774798 > + tx_queue_7_packets: 1821 > + tx_queue_7_bytes: 788356 > tx_queue_7_restart: 0 > - rx_queue_0_packets: 639 > - rx_queue_0_bytes: 312904 > + rx_queue_0_packets: 641 > + rx_queue_0_bytes: 314460 > rx_queue_0_drops: 0 > rx_queue_0_csum_err: 0 > rx_queue_0_alloc_failed: 0 > @@ -94,8 +94,8 @@ > rx_queue_3_drops: 0 > rx_queue_3_csum_err: 0 > rx_queue_3_alloc_failed: 0 > - rx_queue_4_packets: 1030 > - rx_queue_4_bytes: 80195 > + rx_queue_4_packets: 1048 > + rx_queue_4_bytes: 81429 > rx_queue_4_drops: 0 > rx_queue_4_csum_err: 0 > rx_queue_4_alloc_failed: 0 > > > To summarize: > As I've mentioned above, after switching tx-vlan-offload off everything > (ping over VLAN) starts to work like a charm, even on latest i40e driver > available: > # ethtool --features enp132s0f0 tx-vlan-offload off > # ping -c 2 192.168.44.2 > PING 192.168.44.2 (192.168.44.2) 56(84) bytes of data. > 64 bytes from 192.168.44.2: icmp_seq=1 ttl=64 time=0.188 ms > > Last i40e driver where it was not required was i40e-2.18.9 > Starting from i40e-2.19.3 tx-vlan-offload MUST be switched off in order to > get traffic through VLAN interfaces. > On bare HW interfaces everything works on any i40e driver versions. > > How can I help to further investigate this issue? > > -- > Cheers, > Andrey > > On Wed, 4 Jan 2023 at 03:46, Jesse Brandeburg <jesse.brandeb...@intel.com> > wrote: > >> On 12/30/2022 11:15 AM, Andrey Kulikov wrote: >> > Hello, >> > >> > I've got an Intel Fortville XL710-based Ethernet controller with 4 x >> 10GbE >> > SFP+ ports. >> > Platform is based on Intel Xeon CPU E5-2697 v4 >> > Platform running Debian 11.6, kernel 5.10.0 >> > # uname -a >> > Linux 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13) x86_64 >> GNU/Linux >> > >> > Current i40e driver: 2.22.8 (out-of tree, self-built from sources). >> >> Are you loading the 8021q.ko module? >> >> > >> > Relevant fragment from my /etc/network/interfaces on one side: >> > >> > auto enp132s0f0 >> > iface enp132s0f0 inet static >> > address 192.168.33.2 >> > netmask 255.255.255.0 >> > mtu 1500 >> > >> > auto enp132s0f0.545 >> > iface enp132s0f0.545 inet static >> > address 192.168.44.2 >> > netmask 255.255.255.0 >> > mtu 1500 >> >> Once the network setup is done, what does 'ip link' show? >> >> > Does it have something to do with the i40e driver? >> > Is any further information required? >> >> When pinging, it would be useful to see what ethtool -S shows as >> changing, like >> ethtool -S enp132s0f0 >> ping -c2 192.168.33.1 >> <fail> >> ethtool -S enp132s0f0 >> >> arp -an >> output would be useful as well. >> >> >> _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel Ethernet, visit https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet
