I checked with our performance team and I think the only thing we can see is the possible bottleneck with your link partner. We don't do any interoperability testing with Netgear, and are unaware if they have any equipment that isn't consumer-grade.
I would suggest that you follow up with Ubuntu. They will contact us if they need any further help with the issue. Todd Fujinaka Software Application Engineer Data Center Group Intel Corporation todd.fujin...@intel.com -----Original Message----- From: Pekka Pietikäinen <pe...@pietikainen.org> Sent: Thursday, February 4, 2021 2:15 AM To: e1000-devel@lists.sourceforge.net Subject: [E1000-devel] Fake Tx hangs with ixgbe 5.10.2 Hi, We were seeing a lot of Tx hangs (once a day in lab, much more in production) with Ubuntu 18.04.5 built-in driver (5.0.0-k and 5.1.0-k), causing link resets etc.. Trying to isolate the problem a bit more we're now trying the latest out-of-tree driver. Now it's Feb 03 22:46:19 x kernel: NETDEV WATCHDOG: enp13s0 (ixgbe): transmit queue 3 timed out Feb 03 22:46:19 x kernel: ixgbe 0000:0d:00.0 enp13s0: Fake Tx hang detected with timeout of 5 seconds Feb 04 01:56:01 x kernel: ixgbe 0000:0d:00.0 enp13s0: Fake Tx hang detected with timeout of 10 seconds Feb 04 15:13:21 x kernel: ixgbe 0000:0d:00.0 enp13s0: Fake Tx hang detected with timeout of 20 seconds "Fake Tx Hang", but traffic still stops for quite a while. Setup is ixgbe 0000:0d:00.0: enabling device (0000 -> 0002) ixgbe 0000:0d:00.0 0000:0d:00.0 (uninitialized): ixgbe_check_options: FCoE Offload feature enabled ixgbe 0000:0d:00.0: Multiqueue Enabled: Rx Queue count = 8, Tx Queue count = 8 XDP Queue count = 0 ixgbe 0000:0d:00.0: 32.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x8 link at 0000:00:01.0 (capable of 63.008 Gb/s with 8 GT/s x8 link) ixgbe 0000:0d:00.0 eth0: MAC: 2, PHY: 20, SFP+: 5, PBA No: E68787-011 ixgbe 0000:0d:00.0 eth0: Enabled Features: RxQ: 8 TxQ: 8 FdirHash ixgbe 0000:0d:00.0 eth0: Intel(R) 10 Gigabit Network Connection talking via 10Gbps fibre to Netgear switch to 20 clients (1GbE over copper) Disabling irqbalance and using set_irq_affinity (-x local) results in /proc/interrupts 132: 6932 1128 1977 2921 7857 7960 4392 3754 IR-PCI-MSI 520192-edge enp0s31f6 133: 264128 280513 208862 114758 608511 839 90376 187222 IR-PCI-MSI 6815744-edge enp13s0-TxRx-0 134: 2844471 365014 105711 1009702 423599 688773 0 589578 IR-PCI-MSI 6815745-edge enp13s0-TxRx-1 135: 95567 140332 43052 46623 86704 57976 135573 167195 IR-PCI-MSI 6815746-edge enp13s0-TxRx-2 136: 318887 197761 625025 193550 101415 102114 55755 291272 IR-PCI-MSI 6815747-edge enp13s0-TxRx-3 137: 491537 214000 158900 244269 266167 97220 25460 289919 IR-PCI-MSI 6815748-edge enp13s0-TxRx-4 138: 65767 248812 115232 238502 80103 46189 50782 81833 IR-PCI-MSI 6815749-edge enp13s0-TxRx-5 139: 207157 260936 48883 47421 216735 97253 92818 89079 IR-PCI-MSI 6815750-edge enp13s0-TxRx-6 140: 21512 424646 5111 390019 441436 4728 291370 278387 IR-PCI-MSI 6815751-edge enp13s0-TxRx-7 141: 0 0 0 3 0 0 0 0 IR-PCI-MSI 6815752-edge enp13s0 TxRx-1 on CPU0 seems a bit unbalanced, but otherwise fine? ethtool would suggest it's not flow control related (which would explain things?), now experimenting with tuning interrupt moderation / ring size / avoiding CPU0 . Is there anything else to try? NIC statistics: rx_packets: 188092169 tx_packets: 59581798 rx_bytes: 263463989424 tx_bytes: 25727566814 rx_errors: 0 tx_errors: 0 rx_dropped: 0 tx_dropped: 0 multicast: 1237 collisions: 0 rx_over_errors: 0 rx_crc_errors: 0 rx_frame_errors: 0 rx_fifo_errors: 0 rx_missed_errors: 0 tx_aborted_errors: 0 tx_carrier_errors: 0 tx_fifo_errors: 0 tx_heartbeat_errors: 0 rx_pkts_nic: 188092169 tx_pkts_nic: 59581798 rx_bytes_nic: 264216358100 tx_bytes_nic: 25965914840 lsc_int: 3 tx_busy: 0 non_eop_descs: 0 broadcast: 259 rx_no_buffer_count: 0 tx_timeout_count: 0 tx_restart_queue: 4 rx_length_errors: 0 rx_long_length_errors: 0 rx_short_length_errors: 0 tx_flow_control_xon: 0 rx_flow_control_xon: 0 tx_flow_control_xoff: 0 rx_flow_control_xoff: 0 rx_csum_offload_errors: 0 alloc_rx_page: 40526166 alloc_rx_page_failed: 0 alloc_rx_buff_failed: 0 rx_no_dma_resources: 0 hw_rsc_aggregated: 0 hw_rsc_flushed: 0 fdir_match: 188027415 fdir_miss: 178920 fdir_overflow: 0 fcoe_bad_fccrc: 0 fcoe_last_errors: 0 rx_fcoe_dropped: 0 rx_fcoe_packets: 0 rx_fcoe_dwords: 0 fcoe_noddp: 0 fcoe_noddp_ext_buff: 0 tx_fcoe_packets: 0 tx_fcoe_dwords: 0 os2bmc_rx_by_bmc: 0 os2bmc_tx_by_bmc: 0 os2bmc_tx_by_host: 0 os2bmc_rx_by_host: 0 tx_hwtstamp_timeouts: 0 tx_hwtstamp_skipped: 0 rx_hwtstamp_cleared: 0 tx_queue_0_packets: 2551919 tx_queue_0_bytes: 303310258 tx_queue_1_packets: 34799436 tx_queue_1_bytes: 2365301599 tx_queue_2_packets: 726053 tx_queue_2_bytes: 101796295 tx_queue_3_packets: 5109959 tx_queue_3_bytes: 4885506128 tx_queue_4_packets: 4655587 tx_queue_4_bytes: 4755028677 tx_queue_5_packets: 2533314 tx_queue_5_bytes: 2516349075 tx_queue_6_packets: 4109721 tx_queue_6_bytes: 5511688758 tx_queue_7_packets: 5095809 tx_queue_7_bytes: 5288586024 tx_queue_8_packets: 0 tx_queue_8_bytes: 0 tx_queue_9_packets: 0 tx_queue_9_bytes: 0 tx_queue_10_packets: 0 tx_queue_10_bytes: 0 tx_queue_11_packets: 0 tx_queue_11_bytes: 0 tx_queue_12_packets: 0 tx_queue_12_bytes: 0 tx_queue_13_packets: 0 tx_queue_13_bytes: 0 tx_queue_14_packets: 0 tx_queue_14_bytes: 0 tx_queue_15_packets: 0 tx_queue_15_bytes: 0 tx_queue_16_packets: 0 tx_queue_16_bytes: 0 tx_queue_17_packets: 0 tx_queue_17_bytes: 0 tx_queue_18_packets: 0 tx_queue_18_bytes: 0 tx_queue_19_packets: 0 tx_queue_19_bytes: 0 tx_queue_20_packets: 0 tx_queue_20_bytes: 0 tx_queue_21_packets: 0 tx_queue_21_bytes: 0 tx_queue_22_packets: 0 tx_queue_22_bytes: 0 tx_queue_23_packets: 0 tx_queue_23_bytes: 0 tx_queue_24_packets: 0 tx_queue_24_bytes: 0 tx_queue_25_packets: 0 tx_queue_25_bytes: 0 tx_queue_26_packets: 0 tx_queue_26_bytes: 0 tx_queue_27_packets: 0 tx_queue_27_bytes: 0 tx_queue_28_packets: 0 tx_queue_28_bytes: 0 tx_queue_29_packets: 0 tx_queue_29_bytes: 0 tx_queue_30_packets: 0 tx_queue_30_bytes: 0 tx_queue_31_packets: 0 tx_queue_31_bytes: 0 tx_queue_32_packets: 0 tx_queue_32_bytes: 0 tx_queue_33_packets: 0 tx_queue_33_bytes: 0 tx_queue_34_packets: 0 tx_queue_34_bytes: 0 tx_queue_35_packets: 0 tx_queue_35_bytes: 0 tx_queue_36_packets: 0 tx_queue_36_bytes: 0 tx_queue_37_packets: 0 tx_queue_37_bytes: 0 tx_queue_38_packets: 0 tx_queue_38_bytes: 0 tx_queue_39_packets: 0 tx_queue_39_bytes: 0 tx_queue_40_packets: 0 tx_queue_40_bytes: 0 tx_queue_41_packets: 0 tx_queue_41_bytes: 0 tx_queue_42_packets: 0 tx_queue_42_bytes: 0 tx_queue_43_packets: 0 tx_queue_43_bytes: 0 tx_queue_44_packets: 0 tx_queue_44_bytes: 0 tx_queue_45_packets: 0 tx_queue_45_bytes: 0 tx_queue_46_packets: 0 tx_queue_46_bytes: 0 tx_queue_47_packets: 0 tx_queue_47_bytes: 0 tx_queue_48_packets: 0 tx_queue_48_bytes: 0 tx_queue_49_packets: 0 tx_queue_49_bytes: 0 tx_queue_50_packets: 0 tx_queue_50_bytes: 0 tx_queue_51_packets: 0 tx_queue_51_bytes: 0 tx_queue_52_packets: 0 tx_queue_52_bytes: 0 tx_queue_53_packets: 0 tx_queue_53_bytes: 0 tx_queue_54_packets: 0 tx_queue_54_bytes: 0 tx_queue_55_packets: 0 tx_queue_55_bytes: 0 tx_queue_56_packets: 0 tx_queue_56_bytes: 0 tx_queue_57_packets: 0 tx_queue_57_bytes: 0 tx_queue_58_packets: 0 tx_queue_58_bytes: 0 tx_queue_59_packets: 0 tx_queue_59_bytes: 0 tx_queue_60_packets: 0 tx_queue_60_bytes: 0 tx_queue_61_packets: 0 tx_queue_61_bytes: 0 tx_queue_62_packets: 0 tx_queue_62_bytes: 0 tx_queue_63_packets: 0 tx_queue_63_bytes: 0 tx_queue_64_packets: 0 tx_queue_64_bytes: 0 tx_queue_65_packets: 0 tx_queue_65_bytes: 0 tx_queue_66_packets: 0 tx_queue_66_bytes: 0 tx_queue_67_packets: 0 tx_queue_67_bytes: 0 tx_queue_68_packets: 0 tx_queue_68_bytes: 0 tx_queue_69_packets: 0 tx_queue_69_bytes: 0 tx_queue_70_packets: 0 tx_queue_70_bytes: 0 rx_queue_0_packets: 11217199 rx_queue_0_bytes: 16124868046 rx_queue_1_packets: 157176240 rx_queue_1_bytes: 227080033991 rx_queue_2_packets: 1501661 rx_queue_2_bytes: 1972184904 rx_queue_3_packets: 5075867 rx_queue_3_bytes: 5607058971 rx_queue_4_packets: 5326935 rx_queue_4_bytes: 6020504920 rx_queue_5_packets: 1606940 rx_queue_5_bytes: 1282193306 rx_queue_6_packets: 2148621 rx_queue_6_bytes: 1388353208 rx_queue_7_packets: 4038706 rx_queue_7_bytes: 3988792078 rx_queue_8_packets: 0 rx_queue_8_bytes: 0 rx_queue_9_packets: 0 rx_queue_9_bytes: 0 rx_queue_10_packets: 0 rx_queue_10_bytes: 0 rx_queue_11_packets: 0 rx_queue_11_bytes: 0 rx_queue_12_packets: 0 rx_queue_12_bytes: 0 rx_queue_13_packets: 0 rx_queue_13_bytes: 0 rx_queue_14_packets: 0 rx_queue_14_bytes: 0 rx_queue_15_packets: 0 rx_queue_15_bytes: 0 rx_queue_16_packets: 0 rx_queue_16_bytes: 0 rx_queue_17_packets: 0 rx_queue_17_bytes: 0 rx_queue_18_packets: 0 rx_queue_18_bytes: 0 rx_queue_19_packets: 0 rx_queue_19_bytes: 0 rx_queue_20_packets: 0 rx_queue_20_bytes: 0 rx_queue_21_packets: 0 rx_queue_21_bytes: 0 rx_queue_22_packets: 0 rx_queue_22_bytes: 0 rx_queue_23_packets: 0 rx_queue_23_bytes: 0 rx_queue_24_packets: 0 rx_queue_24_bytes: 0 rx_queue_25_packets: 0 rx_queue_25_bytes: 0 rx_queue_26_packets: 0 rx_queue_26_bytes: 0 rx_queue_27_packets: 0 rx_queue_27_bytes: 0 rx_queue_28_packets: 0 rx_queue_28_bytes: 0 rx_queue_29_packets: 0 rx_queue_29_bytes: 0 rx_queue_30_packets: 0 rx_queue_30_bytes: 0 rx_queue_31_packets: 0 rx_queue_31_bytes: 0 rx_queue_32_packets: 0 rx_queue_32_bytes: 0 rx_queue_33_packets: 0 rx_queue_33_bytes: 0 rx_queue_34_packets: 0 rx_queue_34_bytes: 0 rx_queue_35_packets: 0 rx_queue_35_bytes: 0 rx_queue_36_packets: 0 rx_queue_36_bytes: 0 rx_queue_37_packets: 0 rx_queue_37_bytes: 0 rx_queue_38_packets: 0 rx_queue_38_bytes: 0 rx_queue_39_packets: 0 rx_queue_39_bytes: 0 rx_queue_40_packets: 0 rx_queue_40_bytes: 0 rx_queue_41_packets: 0 rx_queue_41_bytes: 0 rx_queue_42_packets: 0 rx_queue_42_bytes: 0 rx_queue_43_packets: 0 rx_queue_43_bytes: 0 rx_queue_44_packets: 0 rx_queue_44_bytes: 0 rx_queue_45_packets: 0 rx_queue_45_bytes: 0 rx_queue_46_packets: 0 rx_queue_46_bytes: 0 rx_queue_47_packets: 0 rx_queue_47_bytes: 0 rx_queue_48_packets: 0 rx_queue_48_bytes: 0 rx_queue_49_packets: 0 rx_queue_49_bytes: 0 rx_queue_50_packets: 0 rx_queue_50_bytes: 0 rx_queue_51_packets: 0 rx_queue_51_bytes: 0 rx_queue_52_packets: 0 rx_queue_52_bytes: 0 rx_queue_53_packets: 0 rx_queue_53_bytes: 0 rx_queue_54_packets: 0 rx_queue_54_bytes: 0 rx_queue_55_packets: 0 rx_queue_55_bytes: 0 rx_queue_56_packets: 0 rx_queue_56_bytes: 0 rx_queue_57_packets: 0 rx_queue_57_bytes: 0 rx_queue_58_packets: 0 rx_queue_58_bytes: 0 rx_queue_59_packets: 0 rx_queue_59_bytes: 0 rx_queue_60_packets: 0 rx_queue_60_bytes: 0 rx_queue_61_packets: 0 rx_queue_61_bytes: 0 rx_queue_62_packets: 0 rx_queue_62_bytes: 0 rx_queue_63_packets: 0 rx_queue_63_bytes: 0 rx_queue_64_packets: 0 rx_queue_64_bytes: 0 rx_queue_65_packets: 0 rx_queue_65_bytes: 0 rx_queue_66_packets: 0 rx_queue_66_bytes: 0 rx_queue_67_packets: 0 rx_queue_67_bytes: 0 rx_queue_68_packets: 0 rx_queue_68_bytes: 0 rx_queue_69_packets: 0 rx_queue_69_bytes: 0 rx_queue_70_packets: 0 rx_queue_70_bytes: 0 tx_pb_0_pxon: 0 tx_pb_0_pxoff: 0 tx_pb_1_pxon: 0 tx_pb_1_pxoff: 0 tx_pb_2_pxon: 0 tx_pb_2_pxoff: 0 tx_pb_3_pxon: 0 tx_pb_3_pxoff: 0 tx_pb_4_pxon: 0 tx_pb_4_pxoff: 0 tx_pb_5_pxon: 0 tx_pb_5_pxoff: 0 tx_pb_6_pxon: 0 tx_pb_6_pxoff: 0 tx_pb_7_pxon: 0 tx_pb_7_pxoff: 0 rx_pb_0_pxon: 0 rx_pb_0_pxoff: 0 rx_pb_1_pxon: 0 rx_pb_1_pxoff: 0 rx_pb_2_pxon: 0 rx_pb_2_pxoff: 0 rx_pb_3_pxon: 0 rx_pb_3_pxoff: 0 rx_pb_4_pxon: 0 rx_pb_4_pxoff: 0 rx_pb_5_pxon: 0 rx_pb_5_pxoff: 0 rx_pb_6_pxon: 0 rx_pb_6_pxoff: 0 rx_pb_7_pxon: 0 rx_pb_7_pxoff: 0 _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel Ethernet, visit https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel Ethernet, visit https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet
