Hi,
We were seeing a lot of Tx hangs (once a day in lab, much more in
production) with Ubuntu 18.04.5 built-in driver (5.0.0-k and 5.1.0-k),
causing link resets etc.. Trying to isolate the problem a bit more we're
now trying the latest out-of-tree driver. Now it's
Feb 03 22:46:19 x kernel: NETDEV WATCHDOG: enp13s0 (ixgbe): transmit
queue 3 timed out
Feb 03 22:46:19 x kernel: ixgbe 0000:0d:00.0 enp13s0: Fake Tx hang
detected with timeout of 5 seconds
Feb 04 01:56:01 x kernel: ixgbe 0000:0d:00.0 enp13s0: Fake Tx hang
detected with timeout of 10 seconds
Feb 04 15:13:21 x kernel: ixgbe 0000:0d:00.0 enp13s0: Fake Tx hang
detected with timeout of 20 seconds
"Fake Tx Hang", but traffic still stops for quite a while. Setup is
ixgbe 0000:0d:00.0: enabling device (0000 -> 0002)
ixgbe 0000:0d:00.0 0000:0d:00.0 (uninitialized): ixgbe_check_options:
FCoE Offload feature enabled
ixgbe 0000:0d:00.0: Multiqueue Enabled: Rx Queue count = 8, Tx Queue
count = 8 XDP Queue count = 0
ixgbe 0000:0d:00.0: 32.000 Gb/s available PCIe bandwidth, limited by 5
GT/s x8 link at 0000:00:01.0 (capable of 63.008 Gb/s with 8 GT/s x8 link)
ixgbe 0000:0d:00.0 eth0: MAC: 2, PHY: 20, SFP+: 5, PBA No: E68787-011
ixgbe 0000:0d:00.0 eth0: Enabled Features: RxQ: 8 TxQ: 8 FdirHash
ixgbe 0000:0d:00.0 eth0: Intel(R) 10 Gigabit Network Connection
talking via 10Gbps fibre to Netgear switch to 20 clients (1GbE over copper)
Disabling irqbalance and using set_irq_affinity (-x local) results in
/proc/interrupts
132: 6932 1128 1977 2921 7857 7960
4392 3754 IR-PCI-MSI 520192-edge enp0s31f6
133: 264128 280513 208862 114758 608511
839 90376 187222 IR-PCI-MSI 6815744-edge enp13s0-TxRx-0
134: 2844471 365014 105711 1009702 423599
688773 0 589578 IR-PCI-MSI 6815745-edge enp13s0-TxRx-1
135: 95567 140332 43052 46623 86704 57976
135573 167195 IR-PCI-MSI 6815746-edge enp13s0-TxRx-2
136: 318887 197761 625025 193550 101415
102114 55755 291272 IR-PCI-MSI 6815747-edge enp13s0-TxRx-3
137: 491537 214000 158900 244269 266167 97220
25460 289919 IR-PCI-MSI 6815748-edge enp13s0-TxRx-4
138: 65767 248812 115232 238502 80103 46189
50782 81833 IR-PCI-MSI 6815749-edge enp13s0-TxRx-5
139: 207157 260936 48883 47421 216735 97253
92818 89079 IR-PCI-MSI 6815750-edge enp13s0-TxRx-6
140: 21512 424646 5111 390019 441436 4728
291370 278387 IR-PCI-MSI 6815751-edge enp13s0-TxRx-7
141: 0 0 0 3 0 0
0 0 IR-PCI-MSI 6815752-edge enp13s0
TxRx-1 on CPU0 seems a bit unbalanced, but otherwise fine?
ethtool would suggest it's not flow control related (which would explain
things?), now experimenting with tuning interrupt moderation / ring size
/ avoiding CPU0 . Is there anything else to try?
NIC statistics:
rx_packets: 188092169
tx_packets: 59581798
rx_bytes: 263463989424
tx_bytes: 25727566814
rx_errors: 0
tx_errors: 0
rx_dropped: 0
tx_dropped: 0
multicast: 1237
collisions: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
rx_pkts_nic: 188092169
tx_pkts_nic: 59581798
rx_bytes_nic: 264216358100
tx_bytes_nic: 25965914840
lsc_int: 3
tx_busy: 0
non_eop_descs: 0
broadcast: 259
rx_no_buffer_count: 0
tx_timeout_count: 0
tx_restart_queue: 4
rx_length_errors: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
tx_flow_control_xon: 0
rx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_flow_control_xoff: 0
rx_csum_offload_errors: 0
alloc_rx_page: 40526166
alloc_rx_page_failed: 0
alloc_rx_buff_failed: 0
rx_no_dma_resources: 0
hw_rsc_aggregated: 0
hw_rsc_flushed: 0
fdir_match: 188027415
fdir_miss: 178920
fdir_overflow: 0
fcoe_bad_fccrc: 0
fcoe_last_errors: 0
rx_fcoe_dropped: 0
rx_fcoe_packets: 0
rx_fcoe_dwords: 0
fcoe_noddp: 0
fcoe_noddp_ext_buff: 0
tx_fcoe_packets: 0
tx_fcoe_dwords: 0
os2bmc_rx_by_bmc: 0
os2bmc_tx_by_bmc: 0
os2bmc_tx_by_host: 0
os2bmc_rx_by_host: 0
tx_hwtstamp_timeouts: 0
tx_hwtstamp_skipped: 0
rx_hwtstamp_cleared: 0
tx_queue_0_packets: 2551919
tx_queue_0_bytes: 303310258
tx_queue_1_packets: 34799436
tx_queue_1_bytes: 2365301599
tx_queue_2_packets: 726053
tx_queue_2_bytes: 101796295
tx_queue_3_packets: 5109959
tx_queue_3_bytes: 4885506128
tx_queue_4_packets: 4655587
tx_queue_4_bytes: 4755028677
tx_queue_5_packets: 2533314
tx_queue_5_bytes: 2516349075
tx_queue_6_packets: 4109721
tx_queue_6_bytes: 5511688758
tx_queue_7_packets: 5095809
tx_queue_7_bytes: 5288586024
tx_queue_8_packets: 0
tx_queue_8_bytes: 0
tx_queue_9_packets: 0
tx_queue_9_bytes: 0
tx_queue_10_packets: 0
tx_queue_10_bytes: 0
tx_queue_11_packets: 0
tx_queue_11_bytes: 0
tx_queue_12_packets: 0
tx_queue_12_bytes: 0
tx_queue_13_packets: 0
tx_queue_13_bytes: 0
tx_queue_14_packets: 0
tx_queue_14_bytes: 0
tx_queue_15_packets: 0
tx_queue_15_bytes: 0
tx_queue_16_packets: 0
tx_queue_16_bytes: 0
tx_queue_17_packets: 0
tx_queue_17_bytes: 0
tx_queue_18_packets: 0
tx_queue_18_bytes: 0
tx_queue_19_packets: 0
tx_queue_19_bytes: 0
tx_queue_20_packets: 0
tx_queue_20_bytes: 0
tx_queue_21_packets: 0
tx_queue_21_bytes: 0
tx_queue_22_packets: 0
tx_queue_22_bytes: 0
tx_queue_23_packets: 0
tx_queue_23_bytes: 0
tx_queue_24_packets: 0
tx_queue_24_bytes: 0
tx_queue_25_packets: 0
tx_queue_25_bytes: 0
tx_queue_26_packets: 0
tx_queue_26_bytes: 0
tx_queue_27_packets: 0
tx_queue_27_bytes: 0
tx_queue_28_packets: 0
tx_queue_28_bytes: 0
tx_queue_29_packets: 0
tx_queue_29_bytes: 0
tx_queue_30_packets: 0
tx_queue_30_bytes: 0
tx_queue_31_packets: 0
tx_queue_31_bytes: 0
tx_queue_32_packets: 0
tx_queue_32_bytes: 0
tx_queue_33_packets: 0
tx_queue_33_bytes: 0
tx_queue_34_packets: 0
tx_queue_34_bytes: 0
tx_queue_35_packets: 0
tx_queue_35_bytes: 0
tx_queue_36_packets: 0
tx_queue_36_bytes: 0
tx_queue_37_packets: 0
tx_queue_37_bytes: 0
tx_queue_38_packets: 0
tx_queue_38_bytes: 0
tx_queue_39_packets: 0
tx_queue_39_bytes: 0
tx_queue_40_packets: 0
tx_queue_40_bytes: 0
tx_queue_41_packets: 0
tx_queue_41_bytes: 0
tx_queue_42_packets: 0
tx_queue_42_bytes: 0
tx_queue_43_packets: 0
tx_queue_43_bytes: 0
tx_queue_44_packets: 0
tx_queue_44_bytes: 0
tx_queue_45_packets: 0
tx_queue_45_bytes: 0
tx_queue_46_packets: 0
tx_queue_46_bytes: 0
tx_queue_47_packets: 0
tx_queue_47_bytes: 0
tx_queue_48_packets: 0
tx_queue_48_bytes: 0
tx_queue_49_packets: 0
tx_queue_49_bytes: 0
tx_queue_50_packets: 0
tx_queue_50_bytes: 0
tx_queue_51_packets: 0
tx_queue_51_bytes: 0
tx_queue_52_packets: 0
tx_queue_52_bytes: 0
tx_queue_53_packets: 0
tx_queue_53_bytes: 0
tx_queue_54_packets: 0
tx_queue_54_bytes: 0
tx_queue_55_packets: 0
tx_queue_55_bytes: 0
tx_queue_56_packets: 0
tx_queue_56_bytes: 0
tx_queue_57_packets: 0
tx_queue_57_bytes: 0
tx_queue_58_packets: 0
tx_queue_58_bytes: 0
tx_queue_59_packets: 0
tx_queue_59_bytes: 0
tx_queue_60_packets: 0
tx_queue_60_bytes: 0
tx_queue_61_packets: 0
tx_queue_61_bytes: 0
tx_queue_62_packets: 0
tx_queue_62_bytes: 0
tx_queue_63_packets: 0
tx_queue_63_bytes: 0
tx_queue_64_packets: 0
tx_queue_64_bytes: 0
tx_queue_65_packets: 0
tx_queue_65_bytes: 0
tx_queue_66_packets: 0
tx_queue_66_bytes: 0
tx_queue_67_packets: 0
tx_queue_67_bytes: 0
tx_queue_68_packets: 0
tx_queue_68_bytes: 0
tx_queue_69_packets: 0
tx_queue_69_bytes: 0
tx_queue_70_packets: 0
tx_queue_70_bytes: 0
rx_queue_0_packets: 11217199
rx_queue_0_bytes: 16124868046
rx_queue_1_packets: 157176240
rx_queue_1_bytes: 227080033991
rx_queue_2_packets: 1501661
rx_queue_2_bytes: 1972184904
rx_queue_3_packets: 5075867
rx_queue_3_bytes: 5607058971
rx_queue_4_packets: 5326935
rx_queue_4_bytes: 6020504920
rx_queue_5_packets: 1606940
rx_queue_5_bytes: 1282193306
rx_queue_6_packets: 2148621
rx_queue_6_bytes: 1388353208
rx_queue_7_packets: 4038706
rx_queue_7_bytes: 3988792078
rx_queue_8_packets: 0
rx_queue_8_bytes: 0
rx_queue_9_packets: 0
rx_queue_9_bytes: 0
rx_queue_10_packets: 0
rx_queue_10_bytes: 0
rx_queue_11_packets: 0
rx_queue_11_bytes: 0
rx_queue_12_packets: 0
rx_queue_12_bytes: 0
rx_queue_13_packets: 0
rx_queue_13_bytes: 0
rx_queue_14_packets: 0
rx_queue_14_bytes: 0
rx_queue_15_packets: 0
rx_queue_15_bytes: 0
rx_queue_16_packets: 0
rx_queue_16_bytes: 0
rx_queue_17_packets: 0
rx_queue_17_bytes: 0
rx_queue_18_packets: 0
rx_queue_18_bytes: 0
rx_queue_19_packets: 0
rx_queue_19_bytes: 0
rx_queue_20_packets: 0
rx_queue_20_bytes: 0
rx_queue_21_packets: 0
rx_queue_21_bytes: 0
rx_queue_22_packets: 0
rx_queue_22_bytes: 0
rx_queue_23_packets: 0
rx_queue_23_bytes: 0
rx_queue_24_packets: 0
rx_queue_24_bytes: 0
rx_queue_25_packets: 0
rx_queue_25_bytes: 0
rx_queue_26_packets: 0
rx_queue_26_bytes: 0
rx_queue_27_packets: 0
rx_queue_27_bytes: 0
rx_queue_28_packets: 0
rx_queue_28_bytes: 0
rx_queue_29_packets: 0
rx_queue_29_bytes: 0
rx_queue_30_packets: 0
rx_queue_30_bytes: 0
rx_queue_31_packets: 0
rx_queue_31_bytes: 0
rx_queue_32_packets: 0
rx_queue_32_bytes: 0
rx_queue_33_packets: 0
rx_queue_33_bytes: 0
rx_queue_34_packets: 0
rx_queue_34_bytes: 0
rx_queue_35_packets: 0
rx_queue_35_bytes: 0
rx_queue_36_packets: 0
rx_queue_36_bytes: 0
rx_queue_37_packets: 0
rx_queue_37_bytes: 0
rx_queue_38_packets: 0
rx_queue_38_bytes: 0
rx_queue_39_packets: 0
rx_queue_39_bytes: 0
rx_queue_40_packets: 0
rx_queue_40_bytes: 0
rx_queue_41_packets: 0
rx_queue_41_bytes: 0
rx_queue_42_packets: 0
rx_queue_42_bytes: 0
rx_queue_43_packets: 0
rx_queue_43_bytes: 0
rx_queue_44_packets: 0
rx_queue_44_bytes: 0
rx_queue_45_packets: 0
rx_queue_45_bytes: 0
rx_queue_46_packets: 0
rx_queue_46_bytes: 0
rx_queue_47_packets: 0
rx_queue_47_bytes: 0
rx_queue_48_packets: 0
rx_queue_48_bytes: 0
rx_queue_49_packets: 0
rx_queue_49_bytes: 0
rx_queue_50_packets: 0
rx_queue_50_bytes: 0
rx_queue_51_packets: 0
rx_queue_51_bytes: 0
rx_queue_52_packets: 0
rx_queue_52_bytes: 0
rx_queue_53_packets: 0
rx_queue_53_bytes: 0
rx_queue_54_packets: 0
rx_queue_54_bytes: 0
rx_queue_55_packets: 0
rx_queue_55_bytes: 0
rx_queue_56_packets: 0
rx_queue_56_bytes: 0
rx_queue_57_packets: 0
rx_queue_57_bytes: 0
rx_queue_58_packets: 0
rx_queue_58_bytes: 0
rx_queue_59_packets: 0
rx_queue_59_bytes: 0
rx_queue_60_packets: 0
rx_queue_60_bytes: 0
rx_queue_61_packets: 0
rx_queue_61_bytes: 0
rx_queue_62_packets: 0
rx_queue_62_bytes: 0
rx_queue_63_packets: 0
rx_queue_63_bytes: 0
rx_queue_64_packets: 0
rx_queue_64_bytes: 0
rx_queue_65_packets: 0
rx_queue_65_bytes: 0
rx_queue_66_packets: 0
rx_queue_66_bytes: 0
rx_queue_67_packets: 0
rx_queue_67_bytes: 0
rx_queue_68_packets: 0
rx_queue_68_bytes: 0
rx_queue_69_packets: 0
rx_queue_69_bytes: 0
rx_queue_70_packets: 0
rx_queue_70_bytes: 0
tx_pb_0_pxon: 0
tx_pb_0_pxoff: 0
tx_pb_1_pxon: 0
tx_pb_1_pxoff: 0
tx_pb_2_pxon: 0
tx_pb_2_pxoff: 0
tx_pb_3_pxon: 0
tx_pb_3_pxoff: 0
tx_pb_4_pxon: 0
tx_pb_4_pxoff: 0
tx_pb_5_pxon: 0
tx_pb_5_pxoff: 0
tx_pb_6_pxon: 0
tx_pb_6_pxoff: 0
tx_pb_7_pxon: 0
tx_pb_7_pxoff: 0
rx_pb_0_pxon: 0
rx_pb_0_pxoff: 0
rx_pb_1_pxon: 0
rx_pb_1_pxoff: 0
rx_pb_2_pxon: 0
rx_pb_2_pxoff: 0
rx_pb_3_pxon: 0
rx_pb_3_pxoff: 0
rx_pb_4_pxon: 0
rx_pb_4_pxoff: 0
rx_pb_5_pxon: 0
rx_pb_5_pxoff: 0
rx_pb_6_pxon: 0
rx_pb_6_pxoff: 0
rx_pb_7_pxon: 0
rx_pb_7_pxoff: 0
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel Ethernet, visit
https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet
