Hi People.
First of all, thanks for all the hard work you are doing in this field. Just
now the number of CVEs is very extreme and I realize nobody is perfect and
things might slip through with all these changes.
We generally are very happy with the stability that Debian provides.
We can confirm this bug. We have multiple AMD-based Systems that all experience
the same regressions, when updating to 6.12.11 connectivity fails very quickly.
The driver in question is: bnxt_en
Given the circumstances we can only test in this scenario on weekends. We did
so on saturday. It turns out that the kernel 6.12.107 is fine, but 6.12.111 is
not. We tried the backports kernel 7.2.something , but that failed too,
similarly. So did trying firmware-bnx2 from forky and backports or ethtool ...
tso off gso off gro off
I am attaching a few text files in the hope it helps to locate the bug.
*
debian_negativ_case.txt (Kernel-Log grepped to bnxt_en in the failing case of
booting 6.2.111)
*
debian_positive_case.txt (Kernel-Log grepped to bnxt_en in the working case of
booting 6.2.107)
*
lspci.txt (PCI Network Cards on this System, lspci grepped)
The example I attached attaching is a system having a AMD EPYC 7643 48-Core
Processor and 512GB RAM.
I can try to give more details if that helps, just contact me.
Sincerely
Matthias Babisch
IT & DevOps Services
Mail
[email protected]<mailto:[email protected]>
Web
www.finnofleet.com<https://www.finnofleet.com>
FINNOFLEET BMI GmbH
Rotenhofer Weg 20
24109 Melsdorf
Follow us on:
LinkedIn<https://www.linkedin.com/company/finnofleet/>
Registergericht: Kiel | Handelsregisternummer: HRB 26960 | Geschäftsführer:
Johannes Boppre
Okt 10 13:29:06 tyr kernel: bnxt_en 0000:81:00.0 eth0: Broadcom BCM57416
NetXtreme-E 10GBase-T Ethernet found at mem 20050110000, node addr
aa:aa:aa:aa:aa:aa
Okt 10 13:29:06 tyr kernel: bnxt_en 0000:81:00.0: 63.008 Gb/s available PCIe
bandwidth (8.0 GT/s PCIe x8 link)
Okt 10 13:29:06 tyr kernel: bnxt_en 0000:81:00.1 eth1: Broadcom BCM57416
NetXtreme-E 10GBase-T Ethernet found at mem 20050100000, node addr
bb:bb:bb:bb:bb:bb
Okt 10 13:29:06 tyr kernel: bnxt_en 0000:81:00.1: 63.008 Gb/s available PCIe
bandwidth (8.0 GT/s PCIe x8 link)
Okt 10 13:29:06 tyr kernel: bnxt_en 0000:48:00.0 eth2: Broadcom BCM57416
NetXtreme-E 10GBase-T Ethernet found at mem 28080110000, node addr
cc:cc:cc:cc:cc:cc
Okt 10 13:29:06 tyr kernel: bnxt_en 0000:48:00.0: 31.504 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x4 link at 0000:40:03.6 (capable of 63.008
Gb/s with 8.0 GT/s PCIe x8 link)
Okt 10 13:29:06 tyr kernel: bnxt_en 0000:48:00.1 eth3: Broadcom BCM57416
NetXtreme-E 10GBase-T Ethernet found at mem 28080100000, node addr
dd:dd:dd:dd:dd:dd
Okt 10 13:29:06 tyr kernel: bnxt_en 0000:48:00.1: 31.504 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x4 link at 0000:40:03.6 (capable of 63.008
Gb/s with 8.0 GT/s PCIe x8 link)
Okt 10 13:29:06 tyr kernel: bnxt_en 0000:48:00.0 eno1np0: renamed from eth2
Okt 10 13:29:06 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: renamed from eth3
Okt 10 13:29:06 tyr kernel: bnxt_en 0000:81:00.0 enp129s0f0np0: renamed from
eth0
Okt 10 13:29:06 tyr kernel: bnxt_en 0000:81:00.1 enp129s0f1np1: renamed from
eth1
Okt 10 13:29:26 tyr kernel: bnxt_en 0000:48:00.0 eno1np0: NIC Link is Up, 1000
Mbps full duplex, Flow control: ON - receive & transmit
Okt 10 13:29:26 tyr kernel: bnxt_en 0000:48:00.0 eno1np0: EEE is not active
Okt 10 13:29:26 tyr kernel: bnxt_en 0000:48:00.0 eno1np0: FEC autoneg off
encoding: None
Okt 10 13:29:27 tyr kernel: bnxt_en 0000:81:00.0 enp129s0f0np0: NIC Link is Up,
1000 Mbps full duplex, Flow control: ON - receive & transmit
Okt 10 13:29:27 tyr kernel: bnxt_en 0000:81:00.0 enp129s0f0np0: EEE is not
active
Okt 10 13:29:27 tyr kernel: bnxt_en 0000:81:00.0 enp129s0f0np0: FEC autoneg off
encoding: None
Okt 10 13:29:28 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: NIC Link is Up, 1000
Mbps full duplex, Flow control: ON - receive & transmit
Okt 10 13:29:28 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: EEE is not active
Okt 10 13:29:28 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: FEC autoneg off
encoding: None
Okt 10 13:29:29 tyr kernel: bnxt_en 0000:81:00.1 enp129s0f1np1: NIC Link is Up,
1000 Mbps full duplex, Flow control: ON - receive & transmit
Okt 10 13:29:29 tyr kernel: bnxt_en 0000:81:00.1 enp129s0f1np1: EEE is not
active
Okt 10 13:29:29 tyr kernel: bnxt_en 0000:81:00.1 enp129s0f1np1: FEC autoneg off
encoding: None
Okt 10 13:29:29 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: entered allmulticast
mode
Okt 10 13:29:29 tyr kernel: bnxt_en 0000:81:00.1 enp129s0f1np1: entered
allmulticast mode
Okt 10 13:29:29 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: entered promiscuous
mode
Okt 10 13:29:29 tyr kernel: bnxt_en 0000:81:00.1 enp129s0f1np1: entered
promiscuous mode
Okt 10 13:29:31 tyr kernel: bnxt_en 0000:48:00.0 eno1np0: entered allmulticast
mode
Okt 10 13:29:31 tyr kernel: bnxt_en 0000:81:00.0 enp129s0f0np0: entered
allmulticast mode
Okt 10 13:29:31 tyr kernel: bnxt_en 0000:48:00.0 eno1np0: entered promiscuous
mode
Okt 10 13:29:31 tyr kernel: bnxt_en 0000:81:00.0 enp129s0f0np0: entered
promiscuous mode
Okt 10 13:29:39 tyr kernel: bnxt_en 0000:48:00.1: AMD-Vi: Event logged
[IO_PAGE_FAULT domain=0x0039 address=0xe5fb7000 flags=0x0000]
Okt 10 13:29:41 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: Error (timeout:
500015) msg {0x23 0x152} len:0
Okt 10 13:29:42 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: Error (timeout:
500015) msg {0xb4 0x153} len:0
Okt 10 13:29:43 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: Error (timeout:
500015) msg {0x23 0x154} len:0
Okt 10 13:29:44 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: Error (timeout:
500015) msg {0xb4 0x155} len:0
Okt 10 13:29:44 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: Error (timeout:
500015) msg {0x23 0x156} len:0
Okt 10 13:29:45 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: Error (timeout:
500015) msg {0xb4 0x157} len:0
Okt 10 13:29:46 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: Error (timeout:
500015) msg {0x23 0x158} len:0
Okt 10 13:29:47 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: Error (timeout:
500015) msg {0xb4 0x159} len:0
Okt 10 13:29:47 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: NETDEV WATCHDOG: CPU:
26: transmit queue 20 timed out 5120 ms
Okt 10 13:29:47 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: TX timeout detected,
starting reset task!
Okt 10 13:29:48 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: Error (timeout:
500015) msg {0x23 0x15a} len:0
Okt 10 13:29:49 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: Error (timeout:
500015) msg {0xb4 0x15b} len:0
Okt 10 13:29:49 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: [0.0]: tx{fw_ring:
170 prod: c cons: c}
Okt 10 13:29:49 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: [0]: rx{fw_ring: 242
prod: 202} rx_agg{fw_ring: 266 agg_prod: 7fc sw_agg_prod: 7fc}
Okt 10 13:57:10 tyr kernel: bnxt_en 0000:81:00.0 eth0: Broadcom BCM57416
NetXtreme-E 10GBase-T Ethernet found at mem 20050110000, node addr
aa:aa:aa:aa:aa:aa
Okt 10 13:57:10 tyr kernel: bnxt_en 0000:81:00.0: 63.008 Gb/s available PCIe
bandwidth (8.0 GT/s PCIe x8 link)
Okt 10 13:57:10 tyr kernel: bnxt_en 0000:81:00.1 eth1: Broadcom BCM57416
NetXtreme-E 10GBase-T Ethernet found at mem 20050100000, node addr
bb:bb:bb:bb:bb:bb
Okt 10 13:57:10 tyr kernel: bnxt_en 0000:81:00.1: 63.008 Gb/s available PCIe
bandwidth (8.0 GT/s PCIe x8 link)
Okt 10 13:57:10 tyr kernel: bnxt_en 0000:48:00.0 eth2: Broadcom BCM57416
NetXtreme-E 10GBase-T Ethernet found at mem 28080110000, node addr
cc:cc:cc:cc:cc:cc
Okt 10 13:57:10 tyr kernel: bnxt_en 0000:48:00.0: 31.504 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x4 link at 0000:40:03.6 (capable of 63.008
Gb/s with 8.0 GT/s PCIe x8 link)
Okt 10 13:57:10 tyr kernel: bnxt_en 0000:48:00.1 eth3: Broadcom BCM57416
NetXtreme-E 10GBase-T Ethernet found at mem 28080100000, node addr
dd:dd:dd:dd:dd:dd
Okt 10 13:57:10 tyr kernel: bnxt_en 0000:48:00.1: 31.504 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x4 link at 0000:40:03.6 (capable of 63.008
Gb/s with 8.0 GT/s PCIe x8 link)
Okt 10 13:57:10 tyr kernel: bnxt_en 0000:48:00.0 eno1np0: renamed from eth2
Okt 10 13:57:10 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: renamed from eth3
Okt 10 13:57:10 tyr kernel: bnxt_en 0000:81:00.0 enp129s0f0np0: renamed from
eth0
Okt 10 13:57:10 tyr kernel: bnxt_en 0000:81:00.1 enp129s0f1np1: renamed from
eth1
Okt 10 13:57:31 tyr kernel: bnxt_en 0000:48:00.0 eno1np0: NIC Link is Up, 1000
Mbps full duplex, Flow control: ON - receive & transmit
Okt 10 13:57:31 tyr kernel: bnxt_en 0000:48:00.0 eno1np0: EEE is not active
Okt 10 13:57:31 tyr kernel: bnxt_en 0000:48:00.0 eno1np0: FEC autoneg off
encoding: None
Okt 10 13:57:32 tyr kernel: bnxt_en 0000:81:00.0 enp129s0f0np0: NIC Link is Up,
1000 Mbps full duplex, Flow control: ON - receive & transmit
Okt 10 13:57:32 tyr kernel: bnxt_en 0000:81:00.0 enp129s0f0np0: EEE is not
active
Okt 10 13:57:32 tyr kernel: bnxt_en 0000:81:00.0 enp129s0f0np0: FEC autoneg off
encoding: None
Okt 10 13:57:33 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: NIC Link is Up, 1000
Mbps full duplex, Flow control: ON - receive & transmit
Okt 10 13:57:33 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: EEE is not active
Okt 10 13:57:33 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: FEC autoneg off
encoding: None
Okt 10 13:57:34 tyr kernel: bnxt_en 0000:81:00.1 enp129s0f1np1: NIC Link is Up,
1000 Mbps full duplex, Flow control: ON - receive & transmit
Okt 10 13:57:34 tyr kernel: bnxt_en 0000:81:00.1 enp129s0f1np1: EEE is not
active
Okt 10 13:57:34 tyr kernel: bnxt_en 0000:81:00.1 enp129s0f1np1: FEC autoneg off
encoding: None
Okt 10 13:57:34 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: entered allmulticast
mode
Okt 10 13:57:34 tyr kernel: bnxt_en 0000:81:00.1 enp129s0f1np1: entered
allmulticast mode
Okt 10 13:57:34 tyr kernel: bnxt_en 0000:48:00.1 eno2np1: entered promiscuous
mode
Okt 10 13:57:34 tyr kernel: bnxt_en 0000:81:00.1 enp129s0f1np1: entered
promiscuous mode
Okt 10 13:57:36 tyr kernel: bnxt_en 0000:48:00.0 eno1np0: entered allmulticast
mode
Okt 10 13:57:36 tyr kernel: bnxt_en 0000:81:00.0 enp129s0f0np0: entered
allmulticast mode
Okt 10 13:57:36 tyr kernel: bnxt_en 0000:48:00.0 eno1np0: entered promiscuous
mode
Okt 10 13:57:36 tyr kernel: bnxt_en 0000:81:00.0 enp129s0f0np0: entered
promiscuous mode
48:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416
NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01)
48:00.1 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416
NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01)
81:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416
NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01)
81:00.1 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416
NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01)
