Public bug reported:
SUMMARY
On 7.0.0-42-generic, a Broadcom BCM57414 NetXtreme-E 10/25Gb NIC
(bnxt_en) suffers IOMMU DMA read faults, followed by a fatal firmware
reset. One port recovers; the other stays down (all firmware commands
abandoned) until the driver is reloaded or the host is rebooted. The
same hardware ran 7.0.0-29-generic for about 12 weeks without any such
events.
ENVIRONMENT
- Ubuntu 26.04, Dell PowerEdge R740xd2 (Intel, IOMMU/DMAR enabled)
- NIC: Broadcom BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet
Controller (PCI 14e4, 5e:00.0 and 5e:00.1), driver bnxt_en
(in-kernel). bnxt_re is not loaded.
- Both ports are bonded (active-backup) with a VLAN on top
- Bad kernel: 7.0.0-42-generic (confirmed for both failing boots)
- Good kernel: 7.0.0-29-generic (booted Aug 5 to Oct 28, and now again)
- A second identical server on 7.0.0-29-generic is also stable
- Server is idle and not in production use; only background traffic
(bond keepalives, ARP, etc.) was present
- NIC firmware: failure occurs on both
233.0.195.0 / pkg 23.31.18.10 (before)
238.1.191.0 / pkg 38.11.91.00 (after a Dell firmware update)
so the firmware does not appear to be the cause.
WHAT HAPPENS (kernel log, 7.0.0-42-generic)
Oct 28 16:32:12 DMAR: [DMA Read NO_PASID] Request device [5e:00.0]
fault addr 0xf6922000 [fault reason 0x06] PTE Read access is not set
Oct 28 17:13:55 DMAR: [DMA Read NO_PASID] Request device [5e:00.1]
fault addr 0xf70f4000 [fault reason 0x06] PTE Read access is not set
Oct 28 17:15:31 bnxt_en 0000:5e:00.1 ens1f1np1: nic open fail (rc: fffffff0)
Oct 29 10:39:04 DMAR: [DMA Read NO_PASID] Request device [5e:00.1]
fault addr 0xf7ae5000 [fault reason 0x06] PTE Read access is not set
Oct 29 10:39:04 bnxt_en 0000:5e:00.0 ens1f0np0: Fatal firmware reset
event, data1: 0x201, data2: 0xda27, min wait 1300 ms, max wait 4200 ms
Oct 29 10:39:04 bnxt_en 0000:5e:00.1 ens1f1np1: Abandoning msg
{0x23 0xbaa} len: 0 due to firmware status: 0x2000001
(repeats until the driver is reloaded)
All three fault addresses fall exactly on 4 KB page boundaries. Port 0
recovered link at 25 Gb/s about 7 seconds after the reset; port 1 did
not. Both crashes happened on an idle server, 14 minutes to about
3 hours after boot.
EXPECTED
Both NIC ports stay up at 25 Gb/s on an idle server, as they did on
7.0.0-29-generic for about 12 weeks.
ACTUAL
On 7.0.0-42-generic the NIC triggers IOMMU DMA read faults, the
firmware reports a fatal reset, and at least one port goes down and
stays down until the driver is reloaded or the host is rebooted.
WORKS ON 7.0.0-29-generic
Over about 12 weeks on 7.0.0-29 there are no "Fatal firmware reset"
messages and no IOMMU faults from the NIC (5e:00.x). Unrelated: weekly
DMA write faults from a SATA controller (3b:00.0) appear on -29 and
are not related to this report.
LIKELY RELATED
The same data1/data2 values (0x201 / 0xda27) and the same
"Abandoning msg ... firmware status: 0x2000001" sequence are reported
on other distributions' 7.0 kernels, for example this Proxmox thread
for kernel 7.0.14-20-pve:
https://forum.proxmox.com/threads/proxmox-ve-9-2-21-bcm57412-bnxt_en-network-failure-after-upgrade-to-kernel-7-0-14-20-pve.186749/
The reports there link it to a bnxt_en DMA mapping problem and show a
driver fix that resolved it on their kernels. I have not verified
which upstream commits are involved.
QUESTIONS
1. Does 7.0.0-42 contain the regressing change, and has the fix been
backported to the Ubuntu 7.0 kernel?
2. Is there a test kernel I can try?
WORKAROUND
Boot 7.0.0-29-generic.
ATTACHED
Kernel log excerpts for both bad boots, a clean-boot search, lspci and
ethtool -i output, and installed kernel package versions.
ProblemType: Bug
DistroRelease: Ubuntu 26.04
Package: linux-image-7.0.0-29-generic 7.0.0-29.29
ProcVersionSignature: Ubuntu 7.0.0-29.29-generic 7.0.12
Uname: Linux 7.0.0-29-generic x86_64
NonfreeKernelModules: zfs
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Oct 29 10:59 seq
crw-rw---- 1 root audio 116, 33 Oct 29 10:59 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.34.1-0ubuntu0.1
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq',
'/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
CasperMD5CheckResult: unknown
Date: Thu Oct 29 11:16:18 2026
InstallationDate: Installed on 2026-08-05 (85 days ago)
InstallationMedia: Ubuntu-Server 26.04 "Resolute Raccoon" - Release amd64
(20260420.1)
Lsusb:
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 1604:10c0 Tascam Dell Integrated Hub
Bus 001 Device 003: ID 1604:10c0 Tascam Dell Integrated Hub
Bus 001 Device 004: ID 1604:10c0 Tascam Dell Integrated Hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Lsusb-t:
/: Bus 001.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/16p, 480M
|__ Port 014: Dev 002, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 001: Dev 003, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 004: Dev 004, If 0, Class=Hub, Driver=hub/4p, 480M
/: Bus 002.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/10p, 5000M
MachineType: Dell Inc. PowerEdge R740xd2
PciMultimedia:
ProcEnviron:
LANG=en_GB.UTF-8
PATH=(custom, no user)
SHELL=/bin/bash
TERM=xterm-256color
ProcFB: 0 mgag200drmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-7.0.0-29-generic
root=/dev/mapper/ubuntu--vg-ubuntu--lv ro
crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 10/03/2025
dmi.bios.release: 2.25
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 2.25.0
dmi.board.name: 0C2PJH
dmi.board.vendor: Dell Inc.
dmi.board.version: A00
dmi.chassis.type: 23
dmi.chassis.vendor: Dell Inc.
dmi.modalias:
dmi:bvnDellInc.:bvr2.25.0:bd10/03/2025:br2.25:svnDellInc.:pnPowerEdgeR740xd2:pvr:rvnDellInc.:rn0C2PJH:rvrA00:cvnDellInc.:ct23:cvr:skuSKU=86C5;ModelName=PowerEdgeR740xd2:pfaPowerEdge:
dmi.product.family: PowerEdge
dmi.product.name: PowerEdge R740xd2
dmi.product.sku: SKU=86C5;ModelName=PowerEdge R740xd2
dmi.sys.vendor: Dell Inc.
** Affects: linux (Ubuntu)
Importance: Undecided
Status: New
** Tags: amd64 apport-bug resolute
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2172450
Title:
bnxt_en: IOMMU DMA read faults followed by fatal firmware reset on
BCM57414 with 7.0.0-42-generic (7.0.0-29-generic is fine)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2172450/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
