Thanks for confirming this. I can also confirm that the equivalent interrupt-ordering workaround works on Ubuntu 26.04.1 with 7.0.0-31-generic on my MSI Vector with Intel VMD 8086:ad0b (Arrow Lake-HX).
I am currently running 7.0.0-31-generic with the patched VMD module via DKMS. The recurring nvme: I/O ... timeout, completion polled events are gone and NVMe performance is normal. My original workaround was developed for 8086:ad0b / ARL004 and performs the same essential operation: a PCI configuration-space read from the MSI-initiating NVMe device before dispatching the child interrupt. So we now have successful results on both 8086:7d0b and 8086:ad0b under Ubuntu kernel 7.0.0-31-generic. С уважением Владимир Недошивин Tel: +79656931076 > > Вторник, 8 сентября 2026, 00:00 +04:00 от Stas Safari > <[email protected] > >: > Confirming the same issue on an Arrow Lake laptop with VMD 8086:7d0b > (Ubuntu 26.04.1, kernel 7.0.0-31-generic): frequent "nvme: I/O ... > timeout, completion polled" events causing 30 s – 2 min application > stalls. APST off, C-state limits, kernel/BIOS/firmware updates did not > help. > > Applying Rickey Bartlett's "PCI: vmd: Flush initiator posted writes before > demuxing interrupts on Meteor Lake" (pci.git controller/vmd) to Ubuntu's > vmd.c via DKMS fixed it completely: 0 events, I/O pressure from ~40% to > ~0%. Please consideran SRU cherry-pick for 7.0.x. > > -- > You received this bug notification because you are subscribed to the bug > report. > https://bugs.launchpad.net/bugs/2166325 > > Title: > Intel VMD 8086:ad0b: Bus Offset 3 prevents NVMe enumeration and ARL004 > causes NVMe completion stalls > > Status in linux package in Ubuntu: > Confirmed > > Bug description: > Hardware > ======== > > Laptop: MSI Vector 17 HX AI > Platform: Intel Arrow Lake > Intel VMD: 8086:ad0b > VMD PCI address: 0000:00:0e.0 > Subsystem: 1462:149c > > Internal NVMe SSD: > Phison 1TB ESR01TBYCCA4-EDJ-2MS > Controller: Phison PS5029-E29T PCIe 4.0 NVMe > PCI ID: 1987:5029 (rev 01) > Firmware: ETFM50.0 > > Ubuntu: 26.04 > Current/final tested kernel: 7.0.0-30-generic > Secure Boot: enabled > > > Problem 1: Intel VMD Bus Offset 3 > ================================ > > With the stock Ubuntu VMD driver the internal NVMe SSD is not exposed. > > The kernel reports: > > vmd 0000:00:0e.0: Unknown Bus Offset Setting (3) > > and no internal /dev/nvme* device is available. > > Linux 6.14 was also tested on this machine and did not solve the VMD > problem. > > The system firmware does not expose a usable BIOS option to disable Intel > VMD/RST. > > Intel's public VMD second-rootbus patch series was integrated into the > Ubuntu VMD source: > > [PATCH v3 0/8] VMD add second rootbus support > > https://www.spinics.net/lists/linux-pci/msg163096.html > > After integrating the second-rootbus support, Linux successfully > enumerates > the internal NVMe controller and SSD. > > > Problem 2: NVMe completion stalls > ================================= > > After the SSD became accessible through the modified VMD driver, a second > problem became visible. > > Kernel messages repeatedly contained: > > nvme nvme0: I/O tag ... timeout, completion polled > > The stalls occurred at approximately 30-second intervals. > > A diagnostic sequential read before the workaround: > > sudo dd if=/dev/nvme0n1 of=/dev/null bs=16M count=64 > status=progress > > Result: > > 1 GiB in approximately 121.339 seconds > approximately 8.8 MB/s > > CPU and memory were not saturated. > > Disabling ASPM was tested and did not solve the problem. > > > ARL004 investigation > ==================== > > The official Intel RST/VMD Windows driver distributed by MSI for this > machine was examined by static analysis for comparison. > > The relevant Windows driver path performs a PCI configuration-space read > after MSI handling when the completion state requires ordering. > > This behavior is consistent with the Intel Arrow Lake ARL004 erratum, > where > an MSI from a VMD-owned device may pass a preceding memory write. > > An experimental Linux workaround was implemented for the tested > 8086:ad0b VMD/NVMe path. It performs a dummy PCI configuration-space read > before the subsequent interrupt handling path. > > > Result > ====== > > With the experimental ARL004 workaround: > > sudo dd if=/dev/nvme0n1 of=/dev/null bs=16M count=64 > status=progress > > completed in: > > 1 GiB in 0.88206 seconds > approximately 1.2 GB/s > > A subsequent 10 GiB sequential diagnostic read also completed > normally. > > After booting with the final tested module: > > sudo journalctl -k -b | grep -E 'timeout|completion polled' > > produced no matching messages. > > The approximately 136x difference above is only a diagnostic comparison on > > this specific machine and is not intended as a general SSD benchmark. > > > Reproduction / reference implementation > ======================================= > > The complete investigation, tested source, separate patches, combined > patch, > known-working module, SHA256 checksums, installation/bootstrap procedure > and > recovery procedure are published here: > > https://github.com/gvozd188/vmd-arl004 > > Exact reference commit: > > https://github.com/gvozd188/vmd-arl004/commit/5438c83 > > Relevant files: > > 0001-vmd-second-rootbus-intel.patch > Integration of Intel's second-rootbus support. > > 0002-vmd-ad0b-arl004-workaround.patch > Experimental ARL004 workaround. > > vmd-arl004.patch > Combined patch. > > vmd-ubuntu-7.0.0-30.c > Ubuntu VMD source used as the patch base. > > vmd.c > Final tested source. > > vmd.ko > Known-working reference module for 7.0.0-30-generic. > > > Known-working source SHA256: > > 0a27aa1379e8b7fc22f509b8d06ed3676a211d7bb3226a3f9b3dadfc208be7a3 > > Known-working module SHA256: > > b2c9eacb720d45fcaab15ff09ad3fdc2e97b8a6dd215c80fa282090ff8d2b2cd > > > Installation history > ==================== > > Because the stock VMD driver could not expose the internal SSD, an > existing > Ubuntu HDD from another laptop was connected to the MSI Vector through a > SATA-to-USB adapter. > > The MSI Vector was booted directly from that HDD. That Ubuntu environment > was running kernel 7.0.0-30-generic. > > The modified VMD driver was used there to expose the internal NVMe > SSD. > > Ubuntu was then manually deployed onto the dedicated Ubuntu partition of > the > internal SSD without modifying the existing Windows and data partitions. > > The manually deployed SSD installation initially contained kernel > 7.0.0-14-generic. This kernel was not intentionally selected as a VMD > workaround; it was simply the kernel present in the manually deployed > system > at that stage. > > A compatible custom VMD module was installed for 7.0.0-14-generic and > added > to its initramfs, allowing the first independent boot from the internal > SSD. > > Only after that successful SSD boot was DKMS configured. > > The SSD installation was subsequently updated to 7.0.0-30-generic, where > the > final VMD/ARL004 development and testing was performed. > > > Expected result > =============== > > The stock Ubuntu kernel should: > > 1. correctly enumerate the second-rootbus / Bus Offset 3 topology on Intel > > VMD 8086:ad0b; > 2. expose the internal NVMe SSD; > 3. handle NVMe completion ordering without repeated approximately > 30-second > completion timeouts. > > > Actual result > ============= > > With the stock VMD driver, the internal SSD is not exposed because Bus > Offset Setting 3 is rejected. > > After adding second-rootbus support alone, the SSD becomes visible but > repeated NVMe completion stalls occur on this hardware. > > The experimental ARL004 ordering workaround eliminates the observed stalls > > on the tested system. > > > Notes > ===== > > The ARL004 workaround is experimental and platform-specific. I am > reporting > the observed hardware behavior and the tested workaround rather than > claiming that this implementation is the appropriate final upstream fix. > > The Windows driver binary is not redistributed in the GitHub repository. > Only identification information, hashes and static-analysis notes are > provided. > > ProblemType: Bug > DistroRelease: Ubuntu 26.04 > Package: linux-image-7.0.0-30-generic 7.0.0-30.30 > ProcVersionSignature: Ubuntu 7.0.0-30.30-generic 7.0.12 > Uname: Linux 7.0.0-30-generic x86_64 > ApportVersion: 2.34.1-0ubuntu0.1 > Architecture: amd64 > AudioDevicesInUse: > USER PID ACCESS COMMAND > /dev/snd/controlC1: gvozd188 2323 F.... pipewire > gvozd188 2360 F.... wireplumber > /dev/snd/controlC0: gvozd188 2360 F.... wireplumber > /dev/snd/seq: gvozd188 2323 F.... pipewire > CasperMD5CheckResult: unknown > CurrentDesktop: ubuntu:GNOME > Date: Thu Sep 3 13:08:10 2026 > MachineType: Micro-Star International Co., Ltd. Vector 17 HX AI A2XWIG > ProcFB: 0 i915drmfb > ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-7.0.0-30-generic > root=UUID=fef65fb9-b1c4-4bbe-9c02-92a4a7280fee ro quiet splash iommu=pt > crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M > > SourcePackage: linux > UpgradeStatus: No upgrade log present (probably fresh install) > WifiSyslog: > > dmi.bios.date: 04/20/2026 > dmi.bios.release: 1.18 > dmi.bios.vendor: American Megatrends International, LLC. > dmi.bios.version: E17S3IMS.112 > dmi.board.asset.tag: Default string > dmi.board.name: MS-17S3 > dmi.board.vendor: Micro-Star International Co., Ltd. > dmi.board.version: REV:1.0 > dmi.chassis.asset.tag: No Asset Tag > dmi.chassis.type: 10 > dmi.chassis.vendor: Micro-Star International Co., Ltd. > dmi.chassis.version: N/A > dmi.modalias: > dmi:bvnAmericanMegatrendsInternational,LLC.:bvrE17S3IMS.112:bd04/20/2026:br1.18:svnMicro-StarInternationalCo.,Ltd.:pnVector17HXAIA2XWIG:pvrREV1.0:rvnMicro-StarInternationalCo.,Ltd.:rnMS-17S3:rvrREV1.0:cvnMicro-StarInternationalCo.,Ltd.:ct10:cvrN/A:sku17S3.1:pfaVector: > > dmi.product.family: Vector > dmi.product.name: Vector 17 HX AI A2XWIG > dmi.product.sku: 17S3.1 > dmi.product.version: REV:1.0 > dmi.sys.vendor: Micro-Star International Co., Ltd. > > To manage notifications about this bug go to: > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2166325/+subscriptions > > -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2166325 Title: Intel VMD 8086:ad0b: Bus Offset 3 prevents NVMe enumeration and ARL004 causes NVMe completion stalls To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2166325/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
