** Description changed: + SRU Justification: + + Impact: + On Intel VMD 8086:ad0b / Arrow Lake-HX systems, Ubuntu 26.04 with the 7.0.x kernel has two VMD-related problems. + + First, the stock VMD driver does not support Bus Offset Setting 3. On + the affected MSI Vector this results in: + + vmd 0000:00:0e.0: Unknown Bus Offset Setting (3) + + and prevents enumeration of the internal NVMe device behind VMD. + + After adding the required second-rootbus / Bus Offset 3 support, a + second issue is exposed: repeated NVMe + + timeout, completion polled + + events caused by the Intel Arrow Lake ARL004 interrupt-ordering erratum. + These events cause severe storage stalls and make the system practically + unusable under sustained I/O. + + Launchpad bug #2166325 is now Confirmed and affects multiple users. + + Fix: + The fix is implemented as two logically separate changes: + + Add Intel VMD second-rootbus / Bus Offset 3 support so that NVMe devices behind VMD 8086:ad0b can be enumerated correctly. + Apply an ARL004-specific interrupt-ordering workaround for VMD 8086:ad0b and child PCI_CLASS_STORAGE_EXPRESS devices. Before dispatching the child interrupt with generic_handle_irq(), perform a dummy PCI configuration-space read from the MSI-initiating NVMe device. The read is serialized with the existing VMD configuration-space lock. + + The workaround is deliberately limited to VMD 8086:ad0b and NVMe-class + child devices, so other VMD devices and non-NVMe children are + unaffected. + + The stock drivers/pci/controller/vmd.c in Ubuntu 7.0.0-30.30 and + 7.0.0-31.31 is byte-for-byte identical: + + SHA256: b631d449d8e1e896bcf1fa1e4ade53789d28db3c86c6a15fb7fb7e506cd274c8 + + The patched vmd.c used successfully with Ubuntu 7.0.0-31-generic is: + + SHA256: 0a27aa1379e8b7fc22f509b8d06ed3676a211d7bb3226a3f9b3dadfc208be7a3 + + Testcase: + Hardware: MSI Vector 17 HX AI, Intel VMD 8086:ad0b, NVMe behind VMD. + + With the stock driver: + + Boot Ubuntu 26.04 with the affected 7.0.x kernel. + Verify that Bus Offset Setting 3 causes the VMD driver to fail NVMe enumeration. + After Bus Offset 3 support is added but without the ARL004 workaround, run sustained sequential NVMe reads and monitor the kernel log. + Repeated nvme nvme0: I/O tag ... timeout, completion polled events occur at approximately 30-second intervals. + A 1 GiB sequential read took approximately 121 seconds (~8.8 MB/s). + + With both fixes applied: + + The internal NVMe is enumerated normally. + The recurring timeout, completion polled events disappear. + The same 1 GiB read completes in approximately 0.88 seconds (~1.2 GB/s). + A subsequent 10 GiB sequential read also completes normally. + The same patched implementation is currently running successfully on Ubuntu 7.0.0-31-generic. + + Source and patch series: + + https://github.com/gvozd188/vmd-arl004 + + Original public implementation: + + https://github.com/gvozd188/vmd-arl004/commit/5438c83 Hardware ======== Laptop: MSI Vector 17 HX AI Platform: Intel Arrow Lake Intel VMD: 8086:ad0b VMD PCI address: 0000:00:0e.0 Subsystem: 1462:149c Internal NVMe SSD: Phison 1TB ESR01TBYCCA4-EDJ-2MS Controller: Phison PS5029-E29T PCIe 4.0 NVMe PCI ID: 1987:5029 (rev 01) Firmware: ETFM50.0 Ubuntu: 26.04 Current/final tested kernel: 7.0.0-30-generic Secure Boot: enabled - Problem 1: Intel VMD Bus Offset 3 ================================ With the stock Ubuntu VMD driver the internal NVMe SSD is not exposed. The kernel reports: - vmd 0000:00:0e.0: Unknown Bus Offset Setting (3) + vmd 0000:00:0e.0: Unknown Bus Offset Setting (3) and no internal /dev/nvme* device is available. Linux 6.14 was also tested on this machine and did not solve the VMD problem. The system firmware does not expose a usable BIOS option to disable Intel VMD/RST. Intel's public VMD second-rootbus patch series was integrated into the Ubuntu VMD source: - [PATCH v3 0/8] VMD add second rootbus support + [PATCH v3 0/8] VMD add second rootbus support https://www.spinics.net/lists/linux-pci/msg163096.html After integrating the second-rootbus support, Linux successfully enumerates the internal NVMe controller and SSD. - Problem 2: NVMe completion stalls ================================= After the SSD became accessible through the modified VMD driver, a second problem became visible. Kernel messages repeatedly contained: - nvme nvme0: I/O tag ... timeout, completion polled + nvme nvme0: I/O tag ... timeout, completion polled The stalls occurred at approximately 30-second intervals. A diagnostic sequential read before the workaround: - sudo dd if=/dev/nvme0n1 of=/dev/null bs=16M count=64 status=progress + sudo dd if=/dev/nvme0n1 of=/dev/null bs=16M count=64 status=progress Result: - 1 GiB in approximately 121.339 seconds - approximately 8.8 MB/s + 1 GiB in approximately 121.339 seconds + approximately 8.8 MB/s CPU and memory were not saturated. Disabling ASPM was tested and did not solve the problem. - ARL004 investigation ==================== The official Intel RST/VMD Windows driver distributed by MSI for this machine was examined by static analysis for comparison. The relevant Windows driver path performs a PCI configuration-space read after MSI handling when the completion state requires ordering. This behavior is consistent with the Intel Arrow Lake ARL004 erratum, where an MSI from a VMD-owned device may pass a preceding memory write. An experimental Linux workaround was implemented for the tested 8086:ad0b VMD/NVMe path. It performs a dummy PCI configuration-space read before the subsequent interrupt handling path. - Result ====== With the experimental ARL004 workaround: - sudo dd if=/dev/nvme0n1 of=/dev/null bs=16M count=64 status=progress + sudo dd if=/dev/nvme0n1 of=/dev/null bs=16M count=64 status=progress completed in: - 1 GiB in 0.88206 seconds - approximately 1.2 GB/s + 1 GiB in 0.88206 seconds + approximately 1.2 GB/s A subsequent 10 GiB sequential diagnostic read also completed normally. After booting with the final tested module: - sudo journalctl -k -b | grep -E 'timeout|completion polled' + sudo journalctl -k -b | grep -E 'timeout|completion polled' produced no matching messages. The approximately 136x difference above is only a diagnostic comparison on this specific machine and is not intended as a general SSD benchmark. - Reproduction / reference implementation ======================================= The complete investigation, tested source, separate patches, combined patch, known-working module, SHA256 checksums, installation/bootstrap procedure and recovery procedure are published here: https://github.com/gvozd188/vmd-arl004 Exact reference commit: https://github.com/gvozd188/vmd-arl004/commit/5438c83 Relevant files: - 0001-vmd-second-rootbus-intel.patch - Integration of Intel's second-rootbus support. - - 0002-vmd-ad0b-arl004-workaround.patch - Experimental ARL004 workaround. - - vmd-arl004.patch - Combined patch. - - vmd-ubuntu-7.0.0-30.c - Ubuntu VMD source used as the patch base. - - vmd.c - Final tested source. - - vmd.ko - Known-working reference module for 7.0.0-30-generic. - + 0001-vmd-second-rootbus-intel.patch + Integration of Intel's second-rootbus support. + + 0002-vmd-ad0b-arl004-workaround.patch + Experimental ARL004 workaround. + + vmd-arl004.patch + Combined patch. + + vmd-ubuntu-7.0.0-30.c + Ubuntu VMD source used as the patch base. + + vmd.c + Final tested source. + + vmd.ko + Known-working reference module for 7.0.0-30-generic. Known-working source SHA256: - 0a27aa1379e8b7fc22f509b8d06ed3676a211d7bb3226a3f9b3dadfc208be7a3 + 0a27aa1379e8b7fc22f509b8d06ed3676a211d7bb3226a3f9b3dadfc208be7a3 Known-working module SHA256: - b2c9eacb720d45fcaab15ff09ad3fdc2e97b8a6dd215c80fa282090ff8d2b2cd - + b2c9eacb720d45fcaab15ff09ad3fdc2e97b8a6dd215c80fa282090ff8d2b2cd Installation history ==================== Because the stock VMD driver could not expose the internal SSD, an existing Ubuntu HDD from another laptop was connected to the MSI Vector through a SATA-to-USB adapter. The MSI Vector was booted directly from that HDD. That Ubuntu environment was running kernel 7.0.0-30-generic. The modified VMD driver was used there to expose the internal NVMe SSD. Ubuntu was then manually deployed onto the dedicated Ubuntu partition of the internal SSD without modifying the existing Windows and data partitions. The manually deployed SSD installation initially contained kernel 7.0.0-14-generic. This kernel was not intentionally selected as a VMD workaround; it was simply the kernel present in the manually deployed system at that stage. A compatible custom VMD module was installed for 7.0.0-14-generic and added to its initramfs, allowing the first independent boot from the internal SSD. Only after that successful SSD boot was DKMS configured. The SSD installation was subsequently updated to 7.0.0-30-generic, where the final VMD/ARL004 development and testing was performed. - Expected result =============== The stock Ubuntu kernel should: 1. correctly enumerate the second-rootbus / Bus Offset 3 topology on Intel - VMD 8086:ad0b; + VMD 8086:ad0b; 2. expose the internal NVMe SSD; 3. handle NVMe completion ordering without repeated approximately 30-second - completion timeouts. - + completion timeouts. Actual result ============= With the stock VMD driver, the internal SSD is not exposed because Bus Offset Setting 3 is rejected. After adding second-rootbus support alone, the SSD becomes visible but repeated NVMe completion stalls occur on this hardware. The experimental ARL004 ordering workaround eliminates the observed stalls on the tested system. - Notes ===== The ARL004 workaround is experimental and platform-specific. I am reporting the observed hardware behavior and the tested workaround rather than claiming that this implementation is the appropriate final upstream fix. The Windows driver binary is not redistributed in the GitHub repository. Only identification information, hashes and static-analysis notes are provided. ProblemType: Bug DistroRelease: Ubuntu 26.04 Package: linux-image-7.0.0-30-generic 7.0.0-30.30 ProcVersionSignature: Ubuntu 7.0.0-30.30-generic 7.0.12 Uname: Linux 7.0.0-30-generic x86_64 ApportVersion: 2.34.1-0ubuntu0.1 Architecture: amd64 AudioDevicesInUse: - USER PID ACCESS COMMAND - /dev/snd/controlC1: gvozd188 2323 F.... pipewire - gvozd188 2360 F.... wireplumber - /dev/snd/controlC0: gvozd188 2360 F.... wireplumber - /dev/snd/seq: gvozd188 2323 F.... pipewire + USER PID ACCESS COMMAND + /dev/snd/controlC1: gvozd188 2323 F.... pipewire + gvozd188 2360 F.... wireplumber + /dev/snd/controlC0: gvozd188 2360 F.... wireplumber + /dev/snd/seq: gvozd188 2323 F.... pipewire CasperMD5CheckResult: unknown CurrentDesktop: ubuntu:GNOME Date: Thu Sep 3 13:08:10 2026 MachineType: Micro-Star International Co., Ltd. Vector 17 HX AI A2XWIG ProcFB: 0 i915drmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-7.0.0-30-generic root=UUID=fef65fb9-b1c4-4bbe-9c02-92a4a7280fee ro quiet splash iommu=pt crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) WifiSyslog: - + dmi.bios.date: 04/20/2026 dmi.bios.release: 1.18 dmi.bios.vendor: American Megatrends International, LLC. dmi.bios.version: E17S3IMS.112 dmi.board.asset.tag: Default string dmi.board.name: MS-17S3 dmi.board.vendor: Micro-Star International Co., Ltd. dmi.board.version: REV:1.0 dmi.chassis.asset.tag: No Asset Tag dmi.chassis.type: 10 dmi.chassis.vendor: Micro-Star International Co., Ltd. dmi.chassis.version: N/A dmi.modalias: dmi:bvnAmericanMegatrendsInternational,LLC.:bvrE17S3IMS.112:bd04/20/2026:br1.18:svnMicro-StarInternationalCo.,Ltd.:pnVector17HXAIA2XWIG:pvrREV1.0:rvnMicro-StarInternationalCo.,Ltd.:rnMS-17S3:rvrREV1.0:cvnMicro-StarInternationalCo.,Ltd.:ct10:cvrN/A:sku17S3.1:pfaVector: dmi.product.family: Vector dmi.product.name: Vector 17 HX AI A2XWIG dmi.product.sku: 17S3.1 dmi.product.version: REV:1.0 dmi.sys.vendor: Micro-Star International Co., Ltd.
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2166325 Title: Intel VMD 8086:ad0b: Bus Offset 3 prevents NVMe enumeration and ARL004 causes NVMe completion stalls To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2166325/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
