** Description changed: + SRU Justification for Resolute + + [Impact] + On Dell systems with Intel PCIe Bluetooth controllers (CID: F14-DVT2-C1X), the btintel_pcie reset and coredump paths have races. Two hardware error events at the same time can pick different reset types for one recovery. A duplicate interrupt can requeue a coredump while the previous one is still finishing. + + The earlier fix for the boot-time NULL pointer crash (LP: #2163723) did + not close all of these races. The ODM stress test could still trigger + them. This is a follow-up for the two commits that are still missing + from the resolute oem-7.0 tree. + + Original crash, about 1 in 500 boots: + + kernel: BUG: kernel NULL pointer dereference, address: 0000000000000db0 + kernel: RIP: 0010:btintel_pcie_dump_debug_registers+0x20/0x8f0 [btintel_pcie] + kernel: Call Trace: + kernel: btintel_pcie_irq_msix_handler+0x1a2/0x230 [btintel_pcie] + kernel: irq_thread_fn+0x26/0x70 + + Other problems from the two races: + - A reset can run with the wrong type (PLDR instead of FLR). + - The debug register dump runs during an FLR. The Bluetooth MMIO window can read all 1s or trigger AER. + + [Fix] + Two btintel_pcie patches. Both are in v7.3. + + 86f866189361 Bluetooth: btintel_pcie: split coredump worker into per-trigger works + Splits the one coredump worker into three workers: coredump, hw exception and fw trigger. Each worker owns one guard bit. A new helper, btintel_pcie_queue_coredump(), is the only place that sets COREDUMP_INPROGRESS. + + e0650618ee83 Bluetooth: btintel_pcie: serialize reset_type with RECOVERY_IN_PROGRESS + reset_type is now written only after RECOVERY_IN_PROGRESS is won. The sysfs/hci reset callback always requests FLR. hw_error() skips the register dump when a reset is already running. + + The second patch builds on the first. Both are clean cherry-picks of the + upstream commits. + + [Test Plan] + On a Dell system with an Intel PCIe Bluetooth controller, run the wireless/Bluetooth stress test over many boot cycles (about 1000): + $ dmesg | grep -E "btintel_pcie|NULL pointer" + + Also trigger resets while Bluetooth is in use: + $ for i in $(seq 100); do echo 1 | sudo tee /sys/class/bluetooth/hci0/reset; sleep 5; done + $ dmesg | grep -E "btintel_pcie|BUG|Call Trace" + + Without patches: NULL pointer dereference in btintel_pcie_dump_debug_registers shows up during boot or reset, or the wrong reset type runs. + With patches: no call trace, no NULL pointer dereference, Bluetooth comes back after each reset. The ODM confirmed these patches fix the issue on their test kernel. + + [Where problems could occur] + Could break the btintel_pcie reset and coredump paths on Intel PCIe Bluetooth controllers (Blazar and Scorpius Peak families). + + If the new worker split has a bug, a coredump could be dropped or + garbled. A guard bit that never clears would block later coredumps. + Teardown could also hang if the disable/enable counts for the three + workers do not match. + + If the RECOVERY_IN_PROGRESS gate in hw_error() is too strict, a real + hardware error could be ignored while a reset is running. Bluetooth + would then stay dead until the next reset. If the FLR request is wrong, + sysfs and HCI timeout resets would fail to bring the controller back. + + ======================================= + [Impact] Bluetooth call trace shows up during boot on Dell systems with Intel PCIe Bluetooth controllers (CID: F14-DVT2-C1X). It can hang the system's shutdown/boot flow. Hit rate is about 1 in 500 boots. Error log: kernel: BUG: kernel NULL pointer dereference, address: 0000000000000db0 kernel: RIP: 0010:btintel_pcie_dump_debug_registers+0x20/0x8f0 [btintel_pcie] kernel: Call Trace: kernel: ? bt_err+0x7a/0xa7 [bluetooth] kernel: btintel_pcie_irq_msix_handler+0x1a2/0x230 [btintel_pcie] kernel: irq_thread_fn+0x26/0x70 kernel: irq_thread+0xbc/0x1c0 kernel: kthread+0xf7/0x130 kernel: ret_from_fork+0x195/0x2a0 kernel: ret_from_fork_asm+0x1a/0x30 The crash hits in the MSI-X interrupt thread while dumping debug registers, and the reset/coredump path around it has several races that can corrupt driver state or drop RX events during a coredump. [Fix] Eight btintel_pcie driver fixes, in dependency order. Merged in v7.2: e43b33bf8d671 Bluetooth: btintel_pcie: Load IOSF debug regs by controller variant 2d00975c841fa Bluetooth: btintel_pcie: Support Product level reset 975a70ff0aec8 Bluetooth: btintel_pcie: Add support for smart trigger dump 4d62d88e7ae6d Bluetooth: btintel_pcie: Add 50 ms delay before MAC init on BlazarIW f70f7f2512c6b Bluetooth: btintel_pcie: Separate coredump work from RX work These add product-level reset support, smart trigger coredumps, and a timing fix for BlazarIW fast-restart, and move coredump processing off the RX workqueue so RX events don't get dropped while a coredump runs. They land ahead of the reset-path rework below. Not yet in a released kernel, merged in the bluetooth-next tree (not yet released upstream): 603b91aeb20a2 Bluetooth: btintel_pcie: Refactor FLR to use device_reprobe() ca75417ab1793 Bluetooth: btintel_pcie: split coredump worker into per-trigger works bd8bee79e1fa8 Bluetooth: btintel_pcie: serialize reset_type with RECOVERY_IN_PROGRESS These rework the reset/coredump internals: FLR now shares the same teardown/re-init path as PLDR through device_reprobe() instead of an open-coded, easy-to-desync copy; the combined coredump worker is split into three single-purpose workers (coredump, hw-exception, fw-trigger) each owning one guard bit, closing a race where a duplicate IRQ could requeue while the previous coredump was still finishing; and reset_type is now written only after the RECOVERY_IN_PROGRESS bit is won, closing a window where two concurrent hardware-error events could pick different reset types for the same recovery attempt. These 3 follow-up patches were requested directly by the ODM (Hugh Chao) after testing showed the debug-register fix alone was not enough; they close races that the ODM's stress test could still trigger. [Test Plan] Reproduce on the affected Dell platform with the Intel PCIe Bluetooth controller: Run WB (wireless/Bluetooth) stress test across repeated boot cycles (the ODM's test hit the bug at roughly 1/500 boots). Without patches: btintel_pcie_dump_debug_registers NULL pointer dereference call trace shows up in dmesg during boot, can block shutdown. With patches: no call trace, no NULL pointer dereference across repeated boot/stress cycles. ODM confirmed the 3 bluetooth-next patches fixed the issue on their test kernel. [Where problems could occur] Could break the Bluetooth btintel_pcie driver's reset and coredump paths on Intel PCIe Bluetooth controllers (Blazar and Scorpius Peak families). If the IOSF base address selection by variant is wrong for some SKU, coredump reads would return garbage instead of crashing, making register dumps useless without an obvious symptom. If the FLR-to-device_reprobe() refactor has a bug, a function-level reset could leave the Bluetooth hci device half torn down, causing btintel_pcie_probe() to fail on reprobe and BT to stay non-functional after a reset event, instead of the driver silently working around it as before. If the split coredump workers or the RECOVERY_IN_PROGRESS serialization miss a case, two reset/coredump triggers could still race, corrupting a coredump buffer or leaving a stale reset_type, which would show up as an incomplete or garbled coredump, or the wrong reset variant (FLR vs PLDR) running when a hardware error hits.
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2163723 Title: btintel_pcie NULL pointer call trace during boot on Intel Bluetooth PCIe controllers To manage notifications about this bug go to: https://bugs.launchpad.net/hwe-next/+bug/2163723/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
