Public bug reported:
Subject: Boot-time SCSI async probe leaks nr_iowait (procs_blocked) — USB mass-storage card reader Category: Drivers → SCSI CC: [email protected], [email protected] == Summary == The kernel's nr_iowait / procs_blocked counter becomes permanently elevated (+4) during boot-time SCSI probing of a USB mass-storage card reader. No actual D-state processes exist, and iotop shows zero disk IO. All monitoring tools (top, vmstat, htop, node_exporter) report phantom 40-60% iowait. The counter remains stuck until reboot. Critical finding: if the same device is unbound and rebound after boot, the SCSI probe runs again through the identical code path but does NOT leak the counter. This proves the bug is specific to boot-time async scanning, not the probe logic itself. == Hardware == Genesys Logic microSD Card Reader (05e3:0751, bcdDevice 14.04) Behind VIA Labs USB hub (2109:2817) built into a monitor Intel xHCI host controller (0000:00:14.0) Intel i7-9700K (8 cores, no HT) MSI motherboard == Kernel versions affected == 6.8.0-106-generic — leaks 4-6 6.17.0-19-generic — leaks 2-4 (improved but not fixed) Ubuntu 24.04.4 LTS (Noble Numbat) with HWE kernel == Reproducer == 1. Boot system with USB card reader attached (no SD card inserted). 2. After boot completes: $ grep procs /proc/stat procs_running 2 procs_blocked 4 3. Verify no D-state processes exist: $ ps -eo pid,state,comm | awk '$2 == "D"' (empty) 4. Verify zero actual disk IO: $ iotop -bon1 | head -5 (shows zero read/write) 5. Verify all SCSI devices are healthy: $ cat /sys/class/scsi_device/*/device/state running running running 6. Rebind the device to trigger a second probe: # echo 1-4.4:1.0 > /sys/bus/usb/drivers/usb-storage/unbind # sleep 3 # echo 1-4.4:1.0 > /sys/bus/usb/drivers/usb-storage/bind # sleep 10 7. Check procs_blocked again: $ grep procs_blocked /proc/stat procs_blocked 4 Counter stays at 4 — the warm reprobe does NOT leak. == ftrace analysis == Traced io_schedule, io_schedule_timeout, blk_execute_rq, scsi_execute_cmd, and scsi_test_unit_ready with func_stack_trace enabled. The boot probe path (which leaks) and the warm reprobe path (which does not leak) both execute: io_schedule_timeout <- __wait_for_common <- wait_for_completion_io_timeout <- blk_execute_rq <- scsi_execute_cmd <- scsi_probe_lun <- scsi_probe_and_add_lun <- __scsi_scan_target <- scsi_scan_host_selected <- do_scsi_scan_host <- do_scan_async <- async_run_entry_fn <- process_one_work The leak count of 4 matches the number of synchronous SCSI commands issued during probe (TEST UNIT READY, REQUEST SENSE, READ CAPACITY, MODE SENSE), each going through blk_execute_rq() → io_schedule_timeout(). During boot, the io_schedule_prepare/io_schedule_finish pairing appears to be broken — the in_iowait flag is set but never cleared, leaking one nr_iowait increment per command. During warm reprobe, the same code path completes correctly. == Hypothesis == The bug is likely in the interaction between the async scan worker (do_scan_async via async_run_entry_fn) and the block layer's IO wait accounting during early boot. Possible causes: 1. Async worker lifecycle issue — worker thread exits or gets requeued before io_schedule_finish clears current->in_iowait 2. Block queue initialization race — request completes during partially initialized queue state, bypassing normal cleanup 3. Early boot async scan concurrency — multiple devices scanning simultaneously causes task flag inheritance issues Recent fixes in related areas suggest this is a known problem class: - Block layer iowait accounting in blk_execute_rq - blk-mq wait accounting in blk_mq_wait_for_completion - SCSI synchronous command completion handling - Async scanning races during device discovery == Suggested diagnostic patch == To confirm, add a check in do_scan_async() just before return: diff --git a/drivers/scsi/scsi_scan.c b/drivers/scsi/scsi_scan.c --- a/drivers/scsi/scsi_scan.c +++ b/drivers/scsi/scsi_scan.c @@ do_scan_async do_scsi_scan_host(shost); + if (current->in_iowait) + pr_warn("scsi_scan: worker exited with in_iowait " + "still set (pid=%d)\n", current->pid); + scsi_autopm_put_host(shost); } If the hypothesis is correct, this should fire during boot. == Workaround == usb-storage.quirks=05e3:0751:i This prevents the device from being probed at all, eliminating the leak. But it completely disables the card reader. == Parameters tested (none helped) == - usb-storage.delay_use=10 - scsi_mod.scan=sync - usb-storage.quirks=05e3:0751:u (ignore UAS) - Booting with SD card inserted == dmesg during boot probe == usb 1-4.4: New USB device found, idVendor=05e3, idProduct=0751 usb-storage 1-4.4:1.0: USB Mass Storage device detected scsi host6: usb-storage 1-4.4:1.0 [8 seconds later] scsi 6:0:0:0: Direct-Access Generic STORAGE DEVICE 1404 PQ: 0 ANSI: 6 sd 6:0:0:0: [sdc] Media removed, stopped polling sd 6:0:0:0: [sdc] Attached SCSI removable disk No SCSI errors logged. Device appears to enumerate successfully. == Attachments == - Full dmesg from boot - Full ftrace output (62k lines, filtered timeline also provided) - lsusb -t (USB topology) - lsusb -v (full USB descriptors) ** Affects: linux (Ubuntu) Importance: Undecided Status: New ** Attachment added: "ftrace of SCSI probe path showing io_schedule_timeout calls during boot and warm reprobe"" https://bugs.launchpad.net/bugs/2146707/+attachment/5956700/+files/ftrace-probe-timeline.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2146707 Title: Boot-time SCSI async probe leaks nr_iowait (procs_blocked) — USB card reader 05e3:0751 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2146707/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
