Hi Krystian, Significant update - I caught a much more detailed crash on srv04 running **mainline v6.8.12** (predates the fix commit). This is the clearest evidence we've had so far, and it directly implicates `migrate_swap()`:
``` [462265.xxx] CPU: 10 PID: 137 Comm: migration/10 Tainted: G D L 6.8.12-060812-generic #202501300202 [462265.xxx] Hardware name: HP ProLiant DL385p Gen8, BIOS A28 03/14/2018 [462265.xxx] Stopper: multi_cpu_stop+0x0/0x120 <- migrate_swap+0xa2/0x130 [462265.xxx] RIP: 0010:stop_machine_yield+0x2/0x10 ``` Before that, the same host logged a soft lockup that had apparently been building for **57178 seconds (~16 hours)**: ``` [462532.xxx] watchdog: BUG: soft lockup - CPU#7 stuck for 57178s! [worker:290645] [462532.xxx] CPU: 7 PID: 290645 Comm: worker Tainted: G D L 6.8.12-060812-generic [462532.xxx] RIP: 0010:smp_call_function_many_cond+0x143/0x580 ``` This lines up very well with the mechanism you and Viktor Pashaiev described: `migrate_swap()` (called from NUMA balancing's `task_numa_compare()`) is invoking `multi_cpu_stop` on a kernel thread (`migration/10`, a per-CPU stopper thread itself), and this appears to deadlock/stall for a very long time (16h) before eventually surfacing as a soft lockup and later a full panic. Since v6.8.12 predates commit 9709eb0f845b (which excludes kernel threads from being selected in `task_numa_compare()`), this is consistent with that fix being directly relevant here. However, I want to flag some conflicting data from the ongoing tests, since it complicates the picture: - **srv04 (mainline v6.8.12)**: just panicked as described above, after being stable for ~3 days - **srv06 (mainline v6.16.0, contains the fix)**: panicked once after ~25 days uptime, with a *different* signature (`stack-protector: Kernel stack is corrupted in: start_hv_timer+0x171/0x180 [kvm]`) - rebooted and now running again for 2+ days without recurrence - **srv08 (Ubuntu 6.8.0-137, the original affected kernel, no extra boot params)**: still stable after 2+ days, no panic yet - longer than the original 1-24h window we saw across all 5 hosts initially - **srv05 (HWE 7.0.0-29)**: stable for 22 days - **srv07 (mainline v6.15.11, predates the fix)**: stable for 20 days So we now have: a kernel *without* the fix (6.8.12) panicking with the exact `migrate_swap`/`multi_cpu_stop` signature after 3 days, while another kernel *without* the fix (6.15.11) has been stable for 20 days, and a kernel *with* the fix (6.16.0) also panicked once (with an unrelated-looking signature) after 25 days. This suggests the failure window is much longer and more variable than we initially estimated (not limited to 1-24h), which makes it hard to draw firm conclusions from uptime alone - but the srv04 crash signature is the strongest direct evidence yet that `migrate_swap`/NUMA-swap-vs-stopper is a real, reproducible trigger on this hardware, even if it's not the only failure mode we're seeing. I haven't yet tested Viktor's suggested boot parameters (`processor.max_cstate=1 idle=nomwait` / `kvm halt_poll_ns=0`) - I wanted to get a baseline confirmation on 6.8.0-137 first (still pending, srv08 above), but given this new evidence I could prioritize that test now if you think it's more useful at this point. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2163642 Title: Kernel panic "Attempted to kill the idle task" on AMD Opteron multi- node NUMA under KVM (6.8.0-137) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2163642/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
