Hi!
I just had the same panic, with a 10.99.14 GENERIC from 20250505,
during a bulk build.
Before that I was running 10.99.12 that was stable.
On Mon, Apr 21, 2025 at 03:50:08PM +0100, Patrick Welche wrote:
> On Mon, Apr 21, 2025 at 07:52:07AM +1000, matthew green wrote:
> > > (gdb) print panicstr
> > > $2 = 0xffffffff80f06020 <scratchstr> "cpu0: softints stuck for 16 seconds"
> >
> > you'll need to find cpu0's back trace. try feeding the core
> > into crash(8) instead of GDB, it should be easier to find.
>
> crash> bt
> end() at 0
> kern_reboot() at kern_reboot+0x93
> vpanic() at vpanic+0x16b
> panic() at vprintf
> heartbeat() at heartbeat+0x1f2
> hardclock() at hardclock+0x9c
> Xresume_lapic_ltimer() at Xresume_lapic_ltimer+0x1e
> --- interrupt ---
> mutex_spin_exit() at mutex_spin_exit+0x5a
> callout_softclock() at callout_softclock+0xad
> /usr/src/sys/arch/amd64/compile/QUARK/../../../../kern/kern_timeout.c:873
> v1.79
> softint_dispatch() at softint_dispatch+0x8f
> /usr/src/sys/arch/amd64/compile/QUARK/../../../../kern/kern_softint.c:605
> v1.76
> (points to an SDT_PROBE ?! so handler already called and done?)
# crash -M netbsd.10.core -N kernel.gdb
Crash version 10.99.14, image version 10.99.14.
crash: _kvm_kvatop(0)
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
crash> bt
__kernel_end() at 0
kern_reboot() at kern_reboot+0x93
vpanic() at vpanic+0x17b
panic() at printf_nostamp
heartbeat() at heartbeat+0x34c
hardclock() at hardclock+0x8b
Xresume_lapic_ltimer() at Xresume_lapic_ltimer+0x1e
--- interrupt ---
Xspllower() at Xspllower+0xe
callout_softclock() at callout_softclock+0xd9
softint_dispatch() at softint_dispatch+0x112
DDB lost frame for Xsoftintr+0x4c, trying 0xffffb0a469e9b0f0
Xsoftintr() at Xsoftintr+0x4c
--- interrupt ---
c7af2521cbef2525:
crash> ps | grep '>' | grep -v idle/
5546 > 5546 7 0 8020000 ffffa6fffa942000 ghc-pkg-9.10.1
25332>25332 7 3 8020100 ffffa70449009c00 ksh
16311>16311 7 1 8020100 ffffa70547db1000 ksh
14468>14468 7 2 8020100 ffffa70bfb23e000 ksh
1560 > 1560 7 1 8020100 ffffa6ff3b4de800 cron
3827 > 3827 7 4 8020100 ffffa6ff228bec00 ntpd
0 > 5 7 0 200 ffffa70e6f261c00 softclk/0
0 > 0 7 5 240 ffffffff8188c000 swapper
crash> show proc /a ffffa6fffa942000
lwp_t ffffa6fffa942000
ghc-pkg-9.10.1: pid 5546 proc ffffa70c6c6ccac0 vmspace/map ffffa70be6e73c80
flags 4000
> lwp 5546 ffffa6fffa942000 pcb ffffb0a488044000
stat 7 flags 8020000 cpu 0 pri 43 ref 0
crash> show proc /a ffffa70449009c00
lwp_t ffffa70449009c00
ksh: pid 25332 proc ffffa70420e5e7c0 vmspace/map ffffa6f180115580 flags 4000
> lwp 25332 ffffa70449009c00 pcb ffffb0a487120000
stat 7 flags 8020100 cpu 3 pri 43 ref 0
crash> show proc /a ffffa70547db1000
lwp_t ffffa70547db1000
ksh: pid 16311 proc ffffa6fc1f924840 vmspace/map ffffa708220c6040 flags 4000
> lwp 16311 ffffa70547db1000 pcb ffffb0a49051c000
stat 7 flags 8020100 cpu 1 pri 43 ref 0
crash> show proc /a ffffa70bfb23e000
lwp_t ffffa70bfb23e000
ksh: pid 14468 proc ffffa70164440b40 vmspace/map ffffa6ff26ac9900 flags 4000
> lwp 14468 ffffa70bfb23e000 pcb ffffb0a488a73000
stat 7 flags 8020100 cpu 2 pri 43 ref 0
crash> show proc /a ffffa6ff3b4de800
lwp_t ffffa6ff3b4de800
cron: pid 1560 proc ffffa6ff313ff800 vmspace/map ffffa6ff310b2040 flags 200
> lwp 1560 ffffa6ff3b4de800 pcb ffffb0a47cfc5000
stat 7 flags 8020100 cpu 1 pri 43 ref 0
crash> show proc /a ffffa6ff228bec00
lwp_t ffffa6ff228bec00
ntpd: pid 3827 proc ffffa6ff22b97100 vmspace/map ffffa6ff28de9040 flags 200
> lwp 3827 ffffa6ff228bec00 pcb ffffb0a47bbd0000
stat 7 flags 8020100 cpu 4 pri 43 ref 0
crash> show proc /a ffffa70e6f261c00
lwp_t ffffa70e6f261c00
system: pid 0 proc ffffffff8188c780 vmspace/map ffffffff819769e0 flags 20002
> lwp 5 [softclk/0] ffffa70e6f261c00 pcb ffffb0a469e96000
stat 7 flags 200 cpu 0 pri 220 ref 0
lwp 4 [softbio/0] ffffa70e6f261800 pcb ffffb0a469e8f000
stat 1 flags 200 cpu 0 pri 221 ref 0
lwp 3 [softnet/0] ffffa70e6f261400 pcb ffffb0a469e88000
stat 2 flags 200 cpu 0 pri 222 ref 0
lwp 2 [idle/0] ffffa70e6f261000 pcb ffffb0a469a34000
stat 1 flags 201 cpu 0 pri 0 ref 0
> lwp 0 [swapper] ffffffff8188c000 pcb ffffffff81d56000
stat 7 flags 240 cpu 5 pri 125 ref 0
crash> show proc /a ffffffff8188c000
lwp_t ffffffff8188c000
system: pid 0 proc ffffffff8188c780 vmspace/map ffffffff819769e0 flags 20002
> lwp 0 [swapper] ffffffff8188c000 pcb ffffffff81d56000
stat 7 flags 240 cpu 5 pri 125 ref 0
Anything else I can provide?
Thomas
