Hello Timothy,
I am not able to reproduce this issue on a Power9/PowerNV
system with the same guest image you used.
Since you didn't mentioned the mode in which you booted the guest,
I tried both KVM and TCG modes, but I was not able to reproduce the
issue in either mode.
Here is my configuration:
Host: Power9/PowerNV (Fedora 43 running 6.19.7-200.fc43.ppc64le kernel)
Guest image: debian-13-generic-ppc64el-20260706-2531.qcow2
Image Source:
https://cloud-image-finder.debian.net/i/debian-13-generic-ppc64el-20260706-2531
qemu-system-ppc64 --version
QEMU emulator version 10.1.5 (qemu-10.1.5-1.fc43)
Copyright (c) 2003-2025 Fabrice Bellard and the QEMU Project developers
[ 0.000000] Linux version 6.12.95+deb13-powerpc64le
([email protected]) (powerpc64le-linux-gnu-gcc-14 (Debian
14.2.0-19) 14.2.0, GNU ld (GNU Binutils for Debian) 2.44) #1 SMP Debian
6.12.95-1 (2026-07-04)
[...]
[ 0.003276] smp: Bringing up secondary CPUs ...
[ 0.159291] smp: Brought up 1 node, 100 CPUs
[ 0.159313] numa: Node 0 CPUs: 0-99
As shown in the above log, I assigned 100 CPUs to the guest. I also
tried with 160 CPUs, but I was still not able to reproduce the issue.
The QEMU command I used to boot the guest is:
KVM Guest:
qemu-system-ppc64 -accel kvm -M pseries -smp 100 -m 8G -drive
file=debian-13-generic-ppc64el-20260706-2531.qcow2,format=qcow2,if=virtio
-nographic
From the logs you shared, it is clear that the problem occurs
during kernel boot, when the primary CPU tries to bring up the
other CPUs through powerpc's __cpu_up().
However, if I can reproduce the issue, it would be much easier to
identify the root cause and fix it.
Could you please share your configuration so that I can try to
reproduce the issue on my system?
For example:
- The QEMU command you used to boot the guest.
- qemu-system-ppc64 --version
- Host distribution and kernel version.
- Any other information that you think might help me reproduce the issue.
Thanks,
Sourabh Jain
On 17/08/26 20:59, Timothy Pearson wrote:
On a POWER9/PowerNV dual CPU host, when launching a pSeries VM via QEMU which
exposes more than around 8 guest CPU cores, there is a rapidly increasing
probability of a stuck CPU after kexec() runs, which is highly sensitive to the
number of cores available in the guest VM. If an attempt is made to pass
through all cores on a 144 core system, multiple stuck CPUs are effectively
guaranteed along with an associated boot failure of the guest.
Here is a typical guest dmesg after kexec(), note the CPU 8 stuck message:
[ 0.000000] random: crng init done
[ 0.000000] radix-mmu: Page sizes from device-tree:
[ 0.000000] radix-mmu: Page size shift = 12 AP=0x0
[ 0.000000] radix-mmu: Page size shift = 16 AP=0x5
[ 0.000000] radix-mmu: Page size shift = 21 AP=0x1
[ 0.000000] radix-mmu: Page size shift = 30 AP=0x2
[ 0.000000] Activating Kernel Userspace Access Prevention
[ 0.000000] Activating Kernel Userspace Execution Prevention
[ 0.000000] radix-mmu: Mapped 0x0000000000000000-0x0000000002600000 with
2.00 MiB pages (exec)
[ 0.000000] radix-mmu: Mapped 0x0000000002600000-0x0000000040000000 with
2.00 MiB pages
[ 0.000000] radix-mmu: Mapped 0x0000000040000000-0x0000008000000000 with
1.00 GiB pages
[ 0.000000] lpar: Using radix MMU under hypervisor
[ 0.000000] Linux version 6.12.95+deb13-powerpc64le
([email protected]) (powerpc64le-linux-gnu-gcc-14 (Debian
14.2.0-19) 14.2.0, GNU ld (GNU Binutils for Debian) 2.44) #1 SMP Debian
6.12.95-1 (2026-07-04)
[ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in
the DT
[ 0.000000] Found initrd at 0xc0000000064f0000:0xc000000007d7979c
[ 0.000000] Hardware name: IBM pSeries (emulated by qemu) POWER9
(architected) 0x4e1203 0xf000005 of:SLOF,HEAD hv:linux,kvm pSeries
[ 0.000000] printk: legacy bootconsole [udbg0] enabled
[ 0.000000] Partition configured for 43 cpus.
[ 0.000000] CPU maps initialized for 1 thread per core
[ 0.000000] (thread shift is 0)
[ 0.000000] Allocated 1624 bytes for 43 pacas
[ 0.000000] numa: Partition configured for 1 NUMA nodes.
[ 0.000000] -----------------------------------------------------
[ 0.000000] phys_mem_size = 0x8000000000
[ 0.000000] dcache_bsize = 0x80
[ 0.000000] icache_bsize = 0x80
[ 0.000000] cpu_features = 0x0001c06b8f4f9183
[ 0.000000] possible = 0x001ffbebcf5fb187
[ 0.000000] always = 0x0000000380008181
[ 0.000000] cpu_user_features = 0xdc0065c2 0xaef00000
[ 0.000000] mmu_features = 0x1c007641
[ 0.000000] firmware_features = 0x00000885455a445f
[ 0.000000] vmalloc start = 0xc008000000000000
[ 0.000000] IO start = 0xc00a000000000000
[ 0.000000] vmemmap start = 0xc00c000000000000
[ 0.000000] -----------------------------------------------------
[ 0.000000] NODE_DATA(0) allocated [mem 0x7fffa28f00-0x7fffa310ff]
[ 0.000000] rfi-flush: fallback displacement flush available
[ 0.000000] rfi-flush: ori type flush available
[ 0.000000] rfi-flush: mttrig type flush available
[ 0.000000] rfi-flush: patched 12 locations (ori+mttrig type flush)
[ 0.000000] count-cache-flush: hardware flush enabled.
[ 0.000000] link-stack-flush: software flush enabled.
[ 0.000000] entry-flush: patched 61 locations (ori+mttrig type flush)
[ 0.000000] uaccess-flush: patched 1 locations (ori+mttrig type flush)
[ 0.000000] stf-barrier: eieio barrier available
[ 0.000000] stf-barrier: patched 61 entry locations (eieio barrier)
[ 0.000000] stf-barrier: patched 12 exit locations (eieio barrier)
[ 0.000000] PPC64 nvram contains 65536 bytes
[ 0.000000] barrier-nospec: using ORI speculation barrier
[ 0.000000] barrier-nospec: patched 176 locations
[ 0.000000] Top of RAM: 0x8000000000, Total RAM: 0x8000000000
[ 0.000000] Memory hole size: 0MB
[ 0.000000] Zone ranges:
[ 0.000000] Normal [mem 0x0000000000000000-0x0000007fffffffff]
[ 0.000000] Device empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000000000-0x0000007fffffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000007fffffffff]
[ 0.000000] percpu: Embedded 39 pages/cpu s120472 r0 d39272 u159744
[ 0.000000] pcpu-alloc: s120472 r0 d39272 u159744 alloc=39*4096
[ 0.000000] pcpu-alloc: [0] 00 [0] 01 [0] 02 [0] 03 [0] 04 [0] 05 [0] 06 [0]
07
[ 0.000000] pcpu-alloc: [0] 08 [0] 09 [0] 10 [0] 11 [0] 12 [0] 13 [0] 14 [0]
15
[ 0.000000] pcpu-alloc: [0] 16 [0] 17 [0] 18 [0] 19 [0] 20 [0] 21 [0] 22 [0]
23
[ 0.000000] pcpu-alloc: [0] 24 [0] 25 [0] 26 [0] 27 [0] 28 [0] 29 [0] 30 [0]
31
[ 0.000000] pcpu-alloc: [0] 32 [0] 33 [0] 34 [0] 35 [0] 36 [0] 37 [0] 38 [0]
39
[ 0.000000] pcpu-alloc: [0] 40 [0] 41 [0] 42
[ 0.000000] Kernel command line: root=/dev/nfs rootfstype=nfs
nfsroot=192.168.2.2:/mnt/udata/nfsroots/ppc64el/astrid,v3,hard,nointr,tcp
ip=192.168.2.36::192.168.2.1:255.255.255.0:embed-o-matic:enp0s0:off
ast.modeset=0 console=tty0 console=hvc0,115200n8 panic=10 rw
[ 0.000000] Unknown kernel command line parameters
"nfsroot=192.168.2.2:/mnt/udata/nfsroots/ppc64el/astrid,v3,hard,nointr,tcp
ip=192.168.2.36::192.168.2.1:255.255.255.0:embed-o-matic:enp0s0:off", will be passed
to user space.
[ 0.000000] printk: log_buf_len individual max cpu contribution: 4096 bytes
[ 0.000000] printk: log_buf_len total cpu_extra contributions: 172032 bytes
[ 0.000000] printk: log_buf_len min size: 131072 bytes
[ 0.000000] printk: log_buf_len: 524288 bytes
[ 0.000000] printk: early log buf free: 126384(96%)
[ 0.000000] Dentry cache hash table entries: 16777216 (order: 15, 134217728
bytes, linear)
[ 0.000000] Inode-cache hash table entries: 8388608 (order: 14, 67108864
bytes, linear)
[ 0.000000] Fallback order for Node 0: 0
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 134217728
[ 0.000000] Policy zone: Normal
[ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off
[ 0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=43, Nodes=1
[ 0.000000] ftrace: allocating 38704 entries in 227 pages
[ 0.000000] ftrace: allocated 227 pages with 5 groups
[ 0.000000] rcu: Hierarchical RCU implementation.
[ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=2048 to nr_cpu_ids=43.
[ 0.000000] Rude variant of Tasks RCU enabled.
[ 0.000000] Tracing variant of Tasks RCU enabled.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25
jiffies.
[ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=43
[ 0.000000] RCU Tasks Rude: Setting shift to 6 and lim to 1
rcu_task_cb_adjust=1 rcu_task_cpu_ids=43.
[ 0.000000] RCU Tasks Trace: Setting shift to 6 and lim to 1
rcu_task_cb_adjust=1 rcu_task_cpu_ids=43.
[ 0.000000] NR_IRQS: 512, nr_irqs: 512, preallocated irqs: 16
[ 0.000000] xive: Using IRQ range [0-2a]
[ 0.000000] xive: Interrupt handling initialized with spapr backend
[ 0.000000] xive: Using priority 6 for all interrupts
[ 0.000000] xive: Using 64kB queues
[ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[ 0.000000] time_init: decrementer frequency = 512.000000 MHz
[ 0.000000] time_init: processor frequency = 2134.000000 MHz
[ 0.000001] time_init: 56 bit decrementer (max: 7fffffffffffff)
[ 0.000735] clocksource: timebase: mask: 0xffffffffffffffff max_cycles:
0x761537d007, max_idle_ns: 440795202126 ns
[ 0.002016] clocksource: timebase mult[1f40000] shift[24] registered
[ 0.002809] clockevent: decrementer mult[83126f] shift[24] cpu[0]
[ 0.002967] Console: colour dummy device 80x25
[ 0.003524] printk: legacy console [tty0] enabled
[ 0.004385] printk: legacy console [hvc0] enabled
[ 0.005019] printk: legacy bootconsole [udbg0] disabled
[ 0.005768] pid_max: default: 44032 minimum: 344
[ 0.005904] LSM: initializing
lsm=capability,landlock,lockdown,yama,apparmor,tomoyo,bpf,ipe,ima,evm
[ 0.006096] landlock: Up and running.
[ 0.006139] Yama: disabled by default; enable with sysctl kernel.yama.*
[ 0.006342] AppArmor: AppArmor initialized
[ 0.006439] TOMOYO Linux initialized
[ 0.006858] LSM support for eBPF active
[ 0.007186] Mount-cache hash table entries: 262144 (order: 9, 2097152 bytes,
linear)
[ 0.007404] Mountpoint-cache hash table entries: 262144 (order: 9, 2097152
bytes, linear)
[ 0.008424] POWER9 performance monitor hardware support registered
[ 0.008547] rcu: Hierarchical SRCU implementation.
[ 0.008602] rcu: Max phase no-delay instances is 1000.
[ 0.008725] Timer migration: 2 hierarchy levels; 8 children per group; 2
crossnode level
[ 0.009704] smp: Bringing up secondary CPUs ...
[ 5.021136] Processor 8 is stuck.
[ 5.040206] smp: Brought up 1 node, 42 CPUs
[ 5.040269] numa: Node 0 CPUs: 0-42
[ 5.539085] node 0 deferred pages initialised in 468ms
[ 5.539259] Memory: 528141096K/536870912K available (16428K kernel code,
2300K rwdata, 16340K rodata, 5080K init, 1320K bss, 8695376K reserved, 0K
cma-reserved)
[ 5.592658] devtmpfs: initialized
[ 5.604671] PCI host bridge /pci@800000020000000 ranges:
[ 5.604729] IO 0x0000200000000000..0x000020000000ffff -> 0x0000000000000000
[ 5.604796] MEM 0x0000200080000000..0x00002000ffffffff -> 0x0000000080000000
[ 5.604861] MEM 0x0000210000000000..0x000021ffffffffff -> 0x0000210000000000
[ 5.605039] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
max_idle_ns: 7645041785100000 ns
[ 5.605248] futex hash table entries: 16384 (order: 9, 2097152 bytes, linear)
[ 5.607696] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[ 5.607954] audit: initializing netlink subsys (disabled)
[ 5.608195] audit: type=2000 audit(1786980313.572:1): state=initialized
audit_enabled=0 res=1
[ 5.608317] thermal_sys: Registered thermal governor 'fair_share'
[ 5.608344] thermal_sys: Registered thermal governor 'bang_bang'
[ 5.608416] thermal_sys: Registered thermal governor 'step_wise'
[ 5.608576] thermal_sys: Registered thermal governor 'user_space'
[ 5.608659] thermal_sys: Registered thermal governor 'power_allocator'
