On Thu, Sep 03, 2026 at 03:28:26PM +0200, Michal Suchánek wrote: > On Thu, Sep 03, 2026 at 06:06:16PM +0530, Harsh Prateek Bora wrote: > > + Amit, Gautam - since they recently came across a similar regression which > > is seen with non KVM workload as well. > > + Venkat - for CI improvement suggestion below. > > > > Hi Michal, > > > > On 03/09/26 5:55 pm, Michal Suchánek wrote: > > > > > > > There is another problem that Linux 7.2.2 often locks up on boot inside > > > the guest, and as a result a different kernel is needed in the guest to > > > reproduce the host crash. This is more recent, linux 6.12 does not have > > > this problem. Will try some different kernel combinations to see if I > > > can narrow this down. For this the repro is merely booting the kernel in > > > the VM, it typically fails early before userspace produces any messages. > > This one regressed between 6.19.12 > https://github.com/openSUSE/kernel-source/blob/9f6830f/config/ppc64le/default > and 7.0.12 > https://github.com/openSUSE/kernel-source/blob/2ebf0bc/config/ppc64le/default > > Thanks > > Michal > > SLOF ********************************************************************** > QEMU Starting > Build Date = Aug 13 2026 14:47:36 > FW Version = abuild@OBS release 20230918 > Press "s" to enter Open Firmware. > > Populating /vdevice methods > Populating /vdevice/vty@30000000 > Populating /vdevice/nvram@71000000 > Populating /pci@800000020000000 > Loading Linux 7.0.12-1.g2ebf0bc-default ... > Loading initial ramdisk ... > OF stdout device is: /vdevice/vty@30000000 > Preparing to boot Linux version 7.0.12-1.g2ebf0bc-default (geeko@buildhost) > (gcc (SUSE Linux) 15.3.0, GNU ld (GNU Binutils; openSUSE Tumbleweed) > 2.45.0.20251103-4) #1 SMP PREEMPT_DYNAMIC Mon Jun 15 08:39:32 UTC 2026 > (2ebf0bc) > Detected machine type: 0000000000000101 > command line: BOOT_IMAGE=/boot/vmlinux-7.0.12-1.g2ebf0bc-default > root=UUID=304c7f07-efbb-1070-2f27-accd935ec088 rw quiet systemd.show_status=1 > security=selinux selinux=1 > Max number of cores passed to firmware: 8192 (NR_CPUS = 8192) > Calling ibm,client-architecture-support... done > memory layout at init: > memory_limit : 0000000000000000 (16 MB aligned) > alloc_bottom : 00000000066b0000 > alloc_top : 0000000030000000 > alloc_top_hi : 0000003e00000000 > rmo_top : 0000000030000000 > ram_top : 0000003e00000000 > instantiating rtas at 0x000000002fff0000... done > prom_hold_cpus: skipped > copying OF device tree... > Building dt strings... > Building dt structure... > Device tree strings 0x00000000066c0000 -> 0x00000000066c0bec > Device tree struct 0x00000000066d0000 -> 0x00000000066f0000 > Quiescing Open Firmware ... > Booting Linux via __start() @ 0x0000000000250000 ... > [ 0.000000][ T0] ERROR: Failed to allocate trace buffer > [ 0.000000][ T0] ERROR: tracer: failed to allocate ring buffer! > Linux ppc64le > #1 SMP PREEMPT_D[ 0.285758][ T673] BUG: Kernel NULL pointer dereference > on read at 0x00000010 > [ 0.285877][ T673] Faulting instruction address: 0xc000000000485bd0 > [ 0.285903][ T1] VFS: Dquot-cache hash table entries: 8192 (order 0, > 65536 bytes) > [ 0.285954][ T673] Oops: Kernel access of bad area, sig: 7 [#1] > [ 0.286133][ T673] LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=8192 NUMA > pSeries > [ 0.286240][ T673] Modules linked in: > [ 0.286288][ T673] CPU: 15 UID: 0 PID: 673 Comm: kworker/u531:0 Not > tainted 7.0.12-1.g2ebf0bc-default #1 PREEMPT(full) openSUSE Tumbleweed > (unreleased) cf0846124d7853fab338aeae87203b36cac18419 > [ 0.286527][ T673] Hardware name: IBM pSeries (emulated by qemu) Power11 > (architected) 0x820200 0xf000007 of:SLOF,HEAD hv:linux,kvm pSeries > [ 0.286739][ T673] Workqueue: trace_init_wq tracer_init_tracefs_work_func > [ 0.286818][ T673] NIP: c000000000485bd0 LR: c000000000448864 CTR: > c0000000005a3420 > [ 0.286906][ T673] REGS: c00000000a7a7960 TRAP: 0300 Not tainted > (7.0.12-1.g2ebf0bc-default) > [ 0.287014][ T673] MSR: 8000000000009033 <SF,EE,ME,IR,DR,RI,LE> CR: > 44088404 XER: 00000000 > [ 0.287122][ T673] CFAR: c000000000448860 DAR: 0000000000000010 DSISR: > 00080000 IRQMASK: 0 > [ 0.287122][ T673] GPR00: c000000000448864 c00000000a7a7c00 > c000000001f38100 c000000002a5c098 > [ 0.287122][ T673] GPR04: c0000000017dc5f0 c0000000017dc5e8 > 000000000000001f 0000000000000064 > [ 0.287122][ T673] GPR08: c0000000017dc628 00000000000005f0 > 00000000000005e8 0000000084000404 > [ 0.287122][ T673] GPR12: c0000000005a3420 c000003dfff43d80 > c0000000018e5600 c0000000018e54f0 > [ 0.287122][ T673] GPR16: 0000000000000000 0000000000000000 > 0000000000000000 0000000000000000 > [ 0.287122][ T673] GPR20: c0000000013ec418 c0000000013ef6d8 > c0000000017dc5a0 c0000000017dc590 > [ 0.287122][ T673] GPR24: c000000001825330 c0000000017dc628 > c000000014b52a05 0000000000000000 > [ 0.287122][ T673] GPR28: c0000000017dc5e8 0000000000000000 > c0000000017dc5f0 c000000002a5e008 > [ 0.287970][ T673] NIP [c000000000485bd0] __find_event_file+0x70/0x3c0 > [ 0.288046][ T673] LR [c000000000448864] init_tracer_tracefs+0x274/0xc80 > [ 0.288122][ T673] Call Trace: > [ 0.288167][ T673] [c00000000a7a7c00] [c00000000a7a7c60] > 0xc00000000a7a7c60 (unreliable) > [ 0.288258][ T673] [c00000000a7a7c60] [c000000000448864] > init_tracer_tracefs+0x274/0xc80 > [ 0.288348][ T673] [c00000000a7a7dc0] [c00000000203fcf0] > tracer_init_tracefs_work_func+0x50/0x320 > [ 0.288452][ T673] [c00000000a7a7e50] [c0000000002620e8] > process_one_work+0x1e8/0x5c0 > [ 0.288541][ T673] [c00000000a7a7f10] [c00000000026309c] > worker_thread+0x1dc/0x3d0 > [ 0.288630][ T673] [c00000000a7a7f90] [c00000000026fa34] > kthread+0x194/0x1b0 > [ 0.288721][ T673] [c00000000a7a7fe0] [c00000000000de58] > start_kernel_thread+0x14/0x18 > [ 0.288810][ T673] Code: fb410030 fb810040 fba10048 7cbc2b78 3ba00000 > fbc10050 7d194378 7c9e2378 2e2a0fc0 2da90fc0 f8010070 60420000 <e93b0010> > 81490058 e8890018 714a0208 > [ 0.289001][ T673] ---[ end trace 0000000000000000 ]--- > [ 0.290718][ T673] pstore: backend (nvram) writing error (-1)
This crash might be relevant to the problem, and while qemu can dump the memory content crash refuses to open it: crash: incompatible arguments: /usr/lib/modules/7.2.2-5.g6824496-default/vmlinux is not SMP -- /scratch/vmcore is SMP The problem is not 100% reproducible, bisecting it may be tricky. Thanks Michal
