https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289682
Bug ID: 289682
Summary: [zfs] [panic] Fatal trap 12: ZFS panic, potentially
during clone, zpool problems afterwards
Product: Base System
Version: 13.5-RELEASE
Hardware: amd64
OS: Any
Status: New
Severity: Affects Only Me
Priority: ---
Component: kern
Assignee: [email protected]
Reporter: [email protected]
One of our machines crashed hard after upgrading to 13.5-RELEASE. It is likely
that ZFS clone ran at the time (machine is part of a nomad cluster that
orchestrates jails by using ZFS clone - could also have been during ZFS receive
or while these operations were going on in parallel).
No errors reported on the NVMEs, also ran long self-tests using smartctl. We
did not test server memory yet.
**Status zpool after the crash**
(status unknown beforehand, so it's unclear if this was the cause or the effect
of the crash - already scrubbed the pool)
# zpool status
pool: zroot
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
scan: scrub repaired 0B in 00:04:44 with 0 errors on Thu Sep 18 09:56:10 2025
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nvd0p3 ONLINE 0 0 1
nvd1p3 ONLINE 0 0 0
errors: No known data errors
I reattach nvd0p3 and resilvered it, now it is working again.
**The actual kernel panic from the crash dump**
GNU gdb (GDB) 15.1 [GDB v15.1 for FreeBSD]
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd13.5".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...
Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x0
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff825b4d4b
stack pointer = 0x28:0xfffffe01925adc20
frame pointer = 0x28:0xfffffe01925adca0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 0 (z_wr_int_0_2)
trap number = 12
panic: page fault
cpuid = 0
time = 1758116042
KDB: stack backtrace:
#0 0xffffffff80c43a35 at kdb_backtrace+0x65
#1 0xffffffff80bf7162 at vpanic+0x182
#2 0xffffffff80bf6fd3 at panic+0x43
#3 0xffffffff810b5169 at trap_fatal+0x389
#4 0xffffffff810b51b6 at trap_pfault+0x46
#5 0xffffffff8108c298 at calltrap+0x8
#6 0xffffffff825b4ef5 at abd_copy_to_buf_off+0x25
#7 0xffffffff825b6c1f at arc_buf_fill+0x9f
#8 0xffffffff825bcd8f at arc_read_done+0x25f
#9 0xffffffff8272b81d at zio_done+0xcbd
#10 0xffffffff82725508 at zio_execute+0x38
#11 0xffffffff80c58152 at taskqueue_run_locked+0x182
#12 0xffffffff80c593a2 at taskqueue_thread_loop+0xc2
#13 0xffffffff80bb29ff at fork_exit+0x7f
#14 0xffffffff8108d30e at fork_trampoline+0xe
Uptime: 9m36s
Dumping 4038 out of 65265 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
Reading symbols from /boot/kernel/cryptodev.ko...
Reading symbols from /usr/lib/debug//boot/kernel/cryptodev.ko.debug...
Reading symbols from /boot/kernel/zfs.ko...
Reading symbols from /usr/lib/debug//boot/kernel/zfs.ko.debug...
Reading symbols from /boot/kernel/geom_mirror.ko...
Reading symbols from /usr/lib/debug//boot/kernel/geom_mirror.ko.debug...
Reading symbols from /boot/kernel/geom_eli.ko...
Reading symbols from /usr/lib/debug//boot/kernel/geom_eli.ko.debug...
Reading symbols from /boot/kernel/acpi_wmi.ko...
Reading symbols from /usr/lib/debug//boot/kernel/acpi_wmi.ko.debug...
Reading symbols from /boot/kernel/ichsmb.ko...
Reading symbols from /usr/lib/debug//boot/kernel/ichsmb.ko.debug...
Reading symbols from /boot/kernel/smbus.ko...
Reading symbols from /usr/lib/debug//boot/kernel/smbus.ko.debug...
Reading symbols from /boot/kernel/pchtherm.ko...
Reading symbols from /usr/lib/debug//boot/kernel/pchtherm.ko.debug...
Reading symbols from /boot/kernel/pf.ko...
Reading symbols from /usr/lib/debug//boot/kernel/pf.ko.debug...
Reading symbols from /boot/kernel/if_epair.ko...
Reading symbols from /usr/lib/debug//boot/kernel/if_epair.ko.debug...
--Type <RET> for more, q to quit, c to continue without paging--
Reading symbols from /boot/kernel/nullfs.ko...
Reading symbols from /usr/lib/debug//boot/kernel/nullfs.ko.debug...
Reading symbols from /boot/kernel/if_bridge.ko...
Reading symbols from /usr/lib/debug//boot/kernel/if_bridge.ko.debug...
Reading symbols from /boot/kernel/bridgestp.ko...
Reading symbols from /usr/lib/debug//boot/kernel/bridgestp.ko.debug...
__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:53
53 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,
(kgdb) bt
#0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:53
#1 doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:394
#2 0xffffffff80bf6cfe in kern_reboot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:482
#3 0xffffffff80bf71ba in vpanic (fmt=0xffffffff81210f6a "%s",
ap=ap@entry=0xfffffe01925ada80)
at /usr/src/sys/kern/kern_shutdown.c:921
#4 0xffffffff80bf6fd3 in panic (fmt=<unavailable>) at
/usr/src/sys/kern/kern_shutdown.c:845
#5 0xffffffff810b5169 in trap_fatal (frame=0xfffffe01925adb60, eva=0) at
/usr/src/sys/amd64/amd64/trap.c:940
#6 0xffffffff810b51b6 in trap_pfault (frame=<unavailable>, usermode=false,
signo=<optimized out>, ucode=<optimized out>)
at /usr/src/sys/amd64/amd64/trap.c:759
#7 <signal handler called>
#8 abd_is_gang (abd=0x0) at /usr/src/sys/contrib/openzfs/include/sys/abd.h:192
#9 abd_iterate_func (abd=0x0, off=off@entry=0, size=16384,
func=0xffffffff825b4f00 <abd_copy_to_buf_off_cb>,
private=private@entry=0xfffffe01925adcb8) at
/usr/src/sys/contrib/openzfs/module/zfs/abd.c:805
#10 0xffffffff825b4ef5 in abd_copy_to_buf_off (buf=<optimized out>, abd=0x0,
off=off@entry=0, size=16384)
at /usr/src/sys/contrib/openzfs/module/zfs/abd.c:855
#11 0xffffffff825b6c1f in abd_copy_to_buf (buf=<optimized out>, abd=<optimized
out>, size=<optimized out>)
at /usr/src/sys/contrib/openzfs/include/sys/abd.h:159
#12 0xffffffff825b6c1f in arc_buf_fill (buf=0xfffff80603369540, spa=<optimized
out>, zb=<optimized out>, flags=<optimized out>)
from /boot/kernel/zfs.ko
#13 0xffffffff825b894a in arc_buf_alloc_impl (hdr=hdr@entry=0xfffff8062ed92720,
spa=0x0, zb=0xfffff804e53e8590,
zb@entry=0xfffff804e53e85a8, tag=<optimized out>, encrypted=0,
compressed=4294965252, noauth=<optimized out>, fill=3116598480,
ret=0xfffff800082c9740) at
/usr/src/sys/contrib/openzfs/module/zfs/arc.c:2838
#14 0xffffffff825bcd8f in arc_read_done (zio=0xfffff804b9c384d0) at
/usr/src/sys/contrib/openzfs/module/zfs/arc.c:5790
#15 0xffffffff8272b81d in zio_done (zio=0xfffff804b9c384d0) at
/usr/src/sys/contrib/openzfs/module/zfs/zio.c:4845
#16 0xffffffff82725508 in __zio_execute (zio=<optimized out>) at
/usr/src/sys/contrib/openzfs/module/zfs/zio.c:2219
#17 zio_execute (zio=<optimized out>) at
/usr/src/sys/contrib/openzfs/module/zfs/zio.c:2130
#18 0xffffffff80c58152 in taskqueue_run_locked
(queue=queue@entry=0xfffff800082cbc00) at
/usr/src/sys/kern/subr_taskqueue.c:518
#19 0xffffffff80c593a2 in taskqueue_thread_loop
(arg=arg@entry=0xfffff80008334da0) at /usr/src/sys/kern/subr_taskqueue.c:830
#20 0xffffffff80bb29ff in fork_exit (callout=0xffffffff80c592e0
<taskqueue_thread_loop>, arg=0xfffff80008334da0,
frame=0xfffffe01925adf40) at /usr/src/sys/kern/kern_fork.c:1151
#21 <signal handler called>
#22 0x988b770da12f0548 in ?? ()
Backtrace stopped: Cannot access memory at address 0xbc8353408d0f34e0
--
You are receiving this mail because:
You are the assignee for the bug.
