** Description changed:
- We observed a kernel panic followed by a ZFS pool hang caused by a race
- between dirty mmap page writeback and VFS inode eviction.
+ [ Impact ]
+
+ ZFS can crash, followed by a pool hang caused by a race condition when a file
is being
+ written to disk in the background but at the same time, it is deleted.
+
+ Explanation of what happened is the following:
+ - A file is changed through memory (mmap), so the kernel saves it to disk
later in the background (writeback).
+ - While saving, ZFS checks whether the file is being deleted. Right after
that check it calls igrab() to take a hold on the file.
+ - If the file gets deleted in the gap between the check and igrab(), igrab()
fails. ZFS has a safety check (VERIFY) that expects igrab() to always work, so
it panics at:
+
+ VERIFY3(igrab(ZTOI((zp))) != NULL) failed PANIC at
+ zfs_log.c:257:zfs_xattr_owner_unlinked()
+
+ which then stops the writeback thread.
+ - The process that deleted the file waits for the writeback to finish but
the stopped writeback thread held an open ZFS transaction, so the pool's
transaction group could never be written to disk. "zpool sync" hangs, other
processes pile up and the system hangs.
+
+ This issue was hit with QuestDB, which writes its WAL (log) files
+ through mmap and deletes them often.
+
+ Note: This was the specific sequence of events that hit the issue:
+ igrab() failed because the inode was already being evicted, so zhold()
+ hit the VERIFY and panicked. The fix also solves the opposite race,
+ where igrab() succeeds. In that case another process unlinks the file
+ and drops its reference while zfs_xattr_owner_unlinked() still holds the
+ extra reference. The flusher's own zrele() then becomes the final iput()
+ and starts evicting the inode inside the writeback thread. evict() waits
+ for I_SYNC to clear, but the flusher set that flag itself, so it waits
+ on itself forever. That would give a hang without any panic message.
+
+ The upstream fix is a change to zfs_xattr_owner_unlinked() where ZFS no
+ longer takes its own hold on the file there, because the caller already
+ holds it. ZFS now only holds and releases the parent files it looks up
+ itself, and releases them in the background (async), which is safe
+ inside a transaction. With no igrab() call, the crash can't happen.
+
+ [ Test Plan ]
+
+ Due to the non-deterministic nature of this bug, it is hard to reliably hit
this with a normal test load. For this reason, the test plan uses a a small
test-only kernel module (kprobe) which adds a 3-second wait just before
igrab(), and only for one ZFS file that is exactly 4095 bytes (this size was
chosen so we can differentiate the file).
+ This only changes timing, not what ZFS does.
+
+ Needs: a VM with at least 2 vCPUs because we have two things running at
+ the same time. It is recommended that the root disk must not be on ZFS
+ because the test hangs ZFS. The pool hangs and reboot will be required.
+
+ 1. Setup: install the required ZFS tools and create a 1GB file, a pool
+ on that file and a dataset to simulate the role of a QuestDB’s WAL
+ directory.
+
+ sudo apt update
+ sudo apt install -y zfsutils-linux build-essential linux-headers-$(uname -r)
+ truncate -s 1G /var/tmp/lp2169003.img
+ sudo zpool create lp2169003 /var/tmp/lp2169003.img
+ sudo zfs create lp2169003/wal
+
+ 2. Build and load the test kprobe
+ (adds 3 s wait before igrab() only from a kernel thread and only for a
4095-byte file that is on ZFS)
+
+ ###### Add the below code to a lp2169003.c file ########
+ // SPDX-License-Identifier: GPL-2.0
+ #include <linux/delay.h>
+ #include <linux/fs.h>
+ #include <linux/kprobes.h>
+ #include <linux/module.h>
+ #include <linux/sched.h>
+
+ static unsigned int delay_ms = 3000;
+ module_param(delay_ms, uint, 0644);
+ static ulong target_size = 4095;
+ module_param(target_size, ulong, 0644);
+
+ static int lp2169003_pre(struct kprobe *p, struct pt_regs *regs)
+ {
+ struct inode *ip = (struct inode *)regs->di;
+
+ if (!(current->flags & PF_KTHREAD) || ip == NULL || ip->i_sb == NULL ||
+ strcmp(ip->i_sb->s_type->name, "zfs") != 0 ||
+ (ulong)i_size_read(ip) != target_size)
+ return 0;
+ pr_info("lp2169003: delaying igrab(ino %lu) %u ms, comm %s, caller %pS\n",
+ ip->i_ino, delay_ms, current->comm,
+ (void *)regs_get_kernel_stack_nth(regs, 0));
+ mdelay(delay_ms);
+ pr_info("lp2169003: delay done, ino %lu i_state 0x%lx\n",
+ ip->i_ino, (unsigned long)READ_ONCE(ip->i_state));
+ return 0;
+ }
+
+ static struct kprobe kp = {
+ .symbol_name = "igrab",
+ .pre_handler = lp2169003_pre,
+ };
+ static int __init lp_init(void) { return register_kprobe(&kp); }
+ static void __exit lp_exit(void) { unregister_kprobe(&kp); }
+ module_init(lp_init);
+ module_exit(lp_exit);
+ MODULE_LICENSE("GPL");
+
+
+ ####### build and load ########
+ echo 'obj-m := lp2169003.o' > Makefile
+ make -C /lib/modules/$(uname -r)/build M=$PWD modules
+ sudo insmod lp2169003.ko
+
+ 3. Trigger the bug.
+ For this, create a python script to write a 4095-byte file through mmap and
close it, like QuestDB does with a WAL file. Start a sync so the background
flusher saves it and gets stuck in the 3 s wait. During that wait, delete the
file.
+
+ sudo dmesg -C
+ sudo python3 - <<'EOF' &
+ import mmap, os, subprocess, time
+ d = "/lp2169003/wal"; p = d + "/victim"
+ fd = os.open(p, os.O_CREAT | os.O_RDWR | os.O_TRUNC, 0o644)
+ os.ftruncate(fd, 4095)
+ m = mmap.mmap(fd, 4095); m[:] = b"x" * 4095; m.close(); os.close(fd)
+ s = subprocess.Popen(["sync", "-f", d])
+ time.sleep(1)
+ os.unlink(p)
+ s.wait()
+ print("PASS: unlink and syncfs completed")
+ EOF
+ sleep 10; sudo dmesg
+
+ 4. Without the fix, in the dmseg you should see the following:
+
+ lp2169003: delaying igrab(ino 128) 3000 ms, comm kworker/u11:4, caller
zfs_xattr_owner_unlinked+0x3b/0x120 [zfs]
+ lp2169003: delay done, ino 128 i_state 0xa5
+ VERIFY3(igrab(ZTOI((zp))) != NULL) failed (0000000000000000 !=
0000000000000000)
+ PANIC at zfs_log.c:257:zfs_xattr_owner_unlinked()
+
+ The kernel trace: zfs_xattr_owner_unlinked / zfs_log_write / zfs_putpage
+ / zpl_writepages / wb_workfn (flush-zfs-*)
+
+ "PASS" is not printed, so it never reaches there.
+ After about 2 minutes, the system hangs.
+ dmesg shows "INFO: task python3 ... blocked for more than 120 seconds" with
inode_wait_for_writeback / evict / iput / do_unlinkat.
+
+ Reboot the VM (or if you have kdump configured it will reboot itself and
+ you can check the crash dump for the stuck traces).
+
+ 5. With the fix:
+
+ If the pool already exists, run steps 2 and 3 again. If it is not
+ imported, run "sudo zpool import -d /var/tmp lp2169003" first or rerun
+ step 1.
+
+ Expected output: We should see that "PASS: unlink and syncfs completed"
+ is printed. No VERIFY, PANIC or hung task in dmesg and no "lp2169003:
+ delaying igrab" line, which is expected: the fixed code no longer calls
+ igrab() when saving, so the probe never fires.
+
+ Also check:
+ sudo zpool sync lp2169003 && sudo zpool status -x
+ It should print "all pools are healthy".
+
+ 6. Clean up:
+ sudo rmmod lp2169003
+ sudo zpool destroy lp2169003
+ sudo rm -f /var/tmp/lp2169003.img
+
+
+ [ Where problems could occur ]
+
+ The change only affects zfs_xattr_owner_unlinked() in
+ module/zfs/zfs_log.c. ZFS calls this function when writing log (ZIL)
+ records: for example for writes, creating or removing files, to decide
+ whether the file (or the owner of an xattr) has already been deleted.
+
+ Holds on files could be counted wrong: too few releases would leak memory,
and the pool or dataset might not export or unmount cleanly or too many
releases could free a file that is still in use. The xattr regression check
above, plus the export/import, covers this.
+ Parent files are now released in the background (zfs_zrele_async) instead of
right away. A file could therefore stay in memory a ittle longer than before
but this should not change what users see.
+ The answer "is this deleted?" could come out wrong. Then ZFS might skip a log
record it should keep, or keep one it should skip, which would show up after a
crash when the log is replayed. The ZIL regression tests should cover this.
+
+ The new code is the same as the FreeBSD code now so there is no
+ Linux/FreeBSD separation now.
+
+ [ Other Info ]
+ Upstream fix:
https://github.com/openzfs/zfs/commit/e9e24ce2c8046c2573d49468e1133770b649018f
+
+ The test kprobe is only for testing.
+
+
+ We observed a kernel panic followed by a ZFS pool hang caused by a race
between dirty mmap page writeback and VFS inode eviction.
The workload is QuestDB, which uses memory-mapped files for its Write-
Ahead Log (WAL). After processing a WAL segment, QuestDB unlinks the
file as part of normal cleanup. In this case, the file was unlinked
while ZFS was still flushing dirty pages belonging to it.
wal-apply_2 was evicting the unlinked inode while the ZFS writeback
thread (flush-zfs-1) was still processing the file. The inode entered
I_FREEING, causing igrab() in zfs_xattr_owner_unlinked() to return NULL
and trigger this assertion:
VERIFY3(igrab(ZTOI((zp))) != NULL)
PANIC at zfs_log.c:257:zfs_xattr_owner_unlinked()
The panic occurred while ZFS had an open DMU transaction, after which
the transaction group could not make progress and the pool hung. The
system required a reboot.
Relevant path:
wb_workfn()
- -> zfs_putpage()
- -> zfs_log_write()
- -> zfs_xattr_owner_unlinked()
- -> igrab()
+ -> zfs_putpage()
+ -> zfs_log_write()
+ -> zfs_xattr_owner_unlinked()
+ -> igrab()
We do not currently have a reliable reproducer.
Expected behaviour
ZFS should handle the inode becoming I_FREEING during this race without
triggering a kernel panic or leaving the ZFS pool unable to make progress.
What happened instead
igrab() returned NULL, the ZFS VERIFY assertion triggered spl_panic(), and
the subsequent failure left the ZFS pool hung until reboot.
#### Environment
Source package: zfs-linux
Ubuntu: 24.04 LTS
Kernel: 6.8.0-138-generic
OpenZFS: 2.2.2-0ubuntu9.4
Architecture: amd64
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2169003
Title:
Panic in zfs_xattr_owner_unlinked() via zhold() on I_FREEING inode
during concurrent mmap writeback
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/2169003/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs