Am 2026-03-03 23:45, schrieb Doug Ambrisko:
On Tue, Mar 03, 2026 at 02:25:11PM -0800, Rick Macklem wrote:| On Tue, Mar 3, 2026 at 12:33 PM Doug Ambrisko <[email protected]> wrote:| >| > On Sun, Nov 02, 2025 at 11:48:06AM +0100, Alexander Leidinger wrote:| > | Am 2025-10-29 22:06, schrieb Doug Ambrisko:| > | > It seems around the switch to OpenZFS I would have arc clean task| > | > running| > | > 100% on a core. I use nullfs on my laptop to map my shared ZFS /data | > | > partiton into a few vnet instances. Over night or so I would get into | > | > this issue. I found that I had a bunch of vnodes being held by other | > | > layers. My solution was to reduce kern.maxvnodes and vfs.zfs.arc.max so | > | > the ARC cache stayed reasonable without killing other applications.| > | >| > | > That is why a while back I added the vnode count to mount -v so that | > | > I could see the usage of vnodes for each mount point. I made a script| > | > to report on things: | > | | > | Do you see this also with the nullfs mount option "nocache"? | > | > I seems to have run into this issue with nocache| > /data/jail/current/usr/local/etc/cups /data/jail/current-other/usr/local/etc/cups nullfs rw,nocache 0 0 | > /data/jail/current/usr/local/etc/sane.d /data/jail/current-other/usr/local/etc/sane.d nullfs rw,nocache 0 0 | > /data/jail/current/usr/local/www /data/jail/current-other/usr/local/www nullfs rw,nocache 0 0 | > /data/jail/current/usr/local/etc/nginx /data/jail/current-other/usr/local/etc/nginx nullfs rw,nocache 0 0 | > /data/jail/current/tftpboot /data/jail/current-other/tftpboot nullfs rw,nocache 0 0 | > /data/jail/current/usr/local/lib/grub /data/jail/current-other/usr/local/lib/grub nullfs rw,nocache 0 0 | > /data/jail /data/jail/current-other/data/jail nullfs rw,nocache 0 0 | > /data/jail /data/jail/current/data/jail nullfs rw,nocache 0 0| >| > After a while (a couple of months or more). My laptop was running slow | > with a high load. The perodic find was running slow. arc_prunee was | > spinning. When I reduced the number of vnodes then things got better. | > My vfs.zfs.arc_max is 1073741824 so that I have memory for other things.| > | > nocache does help taking longer to get into this situation. | Have any of you guys tried increasing vfs.zfs.arc.free_target? || If I understand the code correctly, when freemem < vfs.zfs.arc.free_target | the reaper thread (the one that does uma_zone_reclaim() to return pages | to the system from the uma keg that the arc uses) should be activated.I haven't tried that. I set: kern.maxvnodes vfs.zfs.arc.min vfs.zfs.arc.max vfs.zfs.prefetch.disable=1 I need to make sure kern.maxvnodes is small enough so it doesn't thrash when vfs.zfs.arc.max set to 1G. The issues tend to take a while to happen. On the plus side I can adjust these when I hit them mostly by reducing kern.maxvnodes without having to do a reboot.
There was this commit recently_https://cgit.freebsd.org/src/commit/sys/fs/nullfs?id=8b64d46fab87af3ae062901312187f3a04ad2d67
I have not checked if this race condition could result in anything related to what we see. From the commit message I can not deduct if this could for example lead to a (even temporary) resource leak which may explain this behavior. Mark, what is the high-level result of this race condition you fixed in nullfs? At first look at the commit log I would rather assume vnodes of the lower FS could rather be freed more early and not at all because of the race condition.
Bye, Alexander. -- http://www.Leidinger.net [email protected]: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org [email protected] : PGP 0x8F31830F9F2772BF
signature.asc
Description: OpenPGP digital signature
