[Bug 1854480] Re: zfs_arc_max not working anymore in zfs 0.8.1
The limit in the code does seem to be 64 MiB. I'm not sure why this isn't working. I am not even close to an expert on this part of OpenZFS, so all I can suggest is to file a bug report upstream: https://github.com/openzfs/zfs/issues/new -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1854480 Title: zfs_arc_max not working anymore in zfs 0.8.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1854480/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1854480] Re: zfs_arc_max not working anymore in zfs 0.8.1
The `zfs_arc_max` setting *is* respected when I set it to a value of `1073741824` = 1GiB ``` cat /etc/modprobe.d/zfs.conf; cat /sys/module/zfs/parameters/zfs_arc_max; cat /proc/spl/kstat/zfs/arcstats |grep ^size -B3 options zfs zfs_arc_max=1073741824 1073741824 c 41073741824 c_min 4519804800 c_max 41073741824 size41086124448 ``` The manpage `man zfs-module-parameters` says ``` zfs_arc_max (ulong) Max arc size of ARC in bytes. If set to 0 then it will consume 1/2 of system RAM. This value must be at least 67108864 (64 megabytes). This value can be changed dynamically with some caveats. It cannot be set back to 0 while running and reducing it below the current ARC size will not cause the ARC to shrink without memory pressure to induce shrinking. Default value: 0. ``` Before I had set it to `134217728`, more than 64MiB, but it was not respected. I also tried setting `zfs_arc_max` to `519804800` (495.7MiB, where ever this number comes from), the value reported as `c_min` and it *not* respected. So, the minimum arc size seems to be around 1GB and *not* 64MB, as the manpage claims. (documentation lies, the code does not) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1854480 Title: zfs_arc_max not working anymore in zfs 0.8.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1854480/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1854480] Re: zfs_arc_max not working anymore in zfs 0.8.1
I would like to reopen this issue. I installed Ubuntu 20.04.1 LTS Server (no zfs root, 16GB RAM) and added zfs volumes. I tried to limit the arc with `options zfs zfs_arc_max=134217728` in `/etc/modprobe.d/zfs.conf` to 128MiB (just to try) I updated the initial ramdisk with `update-initramfs -u` and rebooted the machine. Then I observe, that the arc still grows to ~6GB > 128M. ``` al@nas:~$ cat /etc/modprobe.d/zfs.conf; cat /sys/module/zfs/parameters/zfs_arc_max; cat /proc/spl/kstat/zfs/arcstats options zfs zfs_arc_max=134217728 134217728 12 1 0x01 98 26656 9388908203 858454750163 nametype data hits42341389 misses 469038 demand_data_hits4409755 demand_data_misses 410337 demand_metadata_hits41892418 demand_metadata_misses 411520 prefetch_data_hits 420609 prefetch_data_misses436648 prefetch_metadata_hits 418607 prefetch_metadata_misses410533 mru_hits4332923 mru_ghost_hits 40 mfu_hits41981310 mfu_ghost_hits 40 deleted 421 mutex_miss 40 access_skip 40 evict_skip 499 evict_not_enough40 evict_l2_cached 40 evict_l2_eligible 4201728 evict_l2_ineligible 434816 evict_l2_skip 40 hash_elements 470022 hash_elements_max 470117 hash_collisions 42316 hash_chains 41152 hash_chain_max 42 p 44173713920 c 48316876800 c_min 4519804800 c_max 48316876800 size46496367160 compressed_size 45849741824 uncompressed_size 46240316928 overhead_size 4416625152 hdr_size423995776 data_size 45993091584 metadata_size 4273275392 dbuf_size 444426616 dnode_size 4120015552 bonus_size 441562240 anon_size 418524672 anon_evictable_data 40 anon_evictable_metadata 40 mru_size44014986752 mru_evictable_data 43841052672 mru_evictable_metadata 424693760 mru_ghost_size 40 mru_ghost_evictable_data40 mru_ghost_evictable_metadata40 mfu_size4223282 mfu_evictable_data 41694498304 mfu_evictable_metadata 415691264 mfu_ghost_size 40 mfu_ghost_evictable_data40 mfu_ghost_evictable_metadata40 l2_hits 40 l2_misses 40 l2_feeds40 l2_rw_clash 40 l2_read_bytes 40 l2_write_bytes 40 l2_writes_sent 40 l2_writes_done 40 l2_writes_error 40 l2_writes_lock_retry40 l2_evict_lock_retry 40 l2_evict_reading40 l2_evict_l1cached 40 l2_free_on_write40 l2_abort_lowmem 40 l2_cksum_bad40 l2_io_error 40 l2_size 40 l2_asize40 l2_hdr_size 40 memory_throttle_count 40 memory_direct_count 40 memory_indirect_count 40 memory_all_bytes416633753600 memory_free_bytes 48671588352 memory_available_bytes 38411688960 arc_no_grow 40 arc_tempreserve 40 arc_loaned_bytes40 arc_prune 40 arc_meta_used 4503275576 arc_meta_limit 46237657600 arc_dnode_limit 4623765760 arc_meta_max4509194840 arc_meta_min416777216 async_upgrade_sync 43864 demand_hit_predictive_prefetch 433669 demand_hit_prescient_prefetch 40 arc_need_free 40 arc_sys_free4259902400 arc_raw_size40 ``` -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1854480 Title: zfs_arc_max not working anymore in
[Bug 1854480] Re: zfs_arc_max not working anymore in zfs 0.8.1
** Changed in: zfs-linux (Ubuntu) Status: Incomplete => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1854480 Title: zfs_arc_max not working anymore in zfs 0.8.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1854480/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1854480] Re: zfs_arc_max not working anymore in zfs 0.8.1
OK, close the bug-report. On Thu, 2020-03-26 at 22:47 +, Colin Ian King wrote: > I'll close this bug report if that's OK. > -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1854480 Title: zfs_arc_max not working anymore in zfs 0.8.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1854480/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1854480] Re: zfs_arc_max not working anymore in zfs 0.8.1
I'll close this bug report if that's OK. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1854480 Title: zfs_arc_max not working anymore in zfs 0.8.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1854480/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1854480] Re: zfs_arc_max not working anymore in zfs 0.8.1
I moved to Ubuntu 20.04 and I do not have that problem anymore. I stopped with writing directly after the login the value to the file /sys/module/zfs/parameters/zfs_arc_max. I now only rely on "options zfs zfs_arc_max=3221225472" in /etc/modprobe.d/zfs.conf. Problem solved in 20.04 and I tried it by loading various VMs including Windows 10 and run the updates for those VMs. The L1ARC size never exceeded 3.01 GB of the 3.00 GB I allowed. I knew it would exceed for a short period the max value, but in 19.10 at the time of the bug report, it did exceed it sometimes with 1 GB or more. I use and used conky to display L1ARC values constantly; L1ARC size uncompressed; L1ARC size compressed; max allowed vale and L1ARC hit rate and I have a tweaked bar display that would allow up to 102% :) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1854480 Title: zfs_arc_max not working anymore in zfs 0.8.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1854480/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1854480] Re: zfs_arc_max not working anymore in zfs 0.8.1
I've experimented with this on eoan and focal (ZFS 0.8.1 and ZFS 0.8.3)
on a 4GB VM image. I set the /etc/modprobe.d/zfs.conf as follows:
options zfs zfs_arc_max=134217728
# 128 MB
And rebooted. I then exercised the zfs with various greps and git logs
on the linux git repository while running:
while true; do cat /proc/spl/kstat/zfs/arcstats | grep -m 1 size | awk
'{ print $3 / (1024 * 1024) }'; sleep 1; done
The output shows that one does get arc sizes greater than the 128MB
while doing a lot of I/O but it settles down to sub 128MB once the I/O
activity is finished:
2.93295
2.93295
2.93295
63.8354
131.395
127.909
148.942
128.498
138.224
145.271
133.244
134.122
138.132
144.814
129.4
134.788
135.666
140.428
143.811
146.734
132.323
136.709
143.225
143.225
143.225
143.225
143.225
143.225
143.225
143.225
143.225
143.225
143.225
143.225
143.225
143.225
143.225
143.225
143.225
143.225
143.225
130.353
130.34
130.34
130.34
138.63
140.16
139.826
139.826
139.826
139.921
139.921
133.32
122.35
115.929
111.941
110.478
111.219
111.219
Two observations:
1. The driver does some sanity checks; if zfs_arc_max is too small (<
64MB) or too large (> physical memory size) then the setting is ignored
and the default is used. One can double check if the setting has been
updated by checking with c_max, e.g.:
grep c_max /proc/spl/kstat/zfs/arcstats
c_max 4134217728
If c_max is not the same as zfs_arc_max then the zfs_arc_max has worked.
Can you sanity check that?
2. With a high amount of throughput the arcstats show that the "size"
stat can overshot the zfs_arc_max setting but once I/O has completed it
will setting down to the threshold set. This setting is not updated
directly and is only sync'd during an arc_kstat update. So maybe it's a
snapshot that maybe transient and possibly not completely up to date.
Keeping the stats exactly up to date would cause a performance
bottleneck, so the stats are aggregated - I believe these values are not
exactly 100% correct as there is a performance impact in gathering and
summing the stats.
Perhaps you could experiment with a low zfs_arc_max setting (e.g. 128MB
134217728) and see if this works OK and then bumping up the value to see
if it stops working; then we can work out why it's going wrong at that
point.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1854480
Title:
zfs_arc_max not working anymore in zfs 0.8.1
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1854480/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1854480] Re: zfs_arc_max not working anymore in zfs 0.8.1
I display it constantly using conky (see attachment) and conky is using
${color}L1ARC / ${color0}Hits%: ${color}${alignr}(${exec cat
/proc/spl/kstat/zfs/arcstats | grep -m 1 uncompressed_size | awk
'{printf "%4.2f",$3/1073741824}'}) ${exec cat
/proc/spl/kstat/zfs/arcstats | grep -m 1 size | awk '{printf
"%4.2f",$3/1073741824}'}GiB of 2.00GiB - ${color0}${exec arc_summary |
grep -m 1 Actual | awk '{print int($8+0.5)}'}%
${color5}${execbar cat /proc/spl/kstat/zfs/arcstats | grep -m 1 size |
awk '{printf int($3/21474836)}'}
** Attachment added: "Screenshot from 2020-03-10 14-44-21.png"
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1854480/+attachment/5335438/+files/Screenshot%20from%202020-03-10%2014-44-21.png
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1854480
Title:
zfs_arc_max not working anymore in zfs 0.8.1
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1854480/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1854480] Re: zfs_arc_max not working anymore in zfs 0.8.1
Just for clarification, how are you determining that zfs is using more memory for the L1ARC? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1854480 Title: zfs_arc_max not working anymore in zfs 0.8.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1854480/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1854480] Re: zfs_arc_max not working anymore in zfs 0.8.1
Yes you did miss something. Of course the value is changed in /etc/modprobe.d/zfs.conf. I changed it myself! However zfs is ignoring that value and is using considerably more memory for L1ARC. If I wanted to limit the size of L1ARC, I had to change the value in /sys/module/zfs/parameters/zfs_arc_max. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1854480 Title: zfs_arc_max not working anymore in zfs 0.8.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1854480/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1854480] Re: zfs_arc_max not working anymore in zfs 0.8.1
Maybe I'm missing something here. I tested this out on an Eoan server: $ dmesg | grep ZFS [ 10.821332] ZFS: Loaded module v0.8.1-1ubuntu14.3, ZFS pool version 5000, ZFS filesystem version 5 $ uname -a Linux eaon-amd64 5.3.0-42-generic #34-Ubuntu SMP Fri Feb 28 05:49:40 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux I added a zfs_arc_max setting into the zfs.conf: cat /etc/modprobe.d/zfs.conf options zfs zfs_arc_max=50331648 rebooted and I get: cat /sys/module/zfs/parameters/zfs_arc_max 50331648 So this is setting the L1ARC size as I expected. Can you explain further the issue you are seeing? ** Changed in: zfs-linux (Ubuntu) Importance: Undecided => Medium ** Changed in: zfs-linux (Ubuntu) Status: New => Incomplete ** Changed in: zfs-linux (Ubuntu) Assignee: (unassigned) => Colin Ian King (colin-king) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1854480 Title: zfs_arc_max not working anymore in zfs 0.8.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1854480/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1854480] Re: zfs_arc_max not working anymore in zfs 0.8.1
** Description changed: In the past I could limit the size of L1ARC by specifying "options zfs zfs_arc_max=3221225472" in /etc/modprobe.d/zfs.conf. I tried even to - fill /sys/module/zfs/parameters/zfs_arc_max directly after login, but - none of those methods limits the size of L1ARC. It worked nicely in zfs - 0.7.x. + fill /sys/module/zfs/parameters/zfs_arc_max directly, but none of those + methods limits the size of L1ARC. It worked nicely in zfs 0.7.x. Nowadays I use a nvme drive with 3400 MB/s read throughput and I see not much difference between e.g booting the system from nvme and rebooting the system from L1ARC. Having a 96-99% hit rate for the L1ARC, I like to free up some memory, thus avoiding some delay of freeing memory from L1ARC while loading a VM. + + UPDATE: + + The system only limits the L1ARC, if we directly after the login write + the zfs_arc_max value to the file + /sys/module/zfs/parameters/zfs_arc_max. ProblemType: Bug DistroRelease: Ubuntu 19.10 Package: zfsutils-linux 0.8.1-1ubuntu14.1 ProcVersionSignature: Ubuntu 5.3.0-23.25-generic 5.3.7 Uname: Linux 5.3.0-23-generic x86_64 NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair ApportVersion: 2.20.11-0ubuntu8.2 Architecture: amd64 CurrentDesktop: ubuntu:GNOME Date: Fri Nov 29 06:19:15 2019 InstallationDate: Installed on 2019-11-25 (3 days ago) InstallationMedia: Ubuntu 19.10 "Eoan Ermine" - Release amd64 (20191017) SourcePackage: zfs-linux UpgradeStatus: No upgrade log present (probably fresh install) modified.conffile..etc.sudoers.d.zfs: [inaccessible: [Errno 13] Permission denied: '/etc/sudoers.d/zfs'] ** Description changed: In the past I could limit the size of L1ARC by specifying "options zfs zfs_arc_max=3221225472" in /etc/modprobe.d/zfs.conf. I tried even to fill /sys/module/zfs/parameters/zfs_arc_max directly, but none of those methods limits the size of L1ARC. It worked nicely in zfs 0.7.x. Nowadays I use a nvme drive with 3400 MB/s read throughput and I see not much difference between e.g booting the system from nvme and rebooting the system from L1ARC. Having a 96-99% hit rate for the L1ARC, I like to free up some memory, thus avoiding some delay of freeing memory from L1ARC while loading a VM. + - UPDATE: The system only limits the L1ARC, if we directly after the login write the zfs_arc_max value to the file /sys/module/zfs/parameters/zfs_arc_max. + + ProblemType: Bug DistroRelease: Ubuntu 19.10 Package: zfsutils-linux 0.8.1-1ubuntu14.1 ProcVersionSignature: Ubuntu 5.3.0-23.25-generic 5.3.7 Uname: Linux 5.3.0-23-generic x86_64 NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair ApportVersion: 2.20.11-0ubuntu8.2 Architecture: amd64 CurrentDesktop: ubuntu:GNOME Date: Fri Nov 29 06:19:15 2019 InstallationDate: Installed on 2019-11-25 (3 days ago) InstallationMedia: Ubuntu 19.10 "Eoan Ermine" - Release amd64 (20191017) SourcePackage: zfs-linux UpgradeStatus: No upgrade log present (probably fresh install) modified.conffile..etc.sudoers.d.zfs: [inaccessible: [Errno 13] Permission denied: '/etc/sudoers.d/zfs'] -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1854480 Title: zfs_arc_max not working anymore in zfs 0.8.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1854480/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
