Re: ZFS+find(1) wiring all RAM
On Thu, Jun 07, 2018 at 07:04:29PM +0930, Shane Ambler wrote: > On 07/06/2018 16:09, Peter Jeremy wrote: > > I've noticed that 11-stable/amd64 has been wiring seemingly excessive > > amounts of RAM for some time (the problem goes back at least 6 months). > > This extends to getting ENOMEM errors from g_io_deliver() and out-of-swap > > errors killing processes on a low-memory system. I'm not sure when it > > started by it seems to hawe gotten worse between r331535 and r334494. > > Don't know if this will help you at all -- > > I have seen excess wired for a few years, since 10.1, I now run > 11-stable, my experience has seen the severity varying over time. > > I first reported this 28/10/2014 related to heavy disk use on a zpool. > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194654 > > The inability to get solid repeatable steps to reproduce have prevented > me from chasing this more. Can you try https://reviews.freebsd.org/D7538 ? I am try in this patch to resolve trouble w/ wired and unused memory by ARC. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: ZFS+find(1) wiring all RAM
On 07/06/2018 16:09, Peter Jeremy wrote: > I've noticed that 11-stable/amd64 has been wiring seemingly excessive > amounts of RAM for some time (the problem goes back at least 6 months). > This extends to getting ENOMEM errors from g_io_deliver() and out-of-swap > errors killing processes on a low-memory system. I'm not sure when it > started by it seems to hawe gotten worse between r331535 and r334494. Don't know if this will help you at all -- I have seen excess wired for a few years, since 10.1, I now run 11-stable, my experience has seen the severity varying over time. I first reported this 28/10/2014 related to heavy disk use on a zpool. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194654 The inability to get solid repeatable steps to reproduce have prevented me from chasing this more. My desktop machine originally had 8G and when wired went over 7G there was no choice but to reset. I now have 16G and have a script running constantly to monitor the wired and react when needed, allocating a big chunk of ram normally causes wired to be released, this has kept wired under 10G for several months. I see arc usage sitting at the arc_max setting with wired going more than two times that. My vm.max_wired is at 5G (default) and I think this is a limit that is somehow bypassed, if it could be enforced we might be good. This could be the point to chase in a bug report. > 1) Significant amounts of wired memory are in use but I can't find anything > in "vmstat -mz" that would explain where it's going. I started logging my wired usage and have about 6 months of data, in the following sample, allocating 2G more than free caused wired to drop 4G in about 2 secs, you can also see that arc usage did not change in this time. This leads me to think it may be related to changes made to add support for zfs but I don't think zfs is the one allocating this wired amount. The values I am logging are from - vm.max_wired vm.stats.vm.v_wire_count kstat.zfs.misc.arcstats.size vm.stats.vm.v_free_count 2018/06/06_13:22:53 wired: 9779M max_wired: 5210M ARC: 4.1G Free:3.1G 2018/06/06_13:22:53 wired: 9781M max_wired: 5210M ARC: 4.1G Free:3.0G 2018/06/06_13:22:54 Excess wired detected. Allocating 4.9G to reduce wired. 2018/06/06_13:22:54 wired: 9787M max_wired: 5210M ARC: 4.1G Free:3.0G 2018/06/06_13:22:54 wired: 9796M max_wired: 5210M ARC: 4.1G Free:1.9G 2018/06/06_13:22:55 wired: 9809M max_wired: 5210M ARC: 4.1G Free:389.6M 2018/06/06_13:22:55 wired: 8169M max_wired: 5210M ARC: 4.1G Free:1.2G 2018/06/06_13:22:55 wired: 5917M max_wired: 5210M ARC: 4.1G Free:2.6G 2018/06/06_13:22:55 wired: 5566M max_wired: 5210M ARC: 4.1G Free:2.2G -- Shane Ambler Shane (at) ShaneWare (dot) Biz http://ShaneWare.Biz ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
ZFS+find(1) wiring all RAM
I've noticed that 11-stable/amd64 has been wiring seemingly excessive amounts of RAM for some time (the problem goes back at least 6 months). This extends to getting ENOMEM errors from g_io_deliver() and out-of-swap errors killing processes on a low-memory system. I'm not sure when it started by it seems to hawe gotten worse between r331535 and r334494. I can see the "excessive wired memory" on my main home system with 32GB RAM but haven't seen it completely run out of RAM. After some gentle use and a nightly run, there is 10GB more wired RAM than ARC. My "low memory" system is a Google GCE f1-micro instance[*] (600MB RAM) with about 723k inodes used and the following ZFS tuning: vfs.zfs.arc_max="128M" vfs.zfs.arc_meta_limit="50M" vfs.zfs.arc_min="25M" The following numbers were gatherer by looking at top(1). Running r334494, after booting, to multi-user, the system has about 187MB wired (94MB ARC). If I then run /etc/periodic/security/100.chksetuid, wired RAM increases to about 580MB, with 380MB ARC, dropping to 467MB and 217MB ARC when the script exits (this is still nearly twice arc_max). Free memory can drop to <10KB whilst the find(1) is running. I have several issues with this behaviour: 0) ARC usage can significantly exceed arc_max. I understand that arc_max is a soft limit but IMO, 3x is unreasonable - especially when the system is under extreme memory pressuse. 1) Significant amounts of wired memory are in use but I can't find anything in "vmstat -mz" that would explain where it's going. Does anyone have any suggestions for digging into this? [1] I get the same behaviour using a VBox instance with similar dimensioning and the same tuning) -- Peter Jeremy signature.asc Description: PGP signature