Re: Freebsd 8.0 kmem map too small
On Fri, Jun 11, 2010 at 04:12:50PM +0200, Florian Smeets wrote: On 6/8/10 11:11 AM, Yoshiaki Kasahara wrote: Hello, I'd like to add another instance of similar problems. I recently updated my FreeBSD amd64 box with ZFS root and 8GB RAM from 8-STABLE (as of Mar 1st) to 8.1-PRERELEASE (as of May 27th). After that, my box started to crash every couple of days due to kmem_map too small. Here is a (last week) screenshot of Munin graph about the memory usage of the box: http://eron.info/munin-memory.png In by month graph, a white gap at the end of Week 20 is the update period from 8-STABLE to 8.1-PRERELEASE I mentioned above. Before the upgrade, the system was rock solid without any kmem tuning in loader.conf (vm.kmem_size was around 2.7GB IIRC). After the update, I could see that more wired memory was assigned, and then steep drop (crash) occured. by day graph shows my experiment to bump vm.kmem_size=12G (recommended somewhere in this thread) and explicitly limit vfs.zfs.arc_max=2G. I was surprised because the wired memory quickly increased over 5GB... Then I noticed that the default value of vfs.zfs.zio.use_uma was 1 on amd64, so I turned it off and removed other memory tunings (kmem_size and arc_max) in loader.conf on Tuesday at 16:00 and rebooted. It seems that the usage of wired memory was stablized and no crash since then with the default kmem_size and arc_max. Does anyone have any idea about this behavior? My wild guess is that ZFS/UMA code is not stable on amd64 too, and it should be turned off by default for 8.1-RELEASE maybe I can confirm this. I just upgraded a system from late January to yesterdays 8-STABLE. I had to set vfs.zfs.zio.use_uma=0, to get the system survive buildworld again. This is amd64 with only 1GB, (i know this is considered not enough for ZFS) with the kernel from January it was stable without any tuning. After disabling ZFSs UMA usage, it seems to be working fine again. Just another data point. +1 for me, several of my servers crash *every night* on a script that does csup from cvs.freebsd.org (cvs mode if that matters) followed by cvs up -dP in some source trees and one ports tree. Hardware has 12 GB RAM, dual nehalem processors; kernel is amd64, no sysctl tuning related to ZFS. With kernels before april 18 everything is rock solid here, one production server has been up since april 17. Can do some limited testing on one of these servers on mondays if needed. Cheers, Florian Regards, Paul Schenkeveld ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
On 6/8/10 11:11 AM, Yoshiaki Kasahara wrote: Hello, I'd like to add another instance of similar problems. I recently updated my FreeBSD amd64 box with ZFS root and 8GB RAM from 8-STABLE (as of Mar 1st) to 8.1-PRERELEASE (as of May 27th). After that, my box started to crash every couple of days due to kmem_map too small. Here is a (last week) screenshot of Munin graph about the memory usage of the box: http://eron.info/munin-memory.png In by month graph, a white gap at the end of Week 20 is the update period from 8-STABLE to 8.1-PRERELEASE I mentioned above. Before the upgrade, the system was rock solid without any kmem tuning in loader.conf (vm.kmem_size was around 2.7GB IIRC). After the update, I could see that more wired memory was assigned, and then steep drop (crash) occured. by day graph shows my experiment to bump vm.kmem_size=12G (recommended somewhere in this thread) and explicitly limit vfs.zfs.arc_max=2G. I was surprised because the wired memory quickly increased over 5GB... Then I noticed that the default value of vfs.zfs.zio.use_uma was 1 on amd64, so I turned it off and removed other memory tunings (kmem_size and arc_max) in loader.conf on Tuesday at 16:00 and rebooted. It seems that the usage of wired memory was stablized and no crash since then with the default kmem_size and arc_max. Does anyone have any idea about this behavior? My wild guess is that ZFS/UMA code is not stable on amd64 too, and it should be turned off by default for 8.1-RELEASE maybe I can confirm this. I just upgraded a system from late January to yesterdays 8-STABLE. I had to set vfs.zfs.zio.use_uma=0, to get the system survive buildworld again. This is amd64 with only 1GB, (i know this is considered not enough for ZFS) with the kernel from January it was stable without any tuning. After disabling ZFSs UMA usage, it seems to be working fine again. Just another data point. Cheers, Florian ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
Hello, I'd like to add another instance of similar problems. I recently updated my FreeBSD amd64 box with ZFS root and 8GB RAM from 8-STABLE (as of Mar 1st) to 8.1-PRERELEASE (as of May 27th). After that, my box started to crash every couple of days due to kmem_map too small. Here is a (last week) screenshot of Munin graph about the memory usage of the box: http://eron.info/munin-memory.png In by month graph, a white gap at the end of Week 20 is the update period from 8-STABLE to 8.1-PRERELEASE I mentioned above. Before the upgrade, the system was rock solid without any kmem tuning in loader.conf (vm.kmem_size was around 2.7GB IIRC). After the update, I could see that more wired memory was assigned, and then steep drop (crash) occured. by day graph shows my experiment to bump vm.kmem_size=12G (recommended somewhere in this thread) and explicitly limit vfs.zfs.arc_max=2G. I was surprised because the wired memory quickly increased over 5GB... Then I noticed that the default value of vfs.zfs.zio.use_uma was 1 on amd64, so I turned it off and removed other memory tunings (kmem_size and arc_max) in loader.conf on Tuesday at 16:00 and rebooted. It seems that the usage of wired memory was stablized and no crash since then with the default kmem_size and arc_max. Does anyone have any idea about this behavior? My wild guess is that ZFS/UMA code is not stable on amd64 too, and it should be turned off by default for 8.1-RELEASE maybe Regards, -- Yoshiaki Kasahara Research Institute for Information Technology, Kyushu University kasah...@nc.kyushu-u.ac.jp ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
On Tue, Jun 08, 2010 at 06:11:46PM +0900, Yoshiaki Kasahara wrote: Hello, I'd like to add another instance of similar problems. I recently updated my FreeBSD amd64 box with ZFS root and 8GB RAM from 8-STABLE (as of Mar 1st) to 8.1-PRERELEASE (as of May 27th). After that, my box started to crash every couple of days due to kmem_map too small. Here is a (last week) screenshot of Munin graph about the memory usage of the box: http://eron.info/munin-memory.png In by month graph, a white gap at the end of Week 20 is the update period from 8-STABLE to 8.1-PRERELEASE I mentioned above. Before the upgrade, the system was rock solid without any kmem tuning in loader.conf (vm.kmem_size was around 2.7GB IIRC). After the update, I could see that more wired memory was assigned, and then steep drop (crash) occured. by day graph shows my experiment to bump vm.kmem_size=12G (recommended somewhere in this thread) and explicitly limit vfs.zfs.arc_max=2G. I was surprised because the wired memory quickly increased over 5GB... Then I noticed that the default value of vfs.zfs.zio.use_uma was 1 on amd64, so I turned it off and removed other memory tunings (kmem_size and arc_max) in loader.conf on Tuesday at 16:00 and rebooted. It seems that the usage of wired memory was stablized and no crash since then with the default kmem_size and arc_max. Does anyone have any idea about this behavior? My wild guess is that ZFS/UMA code is not stable on amd64 too, and it should be turned off by default for 8.1-RELEASE maybe I realise you're talking about amd64, but I'm not sure how UMA is getting enabled on i386 to begin with. It does look like it's enabled on amd64 by default. src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c -- #if defined(__amd64__) static int zio_use_uma = 1; #else static int zio_use_uma = 0; #endif SYSCTL_DECL(_vfs_zfs); SYSCTL_NODE(_vfs_zfs, OID_AUTO, zio, CTLFLAG_RW, 0, ZFS ZIO); TUNABLE_INT(vfs.zfs.zio.use_uma, zio_use_uma); SYSCTL_INT(_vfs_zfs_zio, OID_AUTO, use_uma, CTLFLAG_RDTUN, zio_use_uma, 0, Use uma(9) for ZIO allocations); Also note that this tunable does not appear in sysctl -a output; I believe this is caused by the first parameter to SYSCTL_INT() being incorrect -- should be _vfs_zfs, not _vfs_zfs_zio. Regardless of that, it's still adjustable in loader.conf. -- | Jeremy Chadwick j...@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
On Tue, Jun 08, 2010 at 02:54:44AM -0700, Jeremy Chadwick wrote: SYSCTL_DECL(_vfs_zfs); SYSCTL_NODE(_vfs_zfs, OID_AUTO, zio, CTLFLAG_RW, 0, ZFS ZIO); TUNABLE_INT(vfs.zfs.zio.use_uma, zio_use_uma); SYSCTL_INT(_vfs_zfs_zio, OID_AUTO, use_uma, CTLFLAG_RDTUN, zio_use_uma, 0, Use uma(9) for ZIO allocations); Also note that this tunable does not appear in sysctl -a output; I believe this is caused by the first parameter to SYSCTL_INT() being incorrect -- should be _vfs_zfs, not _vfs_zfs_zio. Regardless of that, it's still adjustable in loader.conf. Why? I create vfs.zfs.zio node and create vfs.zfs.zio.use_uma sysctl to match tunable name. -- Pawel Jakub Dawidek http://www.wheelsystems.com p...@freebsd.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! pgpu2PnW3a0pF.pgp Description: PGP signature
Re: Freebsd 8.0 kmem map too small
On Tue, 8 Jun 2010 02:54:44 -0700, Jeremy Chadwick free...@jdc.parodius.com said: I realise you're talking about amd64, but I'm not sure how UMA is getting enabled on i386 to begin with. It does look like it's enabled on amd64 by default. I believe that this thread had been started before this tunable was introduced on May 24th. Before that, ZIO_USE_UMA was used to control the use of UMA, and it was enabled unconditionally in /usr/src/sys/modules/zfs. -- Yoshiaki Kasahara Research Institute for Information Technology, Kyushu University kasah...@nc.kyushu-u.ac.jp ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
On Tue, Jun 08, 2010 at 12:20:36PM +0200, Pawel Jakub Dawidek wrote: On Tue, Jun 08, 2010 at 02:54:44AM -0700, Jeremy Chadwick wrote: SYSCTL_DECL(_vfs_zfs); SYSCTL_NODE(_vfs_zfs, OID_AUTO, zio, CTLFLAG_RW, 0, ZFS ZIO); TUNABLE_INT(vfs.zfs.zio.use_uma, zio_use_uma); SYSCTL_INT(_vfs_zfs_zio, OID_AUTO, use_uma, CTLFLAG_RDTUN, zio_use_uma, 0, Use uma(9) for ZIO allocations); Also note that this tunable does not appear in sysctl -a output; I believe this is caused by the first parameter to SYSCTL_INT() being incorrect -- should be _vfs_zfs, not _vfs_zfs_zio. Regardless of that, it's still adjustable in loader.conf. Why? I create vfs.zfs.zio node and create vfs.zfs.zio.use_uma sysctl to match tunable name. Sorry, I was looking at some example code that used _vfs_zfs, not realising the node delimiter is based on the underscore. So to recap, vfs.zfs.zio.use_uma doesn't show up in sysctl output. There are other pieces of code which declare their own node as well, and those do appear. So the question is why this specific tunable doesn't. Looking through comparative code (specifically sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_cache.c) -- 81 SYSCTL_DECL(_vfs_zfs_vdev); 82 SYSCTL_NODE(_vfs_zfs_vdev, OID_AUTO, cache, CTLFLAG_RW, 0, ZFS VDEV Cache); 83 TUNABLE_INT(vfs.zfs.vdev.cache.max, zfs_vdev_cache_max); 84 SYSCTL_INT(_vfs_zfs_vdev_cache, OID_AUTO, max, CTLFLAG_RDTUN, 85 zfs_vdev_cache_max, 0, Maximum I/O request size that increase read size); 86 TUNABLE_INT(vfs.zfs.vdev.cache.size, zfs_vdev_cache_size); 87 SYSCTL_INT(_vfs_zfs_vdev_cache, OID_AUTO, size, CTLFLAG_RDTUN, 88 zfs_vdev_cache_size, 0, Size of VDEV cache); 89 TUNABLE_INT(vfs.zfs.vdev.cache.bshift, zfs_vdev_cache_bshift); 90 SYSCTL_INT(_vfs_zfs_vdev_cache, OID_AUTO, bshift, CTLFLAG_RDTUN, 91 zfs_vdev_cache_bshift, 0, Turn too small requests into 1 this value); While for sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c -- 41 SYSCTL_DECL(_vfs_zfs); 42 SYSCTL_NODE(_vfs_zfs, OID_AUTO, zio, CTLFLAG_RW, 0, ZFS ZIO); 43 TUNABLE_INT(vfs.zfs.zio.use_uma, zio_use_uma); 44 SYSCTL_INT(_vfs_zfs_zio, OID_AUTO, use_uma, CTLFLAG_RDTUN, zio_use_uma, 0, 45 Use uma(9) for ZIO allocations); I think lines 41 and 42 here are incorrect and should have _vfs_zfs_zio as their first parameter? -- | Jeremy Chadwick j...@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
So to recap, vfs.zfs.zio.use_uma doesn't show up in sysctl output. Errr, does for me $ sysctl -a | grep vfs.zfs.zio.use_uma vfs.zfs.zio.use_uma: 1 Thats from 8.1-PRERELEASE on June 2nd. ...but all the question of sysctls is a bit of a red herring to me ?I'm more intyerested in whether we really need to disable UMA to get stable ZFS on amd64 right now. -pete. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
On Tue, Jun 08, 2010 at 07:26:45PM +0900, Yoshiaki Kasahara wrote: On Tue, 8 Jun 2010 02:54:44 -0700, Jeremy Chadwick free...@jdc.parodius.com said: I realise you're talking about amd64, but I'm not sure how UMA is getting enabled on i386 to begin with. It does look like it's enabled on amd64 by default. I believe that this thread had been started before this tunable was introduced on May 24th. Before that, ZIO_USE_UMA was used to control the use of UMA, and it was enabled unconditionally in /usr/src/sys/modules/zfs. Yoshiaki, thanks for the tip. I didn't check CVS commit logs to see when this tunable was introduced. It appears that the systems I'm looking at don't have the use_uma tunable because the kernel was built from RELENG_8 code dated May 23rd. This would also explain why Pete sees the tunable and I don't. Wish this stuff was documented somewhere. :-) -- | Jeremy Chadwick j...@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
On 5/5/10 11:19 AM, Freddie Cash wrote: On Tue, May 4, 2010 at 11:32 PM, Giulio Ferroau...@zirakzigil.org wrote: Giulio Ferro wrote: Thanks, I'll try these settings. I'll keep you posted. Nope, it's happened again... Now I've tried to rise vm.kmem_size to 6G... I'm really astounded at how unstable zfs is, it's causing me a lot of problem. Why isn't it stated in the handbook that zfs isn't up to production yet? As with everything related to computers, it all depends on your uses. Sorry to semi-hijack this, but... I'm also running into frequent kmem_map too small panics on 8-STABLE, such as: panic: kmem_malloc(131072): kmem_map too small: 2023780352 total allocated panic: kmem_malloc(131072): kmem_map too small: 2011525120 total allocated panic: kmem_malloc(114688): kmem_map too small: 1849356288 total allocated panic: kmem_malloc(114688): kmem_map too small: 1849356288 total allocated panic: kmem_malloc(114688): kmem_map too small: 1849356288 total allocated panic: kmem_malloc(131072): kmem_map too small: 2020409344 total allocated panic: kmem_malloc(536576): kmem_map too small: 2022957056 total allocated (those are over the course of 3-4 days) On this specific system, it has 32 GB physical memory and has vfs.zfs.arc_max=2G and vm.kmem_size=64G in /boot/loader.conf. The latter was added per earlier suggestions on this list, but appears to be ignored as sysctl vm.kmem_size returns about 2 GB (2172452864) anyway. Unfortunately I have not yet found a way to reliably reproduce the panic on demand, so it's hard to iteratively narrow down which date the potentially offending commit was on. It happened at least twice overnight, where several memory-intensive jobs run. Backing out to a previous 8-STABLE kernel from March 28 appears (so far) to have stabilized things, so the offending code was likely somewhere between then and May 5 or so. I do have KDB/DDB and serial console on this box, if there's any more info I can give to help troubleshoot other than just a one-month vague date range :) This is happening to more than just one system, but I figure it's easier to troubleshoot memory settings on the 32 GB i7 server instead of the 2 GB Atom one... ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
On Mon, May 10, 2010 at 11:55:09AM -0400, Mike Andrews wrote: Sorry to semi-hijack this, but... I'm also running into frequent kmem_map too small panics on 8-STABLE, such as: panic: kmem_malloc(131072): kmem_map too small: 2023780352 total allocated panic: kmem_malloc(131072): kmem_map too small: 2011525120 total allocated panic: kmem_malloc(114688): kmem_map too small: 1849356288 total allocated panic: kmem_malloc(114688): kmem_map too small: 1849356288 total allocated panic: kmem_malloc(114688): kmem_map too small: 1849356288 total allocated panic: kmem_malloc(131072): kmem_map too small: 2020409344 total allocated panic: kmem_malloc(536576): kmem_map too small: 2022957056 total allocated (those are over the course of 3-4 days) On this specific system, it has 32 GB physical memory and has vfs.zfs.arc_max=2G and vm.kmem_size=64G in /boot/loader.conf. The latter was added per earlier suggestions on this list, but appears to be ignored as sysctl vm.kmem_size returns about 2 GB (2172452864) anyway. ... In your case, vm.kmem_size being stuck at 2GB (despite you setting it larger) is the problem. A workaround would be for you to set vfs.zfs.arc_max to something smaller, such as 1G or 512M. Can you please provide output from the following 5 commands: uname -a sysctl vm.kmem_size_min vm.kmem_size_max vm.kmem_size vm.kmem_size_scale sysctl hw.physmem hw.usermem hw.realmem sysctl hw.machine_arch sysctl hw.pagesize hw.pagesizes hw.availpages Thanks. -- | Jeremy Chadwick j...@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
On this specific system, it has 32 GB physical memory and has vfs.zfs.arc_max=2G and vm.kmem_size=64G in /boot/loader.conf. The latter was added per earlier suggestions on this list, but appears to be ignored as sysctl vm.kmem_size returns about 2 GB (2172452864) anyway. Set vm.kmem_size to slightly below 2x the amount of physical memory your kernel *sees* (sysctl hw.physmem) . Chances are that real amount of physical memory available to kernel is slightly below 32G so your tunable is ignored. My guess would be that vm.kmem_size=63G would work much better. --Artem On Mon, May 10, 2010 at 8:55 AM, Mike Andrews mandr...@bit0.com wrote: On 5/5/10 11:19 AM, Freddie Cash wrote: On Tue, May 4, 2010 at 11:32 PM, Giulio Ferroau...@zirakzigil.org wrote: Giulio Ferro wrote: Thanks, I'll try these settings. I'll keep you posted. Nope, it's happened again... Now I've tried to rise vm.kmem_size to 6G... I'm really astounded at how unstable zfs is, it's causing me a lot of problem. Why isn't it stated in the handbook that zfs isn't up to production yet? As with everything related to computers, it all depends on your uses. Sorry to semi-hijack this, but... I'm also running into frequent kmem_map too small panics on 8-STABLE, such as: panic: kmem_malloc(131072): kmem_map too small: 2023780352 total allocated panic: kmem_malloc(131072): kmem_map too small: 2011525120 total allocated panic: kmem_malloc(114688): kmem_map too small: 1849356288 total allocated panic: kmem_malloc(114688): kmem_map too small: 1849356288 total allocated panic: kmem_malloc(114688): kmem_map too small: 1849356288 total allocated panic: kmem_malloc(131072): kmem_map too small: 2020409344 total allocated panic: kmem_malloc(536576): kmem_map too small: 2022957056 total allocated (those are over the course of 3-4 days) On this specific system, it has 32 GB physical memory and has vfs.zfs.arc_max=2G and vm.kmem_size=64G in /boot/loader.conf. The latter was added per earlier suggestions on this list, but appears to be ignored as sysctl vm.kmem_size returns about 2 GB (2172452864) anyway. Unfortunately I have not yet found a way to reliably reproduce the panic on demand, so it's hard to iteratively narrow down which date the potentially offending commit was on. It happened at least twice overnight, where several memory-intensive jobs run. Backing out to a previous 8-STABLE kernel from March 28 appears (so far) to have stabilized things, so the offending code was likely somewhere between then and May 5 or so. I do have KDB/DDB and serial console on this box, if there's any more info I can give to help troubleshoot other than just a one-month vague date range :) This is happening to more than just one system, but I figure it's easier to troubleshoot memory settings on the 32 GB i7 server instead of the 2 GB Atom one... ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
On 05/10/10 11:55, Mike Andrews wrote: On 5/5/10 11:19 AM, Freddie Cash wrote: On Tue, May 4, 2010 at 11:32 PM, Giulio Ferroau...@zirakzigil.org wrote: Giulio Ferro wrote: Thanks, I'll try these settings. I'll keep you posted. Nope, it's happened again... Now I've tried to rise vm.kmem_size to 6G... I'm really astounded at how unstable zfs is, it's causing me a lot of problem. Why isn't it stated in the handbook that zfs isn't up to production yet? As with everything related to computers, it all depends on your uses. Sorry to semi-hijack this, but... I'm also running into frequent kmem_map too small panics on 8-STABLE, such as: panic: kmem_malloc(131072): kmem_map too small: 2023780352 total allocated panic: kmem_malloc(131072): kmem_map too small: 2011525120 total allocated panic: kmem_malloc(114688): kmem_map too small: 1849356288 total allocated panic: kmem_malloc(114688): kmem_map too small: 1849356288 total allocated panic: kmem_malloc(114688): kmem_map too small: 1849356288 total allocated panic: kmem_malloc(131072): kmem_map too small: 2020409344 total allocated panic: kmem_malloc(536576): kmem_map too small: 2022957056 total allocated (those are over the course of 3-4 days) On this specific system, it has 32 GB physical memory and has vfs.zfs.arc_max=2G and vm.kmem_size=64G in /boot/loader.conf. The latter was added per earlier suggestions on this list, but appears to be ignored as sysctl vm.kmem_size returns about 2 GB (2172452864) anyway. As Artem stated in another reply, you will need to set vm.kmem_size slightly under 2x the physical memory. The kernel will default to 2GB if you pass this limit. 1.5x physical memory size should be sufficient, so try 48G and verify that it gets set correctly on the next boot. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
On Mon, 10 May 2010, Steve Polyack wrote: On 05/10/10 11:55, Mike Andrews wrote: On 5/5/10 11:19 AM, Freddie Cash wrote: On Tue, May 4, 2010 at 11:32 PM, Giulio Ferroau...@zirakzigil.org wrote: Giulio Ferro wrote: Thanks, I'll try these settings. I'll keep you posted. Nope, it's happened again... Now I've tried to rise vm.kmem_size to 6G... I'm really astounded at how unstable zfs is, it's causing me a lot of problem. Why isn't it stated in the handbook that zfs isn't up to production yet? As with everything related to computers, it all depends on your uses. Sorry to semi-hijack this, but... I'm also running into frequent kmem_map too small panics on 8-STABLE, such as: panic: kmem_malloc(131072): kmem_map too small: 2023780352 total allocated panic: kmem_malloc(131072): kmem_map too small: 2011525120 total allocated panic: kmem_malloc(114688): kmem_map too small: 1849356288 total allocated panic: kmem_malloc(114688): kmem_map too small: 1849356288 total allocated panic: kmem_malloc(114688): kmem_map too small: 1849356288 total allocated panic: kmem_malloc(131072): kmem_map too small: 2020409344 total allocated panic: kmem_malloc(536576): kmem_map too small: 2022957056 total allocated (those are over the course of 3-4 days) On this specific system, it has 32 GB physical memory and has vfs.zfs.arc_max=2G and vm.kmem_size=64G in /boot/loader.conf. The latter was added per earlier suggestions on this list, but appears to be ignored as sysctl vm.kmem_size returns about 2 GB (2172452864) anyway. As Artem stated in another reply, you will need to set vm.kmem_size slightly under 2x the physical memory. The kernel will default to 2GB if you pass this limit. 1.5x physical memory size should be sufficient, so try 48G and verify that it gets set correctly on the next boot. OK, I've got vm.kmem_size set a bit lower and it now accepts it. It's still not clear why this just recently (April?) became necessary to do at all :) Meanwhile, I'll see if things get more stable now... ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
vm.kmem_size limitation has been this way for a pretty long time. What's changed recently is that ZFS ARC now uses UMA for its memory allocations. If I understand it correctly, this would make ARC's memory use more efficient as allocated chunks will end up in a zone tuned for allocations of particular size. Increased fragmentation could be the side effect of this change, but I'm guessing here. --Artem On Mon, May 10, 2010 at 1:45 PM, Mike Andrews mandr...@bit0.com wrote: On Mon, 10 May 2010, Steve Polyack wrote: On 05/10/10 11:55, Mike Andrews wrote: On 5/5/10 11:19 AM, Freddie Cash wrote: On Tue, May 4, 2010 at 11:32 PM, Giulio Ferroau...@zirakzigil.org wrote: Giulio Ferro wrote: Thanks, I'll try these settings. I'll keep you posted. Nope, it's happened again... Now I've tried to rise vm.kmem_size to 6G... I'm really astounded at how unstable zfs is, it's causing me a lot of problem. Why isn't it stated in the handbook that zfs isn't up to production yet? As with everything related to computers, it all depends on your uses. Sorry to semi-hijack this, but... I'm also running into frequent kmem_map too small panics on 8-STABLE, such as: panic: kmem_malloc(131072): kmem_map too small: 2023780352 total allocated panic: kmem_malloc(131072): kmem_map too small: 2011525120 total allocated panic: kmem_malloc(114688): kmem_map too small: 1849356288 total allocated panic: kmem_malloc(114688): kmem_map too small: 1849356288 total allocated panic: kmem_malloc(114688): kmem_map too small: 1849356288 total allocated panic: kmem_malloc(131072): kmem_map too small: 2020409344 total allocated panic: kmem_malloc(536576): kmem_map too small: 2022957056 total allocated (those are over the course of 3-4 days) On this specific system, it has 32 GB physical memory and has vfs.zfs.arc_max=2G and vm.kmem_size=64G in /boot/loader.conf. The latter was added per earlier suggestions on this list, but appears to be ignored as sysctl vm.kmem_size returns about 2 GB (2172452864) anyway. As Artem stated in another reply, you will need to set vm.kmem_size slightly under 2x the physical memory. The kernel will default to 2GB if you pass this limit. 1.5x physical memory size should be sufficient, so try 48G and verify that it gets set correctly on the next boot. OK, I've got vm.kmem_size set a bit lower and it now accepts it. It's still not clear why this just recently (April?) became necessary to do at all :) Meanwhile, I'll see if things get more stable now... ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
On Wed, May 05, 2010 at 03:33:02PM +0200, Pawel Jakub Dawidek wrote: On Wed, May 05, 2010 at 10:46:31AM +0200, Giulio Ferro wrote: On 05.05.2010 09:52, Jeremy Chadwick wrote: Nope, it's happened again... Now I've tried to rise vm.kmem_size to 6G... [ ... ] Could you try to track down the commit that is causing your problems? Could you try 8-STABLE kernel from before r206815? A quick note to say same here, but on i386. FreeBSD 8.0-STABLE as of 8/5/2010 paniced last night with same symptoms, approx 48 hours uptime. Previous kernel was FreeBSD 8.0-STABLE from Sun Mar 7 14:31:45 EST 2010, perfectly stable for intervening 2 months, about 2 months uptime. Please let me know if full details would help (as opposed to just adding noise :-) -- Richard Perini Internet: r...@ci.com.au Corinthian Engineering Pty Ltd PHONE: +61 2 9552 5500 Sydney, AustraliaFAX: +61 2 9552 5549 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
You can try disabling ZIO_USE_UMA in sys/modules/zfs/Makefile Comment out following line in that file: CFLAGS+=-DZIO_USE_UMA This should revert memory allocation method back to its previous mode. Let us know whether it helps or not. --Artem On Mon, May 10, 2010 at 4:14 PM, Richard Perini r...@ci.com.au wrote: On Wed, May 05, 2010 at 03:33:02PM +0200, Pawel Jakub Dawidek wrote: On Wed, May 05, 2010 at 10:46:31AM +0200, Giulio Ferro wrote: On 05.05.2010 09:52, Jeremy Chadwick wrote: Nope, it's happened again... Now I've tried to rise vm.kmem_size to 6G... [ ... ] Could you try to track down the commit that is causing your problems? Could you try 8-STABLE kernel from before r206815? A quick note to say same here, but on i386. FreeBSD 8.0-STABLE as of 8/5/2010 paniced last night with same symptoms, approx 48 hours uptime. Previous kernel was FreeBSD 8.0-STABLE from Sun Mar 7 14:31:45 EST 2010, perfectly stable for intervening 2 months, about 2 months uptime. Please let me know if full details would help (as opposed to just adding noise :-) -- Richard Perini Internet: ...@ci.com.au Corinthian Engineering Pty Ltd PHONE: +61 2 9552 5500 Sydney, Australia FAX: +61 2 9552 5549 ___ freebsd...@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to freebsd-fs-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
On Mon, May 10, 2010 at 04:43:26PM -0700, Artem Belevich wrote: You can try disabling ZIO_USE_UMA in sys/modules/zfs/Makefile Comment out following line in that file: CFLAGS+=-DZIO_USE_UMA This should revert memory allocation method back to its previous mode. Let us know whether it helps or not. Thanks Artem, I'll try the suggestion and report back. I'll give it 72 hours. The workload of the machine is fairly consistent day to day. Cheers, -- Richard Perini Internet: r...@ci.com.au Corinthian Engineering Pty Ltd PHONE: +61 2 9552 5500 Sydney, AustraliaFAX: +61 2 9552 5549 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
On 5 May 2010, at 15:33, Pawel Jakub Dawidek wrote: Could you try to track down the commit that is causing your problems? Could you try 8-STABLE kernel from before r206815? I do suspect something similar like that, and try to roll the kernel back too. I was just fine with 8.0 with zfs-on-root till the last update when my typical workload (portupgrade every now and then, make world) resulted in kmem map too small panics and more sluggish performance (spending most time in zio-io_cv) during pkgdb -F on a i386 system with 2gb memory. I have 1gb memory sized amd64 vmware fusion vm's that run without incident using the same workload. I had to bump KVA_PAGES to 512 and set vm.kmem_size=1024M to have this i386 system stable again. Prior to that I tried to lower the arc cache size but that resulted in a unresponsive system that I could not diagnose. Regards, Ruben ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
Giulio Ferro wrote: Thanks, I'll try these settings. I'll keep you posted. Nope, it's happened again... Now I've tried to rise vm.kmem_size to 6G... I'm really astounded at how unstable zfs is, it's causing me a lot of problem. Why isn't it stated in the handbook that zfs isn't up to production yet? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
On Wed, May 05, 2010 at 08:32:03AM +0200, Giulio Ferro wrote: Giulio Ferro wrote: Thanks, I'll try these settings. I'll keep you posted. Nope, it's happened again... Now I've tried to rise vm.kmem_size to 6G... Did you set both vm.kmem_size and vfs.zfs.arc_max, setting the latter to something *less* than vm.kmem_size? I'm really astounded at how unstable zfs is, it's causing me a lot of problem. Why isn't it stated in the handbook that zfs isn't up to production yet? I'm not at liberty to comment + answer this question. -- | Jeremy Chadwick j...@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
On 05.05.2010 09:52, Jeremy Chadwick wrote: Nope, it's happened again... Now I've tried to rise vm.kmem_size to 6G... Did you set both vm.kmem_size and vfs.zfs.arc_max, setting the latter to something *less* than vm.kmem_size? Yes. After your suggestion, I set vfs.zfs.arc_max: 3758096384 vm.kmem_size: 4G Now: vfs.zfs.arc_max: 3758096384 vm.kmem_size: 6392119296 I'm really astounded at how unstable zfs is, it's causing me a lot of problem. Why isn't it stated in the handbook that zfs isn't up to production yet? I'm not at liberty to comment + answer this question. Why people responsible for ZFS on freebsd aren't saying anything? What's the status of this issue? Is someone working on this??? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
On 05.05.2010 11:11, Simun Mikecin wrote: - Original Message I'm really astounded at how unstable zfs is, it's causing me a lot of problem. Why isn't it stated in the handbook that zfs isn't up to production yet? Why people responsible for ZFS on freebsd aren't saying anything? What's the status of this issue? Is someone working on this??? How much RAM do you have? Are you using i386 or amd64? Have you tried removing all zfs and kmem sysctl's so the system uses default values? This is the first post of the current thread: it should tell you everything NFS server amd64 Freebsd 8.0 recent (2 days ago) This server has been running for several months without problems. Beginning last week, however, I'm experiencing panics (about 1 per day) with the error in the subject Current settings: vm.kmem_size_scale: 3 vm.kmem_size_max: 329853485875 vm.kmem_size_min: 0 vm.kmem_size: 2764046336 ... hw.physmem: 8568225792 hw.usermem: 6117404672 hw.realmem: 9395240960 ... vfs.zfs.l2arc_noprefetch: 0 vfs.zfs.l2arc_feed_secs_shift: 1 vfs.zfs.l2arc_feed_secs: 1 vfs.zfs.l2arc_headroom: 128 vfs.zfs.l2arc_write_boost: 67108864 vfs.zfs.l2arc_write_max: 67108864 vfs.zfs.arc_meta_limit: 431882240 vfs.zfs.arc_meta_used: 431874720 vfs.zfs.arc_min: 215941120 vfs.zfs.arc_max: 1727528960 I've set nothing in either /boot/loader.conf or sysctl.conf What should I do? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
On Wed, May 05, 2010 at 10:46:31AM +0200, Giulio Ferro wrote: On 05.05.2010 09:52, Jeremy Chadwick wrote: Nope, it's happened again... Now I've tried to rise vm.kmem_size to 6G... Did you set both vm.kmem_size and vfs.zfs.arc_max, setting the latter to something *less* than vm.kmem_size? Yes. After your suggestion, I set vfs.zfs.arc_max: 3758096384 vm.kmem_size: 4G Now: vfs.zfs.arc_max: 3758096384 vm.kmem_size: 6392119296 Could you try to track down the commit that is causing your problems? Could you try 8-STABLE kernel from before r206815? -- Pawel Jakub Dawidek http://www.wheelsystems.com p...@freebsd.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! pgptmoGQAAce1.pgp Description: PGP signature
Re: Freebsd 8.0 kmem map too small
On May 5, 2010, at 9:33 AM, Pawel Jakub Dawidek wrote: On Wed, May 05, 2010 at 10:46:31AM +0200, Giulio Ferro wrote: On 05.05.2010 09:52, Jeremy Chadwick wrote: Nope, it's happened again... Now I've tried to rise vm.kmem_size to 6G... Did you set both vm.kmem_size and vfs.zfs.arc_max, setting the latter to something *less* than vm.kmem_size? Yes. After your suggestion, I set vfs.zfs.arc_max: 3758096384 vm.kmem_size: 4G Now: vfs.zfs.arc_max: 3758096384 vm.kmem_size: 6392119296 Could you try to track down the commit that is causing your problems? Could you try 8-STABLE kernel from before r206815? Are others generally able to run ARC values so close to kmem size? My experience has been that you really need the ARC to be much smaller than the kmem limit (like 25% or less) due to fragmentation of kmem_map. - Ben___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
On 05/05/10 10:12, Ben Kelly wrote: On May 5, 2010, at 9:33 AM, Pawel Jakub Dawidek wrote: On Wed, May 05, 2010 at 10:46:31AM +0200, Giulio Ferro wrote: On 05.05.2010 09:52, Jeremy Chadwick wrote: Nope, it's happened again... Now I've tried to rise vm.kmem_size to 6G... Did you set both vm.kmem_size and vfs.zfs.arc_max, setting the latter to something *less* than vm.kmem_size? Yes. After your suggestion, I set vfs.zfs.arc_max: 3758096384 vm.kmem_size: 4G Now: vfs.zfs.arc_max: 3758096384 vm.kmem_size: 6392119296 Could you try to track down the commit that is causing your problems? Could you try 8-STABLE kernel from before r206815? Are others generally able to run ARC values so close to kmem size? My experience has been that you really need the ARC to be much smaller than the kmem limit (like 25% or less) due to fragmentation of kmem_map. On a system here with 8GB of RAM and a very large zpool consisting of multiple zdevs, little tuning was needed. The system is running 8-STABLE(amd64) as of a month or two ago. The only things I have set in /boot/loader.conf are: vm.kmem_size=12G vfs.zfs.arc_max=4G Setting kmem_size to 12G came from a combination of recommendations I saw in various mailing list posts (you can probably dig them up). Setting it to the physical memory size was the initial recommendation, while others recommended 1.5x physical memory size to help prevent fragmentation/wasted space in kmem. Regardless, this has served us quite well for the ~6 months the system has been in use. It has never crashed, even under intensive multi-threaded benchmarking. -Steve ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
On Tue, May 4, 2010 at 11:32 PM, Giulio Ferro au...@zirakzigil.org wrote: Giulio Ferro wrote: Thanks, I'll try these settings. I'll keep you posted. Nope, it's happened again... Now I've tried to rise vm.kmem_size to 6G... I'm really astounded at how unstable zfs is, it's causing me a lot of problem. Why isn't it stated in the handbook that zfs isn't up to production yet? As with everything related to computers, it all depends on your uses. We've been using FreeBSD+ZFS since it first hit 7.0 with very few problems. The first month of use required a lot of tuning and testing, though, to find the correct kmem, arc, etc settings for our workload. Once we found them, though, things have been rockstable. We rsync 120 remote Linux and FreeBSD servers to our 24-drive, 18 TB backup server, every night. And then rsync that to another similar server. And share out a couple of filesytems via NFS and another couple via Samba. Expect to spend a week or two to tune ZFS, and to rebuild the pool at least once to find the optimal vdev layout. But once that's done, expect it to be solid. -- Freddie Cash fjwc...@gmail.com ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Freebsd 8.0 kmem map too small
NFS server amd64 Freebsd 8.0 recent (2 days ago) This server has been running for several months without problems. Beginning last week, however, I'm experiencing panics (about 1 per day) with the error in the subject Current settings: vm.kmem_size_scale: 3 vm.kmem_size_max: 329853485875 vm.kmem_size_min: 0 vm.kmem_size: 2764046336 ... hw.physmem: 8568225792 hw.usermem: 6117404672 hw.realmem: 9395240960 ... vfs.zfs.l2arc_noprefetch: 0 vfs.zfs.l2arc_feed_secs_shift: 1 vfs.zfs.l2arc_feed_secs: 1 vfs.zfs.l2arc_headroom: 128 vfs.zfs.l2arc_write_boost: 67108864 vfs.zfs.l2arc_write_max: 67108864 vfs.zfs.arc_meta_limit: 431882240 vfs.zfs.arc_meta_used: 431874720 vfs.zfs.arc_min: 215941120 vfs.zfs.arc_max: 1727528960 I've set nothing in either /boot/loader.conf or sysctl.conf What should I do? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
On Mon, May 03, 2010 at 12:41:50PM +0200, Giulio Ferro wrote: NFS server amd64 Freebsd 8.0 recent (2 days ago) This server has been running for several months without problems. Beginning last week, however, I'm experiencing panics (about 1 per day) with the error in the subject Current settings: vm.kmem_size_scale: 3 vm.kmem_size_max: 329853485875 vm.kmem_size_min: 0 vm.kmem_size: 2764046336 ... hw.physmem: 8568225792 hw.usermem: 6117404672 hw.realmem: 9395240960 ... vfs.zfs.l2arc_noprefetch: 0 vfs.zfs.l2arc_feed_secs_shift: 1 vfs.zfs.l2arc_feed_secs: 1 vfs.zfs.l2arc_headroom: 128 vfs.zfs.l2arc_write_boost: 67108864 vfs.zfs.l2arc_write_max: 67108864 vfs.zfs.arc_meta_limit: 431882240 vfs.zfs.arc_meta_used: 431874720 vfs.zfs.arc_min: 215941120 vfs.zfs.arc_max: 1727528960 I've set nothing in either /boot/loader.conf or sysctl.conf What should I do? You need to adjust vm.kmem_size to provide more space for the ARC. Below are ZFS-relevant entries in our /boot/loader.conf on production RELENG_8 systems with 8GB of RAM. The reason we set kmem_size to half our physical system memory is because I didn't want to risk other processes which use a larger maxdsiz/dfldsiz/maxssiz to potentially exhaust all memory. # Increase vm.kmem_size to allow for ZFS ARC to utilise more memory. vm.kmem_size=4096M vfs.zfs.arc_max=3584M # Disable ZFS prefetching # http://southbrain.com/south/2008/04/the-nightmare-comes-slowly-zfs.html # Increases overall speed of ZFS, but when disk flushing/writes occur, # system is less responsive (due to extreme disk I/O). # NOTE: 8.0-RC1 disables this by default on systems = 4GB RAM anyway # NOTE: System has 8GB of RAM, so prefetch would be enabled by default. vfs.zfs.prefetch_disable=1 # Decrease ZFS txg timeout value from 30 (default) to 5 seconds. This # should increase throughput and decrease the bursty stalls that # happen during immense I/O with ZFS. # http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007343.html # http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007355.html vfs.zfs.txg.timeout=5 -- | Jeremy Chadwick j...@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Freebsd 8.0 kmem map too small
On 03.05.2010 13:01, Jeremy Chadwick wrote: On Mon, May 03, 2010 at 12:41:50PM +0200, Giulio Ferro wrote: NFS server amd64 Freebsd 8.0 recent (2 days ago) This server has been running for several months without problems. Beginning last week, however, I'm experiencing panics (about 1 per day) with the error in the subject Current settings: vm.kmem_size_scale: 3 vm.kmem_size_max: 329853485875 vm.kmem_size_min: 0 vm.kmem_size: 2764046336 ... hw.physmem: 8568225792 hw.usermem: 6117404672 hw.realmem: 9395240960 ... vfs.zfs.l2arc_noprefetch: 0 vfs.zfs.l2arc_feed_secs_shift: 1 vfs.zfs.l2arc_feed_secs: 1 vfs.zfs.l2arc_headroom: 128 vfs.zfs.l2arc_write_boost: 67108864 vfs.zfs.l2arc_write_max: 67108864 vfs.zfs.arc_meta_limit: 431882240 vfs.zfs.arc_meta_used: 431874720 vfs.zfs.arc_min: 215941120 vfs.zfs.arc_max: 1727528960 I've set nothing in either /boot/loader.conf or sysctl.conf What should I do? You need to adjust vm.kmem_size to provide more space for the ARC. Below are ZFS-relevant entries in our /boot/loader.conf on production RELENG_8 systems with 8GB of RAM. The reason we set kmem_size to half our physical system memory is because I didn't want to risk other processes which use a larger maxdsiz/dfldsiz/maxssiz to potentially exhaust all memory. # Increase vm.kmem_size to allow for ZFS ARC to utilise more memory. vm.kmem_size=4096M vfs.zfs.arc_max=3584M # Disable ZFS prefetching # http://southbrain.com/south/2008/04/the-nightmare-comes-slowly-zfs.html # Increases overall speed of ZFS, but when disk flushing/writes occur, # system is less responsive (due to extreme disk I/O). # NOTE: 8.0-RC1 disables this by default on systems= 4GB RAM anyway # NOTE: System has 8GB of RAM, so prefetch would be enabled by default. vfs.zfs.prefetch_disable=1 # Decrease ZFS txg timeout value from 30 (default) to 5 seconds. This # should increase throughput and decrease the bursty stalls that # happen during immense I/O with ZFS. # http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007343.html # http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007355.html vfs.zfs.txg.timeout=5 Thanks, I'll try these settings. I'll keep you posted. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
