Re: High cpu usage when using ZFS cache device
- Original Message - From: Mickaël Maillot mickael.mail...@gmail.com same problem here after ~ 30 days with a production server and 2 SSD Intel X25M as L2. so we update and reboot the 8-STABLE server every month. Old thread but also seeing this on 8.2-RELEASE so looks like this may still be an issue. In our case this machine was running mysql with 2 x 60GB cache SSD's. I checked for usage when the machine was idle just before reboot to fix and the l2arc thread was still using 100% of a core even with no disk access happening. Was a PR ever raised for this? Regards Steve This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmas...@multiplay.co.uk. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: High cpu usage when using ZFS cache device
- Original Message - From: Artem Belevich a...@freebsd.org No, there was no PR. L2arc CPU hogging after ~24 days was fixed in r218180 in -HEAD and was MFC'ed to 8-stable in r218429 early in February '11. If you're using 8-RELEASE, upgrading to 8-STABLE would be something to consider as there were other ZFS-related issues fixed there that didn't make it into -RELEASE. Thanks for the confirmation there Artem, we currently can't use 8-STABLE due to the serious routing issue, seem like every packet generates a RTM_MISS routing packet to be sent, which causes high cpu load. Thread: Re: serious packet routing issue causing ntpd high load? Regards Steve This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmas...@multiplay.co.uk. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: High cpu usage when using ZFS cache device
On Tue, Oct 11, 2011 at 2:34 AM, Steven Hartland kill...@multiplay.co.uk wrote: - Original Message - From: Mickaël Maillot mickael.mail...@gmail.com same problem here after ~ 30 days with a production server and 2 SSD Intel X25M as L2. so we update and reboot the 8-STABLE server every month. Old thread but also seeing this on 8.2-RELEASE so looks like this may still be an issue. In our case this machine was running mysql with 2 x 60GB cache SSD's. I checked for usage when the machine was idle just before reboot to fix and the l2arc thread was still using 100% of a core even with no disk access happening. Was a PR ever raised for this? No, there was no PR. L2arc CPU hogging after ~24 days was fixed in r218180 in -HEAD and was MFC'ed to 8-stable in r218429 early in February '11. If you're using 8-RELEASE, upgrading to 8-STABLE would be something to consider as there were other ZFS-related issues fixed there that didn't make it into -RELEASE. --Artem ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: High cpu usage when using ZFS cache device
On Tue, Oct 11, 2011 at 10:21 AM, Steven Hartland kill...@multiplay.co.uk wrote: Thanks for the confirmation there Artem, we currently can't use 8-STABLE due to the serious routing issue, seem like every packet generates a RTM_MISS routing packet to be sent, which causes high cpu load. Thread: Re: serious packet routing issue causing ntpd high load? It's a bummer. If you can build your own kernel cherry-picking following revisions may help with long-term stability: r218429 - fixes original overflow causing CPU hogging by l2arc feeding thread. It will keep you up and running for longer until you hit another overflow. If I remember correctly, it will hit you around 100-days of uptime. Following changes were done after ZFSv28 import, so they will not apply directly to 8-RELEASE, but the idea applies to ZFSv15 as well. The changes should be easy to backport. r223412 - avoids more early overflows in time routines. r224647 - avoids time overflow in TXG processing. --Artem ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: High cpu usage when using ZFS cache device
- Original Message - From: Artem Belevich a...@freebsd.org On Tue, Oct 11, 2011 at 10:21 AM, Steven Hartland kill...@multiplay.co.uk wrote: Thanks for the confirmation there Artem, we currently can't use 8-STABLE due to the serious routing issue, seem like every packet generates a RTM_MISS routing packet to be sent, which causes high cpu load. Thread: Re: serious packet routing issue causing ntpd high load? It's a bummer. If you can build your own kernel cherry-picking following revisions may help with long-term stability: r218429 - fixes original overflow causing CPU hogging by l2arc feeding thread. It will keep you up and running for longer until you hit another overflow. If I remember correctly, it will hit you around 100-days of uptime. This is the main issue we have been keeping an eye out for as we've seen it several times, we don't have too many machines with L2ARC so was surprised to see this with just 26 days up time in this case. Following changes were done after ZFSv28 import, so they will not apply directly to 8-RELEASE, but the idea applies to ZFSv15 as well. The changes should be easy to backport. r223412 - avoids more early overflows in time routines. r224647 - avoids time overflow in TXG processing. We already maintain a custom set of patches for our 8.2 installs so shouldn't be an issue to add these so thanks for the info :) With all three should we expect no uptime overflow issues or still are we still going to look at ~100 day reboots required? Regards Steve This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmas...@multiplay.co.uk. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: High cpu usage when using ZFS cache device
On Tue, Oct 11, 2011 at 1:17 PM, Steven Hartland kill...@multiplay.co.uk wrote: It's a bummer. If you can build your own kernel cherry-picking following revisions may help with long-term stability: r218429 - fixes original overflow causing CPU hogging by l2arc feeding thread. It will keep you up and running for longer until you hit another overflow. If I remember correctly, it will hit you around 100-days of uptime. This is the main issue we have been keeping an eye out for as we've seen it several times, we don't have too many machines with L2ARC so was surprised to see this with just 26 days up time in this case. Following changes were done after ZFSv28 import, so they will not apply directly to 8-RELEASE, but the idea applies to ZFSv15 as well. The changes should be easy to backport. r223412 - avoids more early overflows in time routines. r224647 - avoids time overflow in TXG processing. We already maintain a custom set of patches for our 8.2 installs so shouldn't be an issue to add these so thanks for the info :) With all three should we expect no uptime overflow issues or still are we still going to look at ~100 day reboots required? Those should get you through the known (to me) sources of LBOLT and clock_t related overflows. Can't say whether you'll run into some other problems. --Artem ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: High cpu usage when using ZFS cache device
On Wed, Nov 17, 2010 at 8:47 AM, Christer Solskogen christer.solsko...@gmail.com wrote: Will try to reboot server now to se if that has any impact. It seems to have solved it. At least temporary. -- chs, ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: High cpu usage when using ZFS cache device
same problem here after ~ 30 days with a production server and 2 SSD Intel X25M as L2. so we update and reboot the 8-STABLE server every month. 2010/11/17 Christer Solskogen christer.solsko...@gmail.com On Wed, Nov 17, 2010 at 8:47 AM, Christer Solskogen christer.solsko...@gmail.com wrote: Will try to reboot server now to se if that has any impact. It seems to have solved it. At least temporary. -- chs, ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: High cpu usage when using ZFS cache device
On 11/16/10 08:16, Christer Solskogen wrote: On Tue, Nov 16, 2010 at 1:30 AM, Brian Reichertreich...@numachi.com wrote: On Mon, Nov 15, 2010 at 09:50:50PM +0100, Christer Solskogen wrote: My load on my i7 920 is certainly higher when I add a 8GB usb stick as a ZFS cache device. USB 1.0? 2.0? Dunno even if that would make a difference... This is USB 2.0. I didn't know USB had such much to say on the cpu. You can easily test it - use the stick as a simple disk device with UFS and see how much CPU does it take simply to talk to the device. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: High cpu usage when using ZFS cache device
On Tue, Nov 16, 2010 at 12:47 PM, Ivan Voras ivo...@freebsd.org wrote: You can easily test it - use the stick as a simple disk device with UFS and see how much CPU does it take simply to talk to the device. See, that is why I think it is a ZFS issue. Because I did that. I created a UFS filesystem on the same usb stick. Mounted it and did a dd if=/dev/zero of=/mnt/file. The systemload goes +0.6 instead if +10.3. See: CPU: 0.0% user, 0.0% nice, 0.6% system, 0.0% interrupt, 99.3% idle Mem: 832M Active, 960M Inact, 7017M Wired, 2600K Cache, 1237M Buf, 3063M Free Swap: 8192M Total, 8192M Free PID USERNAMETHR PRI NICE SIZERES STATE C TIME WCPU COMMAND 38261 root 1 460 5776K 1112K wdrain 7 0:07 4.98% dd But when using it as cache device for zfs: CPU: 0.0% user, 0.0% nice, 11.9% system, 0.0% interrupt, 88.1% idle Mem: 832M Active, 193M Inact, 5782M Wired, 2592K Cache, 1237M Buf, 5066M Free Swap: 8192M Total, 8192M Free The funny thing is that when I add the device (and some cache is added to it) the load is normal. But the load goes up when nothing is written to it (or beeing read from it) -- chs, ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: High cpu usage when using ZFS cache device
On Tue, Nov 16, 2010 at 01:15:32PM +0100, Christer Solskogen wrote: On Tue, Nov 16, 2010 at 12:47 PM, Ivan Voras ivo...@freebsd.org wrote: You can easily test it - use the stick as a simple disk device with UFS and see how much CPU does it take simply to talk to the device. See, that is why I think it is a ZFS issue. Because I did that. I created a UFS filesystem on the same usb stick. Mounted it and did a dd if=/dev/zero of=/mnt/file. The systemload goes +0.6 instead if +10.3. See: CPU: 0.0% user, 0.0% nice, 0.6% system, 0.0% interrupt, 99.3% idle Mem: 832M Active, 960M Inact, 7017M Wired, 2600K Cache, 1237M Buf, 3063M Free Swap: 8192M Total, 8192M Free PID USERNAMETHR PRI NICE SIZERES STATE C TIME WCPU COMMAND 38261 root 1 460 5776K 1112K wdrain 7 0:07 4.98% dd But when using it as cache device for zfs: CPU: 0.0% user, 0.0% nice, 11.9% system, 0.0% interrupt, 88.1% idle Mem: 832M Active, 193M Inact, 5782M Wired, 2592K Cache, 1237M Buf, 5066M Free Swap: 8192M Total, 8192M Free The funny thing is that when I add the device (and some cache is added to it) the load is normal. But the load goes up when nothing is written to it (or beeing read from it) Since you're running 8.1-RELEASE, can you please test this issue on RELENG_8 (8.1-STABLE) and see if it exists there? You can download a livefs snapshot or equivalent and test via that (preferably one which has ZFS v15 support; you'll need to make a new pool rather than upgrade your existing pool, unless you plan on moving to RELENG_8 permanently). Here's such a snapshot: ftp://ftp4.freebsd.org/pub/FreeBSD/snapshots/201011/ -- | Jeremy Chadwick j...@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: High cpu usage when using ZFS cache device
Quoting Christer Solskogen christer.solsko...@gmail.com (from Tue, 16 Nov 2010 13:15:32 +0100): On Tue, Nov 16, 2010 at 12:47 PM, Ivan Voras ivo...@freebsd.org wrote: You can easily test it - use the stick as a simple disk device with UFS and see how much CPU does it take simply to talk to the device. See, that is why I think it is a ZFS issue. Because I did that. I created a UFS filesystem on the same usb stick. Mounted it and did a dd if=/dev/zero of=/mnt/file. The systemload goes +0.6 instead if +10.3. See: CPU: 0.0% user, 0.0% nice, 0.6% system, 0.0% interrupt, 99.3% idle Mem: 832M Active, 960M Inact, 7017M Wired, 2600K Cache, 1237M Buf, 3063M Free Swap: 8192M Total, 8192M Free PID USERNAMETHR PRI NICE SIZERES STATE C TIME WCPU COMMAND 38261 root 1 460 5776K 1112K wdrain 7 0:07 4.98% dd But when using it as cache device for zfs: CPU: 0.0% user, 0.0% nice, 11.9% system, 0.0% interrupt, 88.1% idle Mem: 832M Active, 193M Inact, 5782M Wired, 2592K Cache, 1237M Buf, 5066M Free Swap: 8192M Total, 8192M Free The funny thing is that when I add the device (and some cache is added to it) the load is normal. But the load goes up when nothing is written to it (or beeing read from it) How do you measure that nothing is read or written to it? Please check with gstat -f '^DEVICE$' if there are really no reads/writes to the device (please replace DEVICE with the name of your USB device, e.g. da0). If you see writes, I would say - this is the reason for the load - your cache is on the way to be filled with useful data If gstat shows zero activity, I suggest to run 'top -S' and look at the process(es) which consume about 10% CPU (do not take care about the idle process). Based upon this we can maybe suggest further things to investigate. Bye, Alexander. -- Is a tattoo real, like a curb or a battleship? Or are we suffering in Safeway? http://www.Leidinger.netAlexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: High cpu usage when using ZFS cache device
On Tue, Nov 16, 2010 at 1:55 PM, Alexander Leidinger alexan...@leidinger.net wrote: How do you measure that nothing is read or written to it? I used zpool iostat -v Please check with gstat -f '^DEVICE$' if there are really no reads/writes to the device (please replace DEVICE with the name of your USB device, e.g. da0). If you see writes, I would say - this is the reason for the load - your cache is on the way to be filled with useful data I see almost no writes (nor reads) If gstat shows zero activity, I suggest to run 'top -S' and look at the process(es) which consume about 10% CPU (do not take care about the idle process). Based upon this we can maybe suggest further things to investigate. Heres the output of that: CPU: 0.0% user, 0.0% nice, 11.8% system, 0.0% interrupt, 88.1% idle Mem: 841M Active, 193M Inact, 5086M Wired, 4876K Cache, 1237M Buf, 5750M Free Swap: 8192M Total, 8192M Free PID USERNAMETHR PRI NICE SIZERES STATE C TIME WCPU COMMAND 11 root 8 171 ki31 0K 128K CPU00??? 713.62% idle 5 root 5 -8- 0K76K zvol:i 5 401.9H 91.16% zfskern Thanks for your time on looking into this :-) -- chs, ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: High cpu usage when using ZFS cache device
According to Christer Solskogen: See, that is why I think it is a ZFS issue. Because I did that. I created a UFS filesystem on the same usb stick. Mounted it and did a dd if=/dev/zero of=/mnt/file. The systemload goes +0.6 instead if +10.3. Do not forget that everything that is read/written from/to USB devices goes through the CPU (no DMA or anything for USB devices). -- Ollivier ROBERT -=- FreeBSD: The Power to Serve! -=- robe...@keltia.net In memoriam to Ondine, our 2nd child: http://ondine.keltia.net/ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: High cpu usage when using ZFS cache device
On 16 November 2010 13:15, Christer Solskogen christer.solsko...@gmail.com wrote: On Tue, Nov 16, 2010 at 12:47 PM, Ivan Voras ivo...@freebsd.org wrote: You can easily test it - use the stick as a simple disk device with UFS and see how much CPU does it take simply to talk to the device. See, that is why I think it is a ZFS issue. Because I did that. I created a UFS filesystem on the same usb stick. Mounted it and did a dd if=/dev/zero of=/mnt/file. The systemload goes +0.6 instead if +10.3. See: CPU: 0.0% user, 0.0% nice, 0.6% system, 0.0% interrupt, 99.3% idle Mem: 832M Active, 960M Inact, 7017M Wired, 2600K Cache, 1237M Buf, 3063M Free Swap: 8192M Total, 8192M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 38261 root 1 46 0 5776K 1112K wdrain 7 0:07 4.98% dd But when using it as cache device for zfs: CPU: 0.0% user, 0.0% nice, 11.9% system, 0.0% interrupt, 88.1% idle Mem: 832M Active, 193M Inact, 5782M Wired, 2592K Cache, 1237M Buf, 5066M Free Swap: 8192M Total, 8192M Free The funny thing is that when I add the device (and some cache is added to it) the load is normal. But the load goes up when nothing is written to it (or beeing read from it) You mean you have system load on an otherwise idle system? Try this: 1) start top with parameters -H -S, see if anything is using the CPU time 2) start gstat, see if anything is using IO, and if it's particularly slow or busying the device too much ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: High cpu usage when using ZFS cache device
On Tue, Nov 16, 2010 at 2:11 PM, Ollivier Robert
robe...@keltia.freenix.fr wrote:
According to Christer Solskogen:
See, that is why I think it is a ZFS issue. Because I did that.
I created a UFS filesystem on the same usb stick. Mounted it and did a
dd if=/dev/zero of=/mnt/file.
The systemload goes +0.6 instead if +10.3.
Do not forget that everything that is read/written from/to USB devices goes
through the CPU (no DMA or anything for USB devices).
I didn't forget that. Thats why I also tested with UFS. And I do not
have the same issue with that.
With top -HS I discovered this:
5 root-8- 0K76K zio-i 4 403.2H 85.06%
{l2arc_feed_threa}
--
chs,
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: High cpu usage when using ZFS cache device
On Tue, Nov 16, 2010 at 04:53:57PM +0100, Christer Solskogen wrote:
On Tue, Nov 16, 2010 at 2:11 PM, Ollivier Robert
robe...@keltia.freenix.fr wrote:
According to Christer Solskogen:
See, that is why I think it is a ZFS issue. Because I did that.
I created a UFS filesystem on the same usb stick. Mounted it and did a
dd if=/dev/zero of=/mnt/file.
The systemload goes +0.6 instead if +10.3.
Do not forget that everything that is read/written from/to USB devices goes
through the CPU (no DMA or anything for USB devices).
I didn't forget that. Thats why I also tested with UFS. And I do not
have the same issue with that.
With top -HS I discovered this:
5 root-8- 0K76K zio-i 4 403.2H 85.06%
{l2arc_feed_threa}
Please provide the output from:
sysctl -a | grep vfs.zfs.arc
sysctl -a | grep vm.kmem
sysctl kstat.zfs.misc.arcstats
Thank you.
--
| Jeremy Chadwick j...@parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: High cpu usage when using ZFS cache device
Quoting Christer Solskogen christer.solsko...@gmail.com (from Tue, 16 Nov 2010 14:00:48 +0100): On Tue, Nov 16, 2010 at 1:55 PM, Alexander Leidinger alexan...@leidinger.net wrote: How do you measure that nothing is read or written to it? I used zpool iostat -v zpool iostat (without -v) does not show cache filling writes to the cache device. I do not know about -v, but I would not be surprised if it does not show this too. Please check with gstat -f '^DEVICE$' if there are really no reads/writes to the device (please replace DEVICE with the name of your USB device, e.g. da0). If you see writes, I would say - this is the reason for the load - your cache is on the way to be filled with useful data I see almost no writes (nor reads) I'm not sure: you verified the output of zpool iostat -v with gstat or not? If not, please do. If gstat shows zero activity, I suggest to run 'top -S' and look at the process(es) which consume about 10% CPU (do not take care about the idle process). Based upon this we can maybe suggest further things to investigate. Heres the output of that: CPU: 0.0% user, 0.0% nice, 11.8% system, 0.0% interrupt, 88.1% idle Mem: 841M Active, 193M Inact, 5086M Wired, 4876K Cache, 1237M Buf, 5750M Free Swap: 8192M Total, 8192M Free PID USERNAMETHR PRI NICE SIZERES STATE C TIME WCPU COMMAND 11 root 8 171 ki31 0K 128K CPU00??? 713.62% idle 5 root 5 -8- 0K76K zvol:i 5 401.9H 91.16% zfskern Thanks for your time on looking into this :-) Based upon you other answer (with -H), I would still think the L2arc (cache) device is being filled in the background (which means there should be something visible with gstat). Bye, Alexander. -- Sorry never means having you're say to love. http://www.Leidinger.netAlexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: High cpu usage when using ZFS cache device
On Tue, Nov 16, 2010 at 5:31 PM, Alexander Leidinger alexan...@leidinger.net wrote: Quoting Christer Solskogen christer.solsko...@gmail.com (from Tue, 16 Nov 2010 14:00:48 +0100): On Tue, Nov 16, 2010 at 1:55 PM, Alexander Leidinger alexan...@leidinger.net wrote: How do you measure that nothing is read or written to it? I used zpool iostat -v zpool iostat (without -v) does not show cache filling writes to the cache device. I do not know about -v, but I would not be surprised if it does not show this too. Ah, but it does. capacity operationsbandwidth pool used avail read write read write -- - - - - - - data2.11T 1.96T 0 0 0 0 raidz12.11T 1.96T 0 0 0 0 ada1- - 0 0 0 0 ada2- - 0 0 0 0 ada3- - 0 0 0 0 cache - - - - - - da0 2.64G 4.89G 0 0 0 0 -- - - - - - - Please check with gstat -f '^DEVICE$' if there are really no reads/writes to the device (please replace DEVICE with the name of your USB device, e.g. da0). If you see writes, I would say - this is the reason for the load - your cache is on the way to be filled with useful data I see almost no writes (nor reads) I'm not sure: you verified the output of zpool iostat -v with gstat or not? If not, please do. Yeah, gstat shows (almost) the same as zpool iostat -v. (gstat have a higher refresh rate than iostat) Based upon you other answer (with -H), I would still think the L2arc (cache) device is being filled in the background (which means there should be something visible with gstat). gstat shows that something is going on when I add the cache device(about one minute in my case). But the systemload is +0.01%. But when it's settled the system load goes up. top -HS says that the command called: l2arc_feed_threa goes up when nothing is happening to the cache. (according to zpool iostat -v and gstat) -- chs, ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: High cpu usage when using ZFS cache device
On Tue, Nov 16, 2010 at 5:01 PM, Jeremy Chadwick free...@jdc.parodius.com wrote: sysctl -a | grep vfs.zfs.arc sysctl -a | grep vm.kmem sysctl kstat.zfs.misc.arcstats $ sysctl -a | grep vfs.zfs.arc vfs.zfs.arc_meta_limit: 1342177280 vfs.zfs.arc_meta_used: 1319657696 vfs.zfs.arc_min: 671088640 vfs.zfs.arc_max: 5368709120 $ sysctl -a | grep vm.kmem vm.kmem_size_scale: 3 vm.kmem_size_max: 329853485875 vm.kmem_size_min: 0 vm.kmem_size: 17179869184 $ sysctl kstat.zfs.misc.arcstats kstat.zfs.misc.arcstats.hits: 1222580509 kstat.zfs.misc.arcstats.misses: 68466812 kstat.zfs.misc.arcstats.demand_data_hits: 484092448 kstat.zfs.misc.arcstats.demand_data_misses: 4205673 kstat.zfs.misc.arcstats.demand_metadata_hits: 491631116 kstat.zfs.misc.arcstats.demand_metadata_misses: 19606091 kstat.zfs.misc.arcstats.prefetch_data_hits: 24766020 kstat.zfs.misc.arcstats.prefetch_data_misses: 37797493 kstat.zfs.misc.arcstats.prefetch_metadata_hits: 222090925 kstat.zfs.misc.arcstats.prefetch_metadata_misses: 6857555 kstat.zfs.misc.arcstats.mru_hits: 69264838 kstat.zfs.misc.arcstats.mru_ghost_hits: 11925502 kstat.zfs.misc.arcstats.mfu_hits: 974464567 kstat.zfs.misc.arcstats.mfu_ghost_hits: 12990137 kstat.zfs.misc.arcstats.allocated: 103515682 kstat.zfs.misc.arcstats.deleted: 48289237 kstat.zfs.misc.arcstats.stolen: 28519978 kstat.zfs.misc.arcstats.recycle_miss: 6634926 kstat.zfs.misc.arcstats.mutex_miss: 80863 kstat.zfs.misc.arcstats.evict_skip: 59151426 kstat.zfs.misc.arcstats.evict_l2_cached: 1196327357952 kstat.zfs.misc.arcstats.evict_l2_eligible: 2965132368896 kstat.zfs.misc.arcstats.evict_l2_ineligible: 1384672357376 kstat.zfs.misc.arcstats.hash_elements: 281528 kstat.zfs.misc.arcstats.hash_elements_max: 482648 kstat.zfs.misc.arcstats.hash_collisions: 33649608 kstat.zfs.misc.arcstats.hash_chains: 76224 kstat.zfs.misc.arcstats.hash_chain_max: 12 kstat.zfs.misc.arcstats.p: 3661654753 kstat.zfs.misc.arcstats.c: 5296620032 kstat.zfs.misc.arcstats.c_min: 671088640 kstat.zfs.misc.arcstats.c_max: 5368709120 kstat.zfs.misc.arcstats.size: 5296227856 kstat.zfs.misc.arcstats.hdr_size: 67212264 kstat.zfs.misc.arcstats.data_size: 4485484032 kstat.zfs.misc.arcstats.other_size: 743532072 kstat.zfs.misc.arcstats.l2_hits: 13784878 kstat.zfs.misc.arcstats.l2_misses: 5542 kstat.zfs.misc.arcstats.l2_feeds: 451293848 kstat.zfs.misc.arcstats.l2_rw_clash: 471 kstat.zfs.misc.arcstats.l2_read_bytes: 232047452160 kstat.zfs.misc.arcstats.l2_write_bytes: 1000487455744 kstat.zfs.misc.arcstats.l2_writes_sent: 1419007 kstat.zfs.misc.arcstats.l2_writes_done: 1419007 kstat.zfs.misc.arcstats.l2_writes_error: 6 kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 417 kstat.zfs.misc.arcstats.l2_evict_lock_retry: 377 kstat.zfs.misc.arcstats.l2_evict_reading: 91 kstat.zfs.misc.arcstats.l2_free_on_write: 896559 kstat.zfs.misc.arcstats.l2_abort_lowmem: 75001 kstat.zfs.misc.arcstats.l2_cksum_bad: 4 kstat.zfs.misc.arcstats.l2_io_error: 5 kstat.zfs.misc.arcstats.l2_size: 2325617664 kstat.zfs.misc.arcstats.l2_hdr_size: 0 kstat.zfs.misc.arcstats.memory_throttle_count: 2493 kstat.zfs.misc.arcstats.l2_write_trylock_fail: 148110262 kstat.zfs.misc.arcstats.l2_write_passed_headroom: 6266060163 kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 0 kstat.zfs.misc.arcstats.l2_write_in_l2: 11644519641883 kstat.zfs.misc.arcstats.l2_write_io_in_progress: 1099778 kstat.zfs.misc.arcstats.l2_write_not_cacheable: 643516792351 kstat.zfs.misc.arcstats.l2_write_full: 86844 kstat.zfs.misc.arcstats.l2_write_buffer_iter: 451293848 kstat.zfs.misc.arcstats.l2_write_pios: 1419007 kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 189074461394960896 kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 28880609174 kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 1367002655 Hope this helps! -- chs, ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: High cpu usage when using ZFS cache device
On Tue, Nov 16, 2010 at 1:54 PM, Jeremy Chadwick free...@jdc.parodius.com wrote: Since you're running 8.1-RELEASE, can you please test this issue on RELENG_8 (8.1-STABLE) and see if it exists there? Sure, I could do that. 8.2-RELEASE isn't that far away, is it? But I think that Alexander should get the necessary info first (since he is the ZFS expert ;-) I also have another machine which I can installed it to. I just have to verify if that machine also have the same problem. It might be the usb stick (but I doubt that), it might be something with cache device on usb when using raidz, it might be anything else. -- chs, ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: High cpu usage when using ZFS cache device
On Tue, Nov 16, 2010 at 5:55 PM, Christer Solskogen christer.solsko...@gmail.com wrote: snip Yesterday I installed 8.1-RELEASE on another machine, made a zpool and added the same usb device as cache. That machine does not have same issue as my other machine. -- chs, ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: High cpu usage when using ZFS cache device
On Wed, Nov 17, 2010 at 7:46 AM, Christer Solskogen christer.solsko...@gmail.com wrote: On Tue, Nov 16, 2010 at 5:55 PM, Christer Solskogen christer.solsko...@gmail.com wrote: snip Yesterday I installed 8.1-RELEASE on another machine, made a zpool and added the same usb device as cache. That machine does not have same issue as my other machine. I also tried adding a SSD as a cache device to the machine that gets the high load. Same symptoms as if I added a USB stick. Will try to reboot server now to se if that has any impact. -- chs, ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: High cpu usage when using ZFS cache device
On Mon, Nov 15, 2010 at 09:50:50PM +0100, Christer Solskogen wrote: My load on my i7 920 is certainly higher when I add a 8GB usb stick as a ZFS cache device. USB 1.0? 2.0? Dunno even if that would make a difference... -- Brian Reichert reich...@numachi.com 55 Crystal Ave. #286 Derry NH 03038-1725 USA BSD admin/developer at large ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: High cpu usage when using ZFS cache device
On Tue, Nov 16, 2010 at 1:30 AM, Brian Reichert reich...@numachi.com wrote: On Mon, Nov 15, 2010 at 09:50:50PM +0100, Christer Solskogen wrote: My load on my i7 920 is certainly higher when I add a 8GB usb stick as a ZFS cache device. USB 1.0? 2.0? Dunno even if that would make a difference... This is USB 2.0. I didn't know USB had such much to say on the cpu. -- chs, ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
