Re: Swapping caused by very large (regular) file size
Thomas Hurst wrote:
* Kris Kennaway ([EMAIL PROTECTED]) wrote:
I don't understand your test procedure, can you elaborate?
The spikes from last night are from:
(/sbin/dump -$level -LuaC128 -f - $fs | /usr/bin/tee ${target} |
/sbin/sha1 > ${target}.sha1)
Followed by:
nice -n 19 /home/freaky/bin/par2 c -t+ -r5 -m256 ${target}
Graphs from this run clearly slow a modest amount of paging activity
while it occurs; you can see the other graphs here:
http://voi.aagh.net/backupmunin/
More simply, it can be reproduced by doing this:
-# swapoff -a # clear swap
-# swapon -a
-# top |grep -A1 Mem
Mem: 1371M Active, 5727M Inact, 416M Wired, 271M Cache, 214M Buf, 124M
Free
Swap: 10G Total, 10G Free
-# ls -l voi_20080119_#usr.dmp.0
-rw-r--r--1 root wheel106527672320 Jan 19 05:27
voi_20080119_#usr.dmp.0
-# cat voi_20080119_#usr.dmp.0 >/dev/null &
-% sleep 20 && top |grep -A1 Mem
Mem: 1407M Active, 5789M Inact, 431M Wired, 272M Cache, 214M Buf, 8580K
Free
Swap: 10G Total, 400K Used, 10G Free
The system's occasionally deciding to swap pages out rather than recycle
something from the huge pool of inactive/cache memory (which atm will be
mostly cached pages from this file, since this is my second test run;
the first took longer to ramp up).
-% uname -a
FreeBSD voi.nightsdawn.sf 7.0-RC1 FreeBSD 7.0-RC1 #0: Wed Jan 16
14:56:50 GMT 2008
Thanks, I will try to reproduce.
Kris
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Swapping caused by very large (regular) file size
* Kris Kennaway ([EMAIL PROTECTED]) wrote:
> I don't understand your test procedure, can you elaborate?
The spikes from last night are from:
(/sbin/dump -$level -LuaC128 -f - $fs | /usr/bin/tee ${target} |
/sbin/sha1 > ${target}.sha1)
Followed by:
nice -n 19 /home/freaky/bin/par2 c -t+ -r5 -m256 ${target}
Graphs from this run clearly slow a modest amount of paging activity
while it occurs; you can see the other graphs here:
http://voi.aagh.net/backupmunin/
More simply, it can be reproduced by doing this:
-# swapoff -a # clear swap
-# swapon -a
-# top |grep -A1 Mem
Mem: 1371M Active, 5727M Inact, 416M Wired, 271M Cache, 214M Buf, 124M
Free
Swap: 10G Total, 10G Free
-# ls -l voi_20080119_#usr.dmp.0
-rw-r--r--1 root wheel106527672320 Jan 19 05:27
voi_20080119_#usr.dmp.0
-# cat voi_20080119_#usr.dmp.0 >/dev/null &
-% sleep 20 && top |grep -A1 Mem
Mem: 1407M Active, 5789M Inact, 431M Wired, 272M Cache, 214M Buf, 8580K
Free
Swap: 10G Total, 400K Used, 10G Free
The system's occasionally deciding to swap pages out rather than recycle
something from the huge pool of inactive/cache memory (which atm will be
mostly cached pages from this file, since this is my second test run;
the first took longer to ramp up).
-% uname -a
FreeBSD voi.nightsdawn.sf 7.0-RC1 FreeBSD 7.0-RC1 #0: Wed Jan 16
14:56:50 GMT 2008
--
Thomas 'Freaky' Hurst
http://hur.st/
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Swapping caused by very large (regular) file size
Thomas Hurst wrote: * Thomas Hurst ([EMAIL PROTECTED]) wrote: * Kris Kennaway ([EMAIL PROTECTED]) wrote: Excellent. I've been seeing this behavior for a long time, mostly on backup runs (RAID-1 amr SATA -> 1 disk Marvell ata). It's pretty odd seeing a system with 8G of memory, 60% of which is just cache, swap out half a dozen things for no apparant reason. And to think, people on FreeNode ##freebsd just insisted I had a misconfigured system ;) 7 does indeed seem to resolve this. Also, no sign of corruption on my Marvell 88SX6081, at least during serial read tests. I'll test with a level 0 dump tonight; my writes go via tee to the filesystem and sha1(1) so I should pick up any silent corruption. Doh: http://voi.aagh.net/voi.nightsdawn.sf-swap-day.png Making a 100GB dump of /usr, then making a par2 set results in minor paging activity all through it. Running sha1 over the dump has a similar effect. It settles around 2.5MB swap now, though; 6 would quickly settle around 10MB, so it does appear to be improved somewhat; top isn't showing fetchmail and friends as for instance. This is off a single ata(4) disk, which peaks around 65MB/s. sha1 file results match that from the dump|tee file|sha1 > file.sha1 at least :) I don't understand your test procedure, can you elaborate? Kris ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Swapping caused by very large (regular) file size
* Thomas Hurst ([EMAIL PROTECTED]) wrote: > * Kris Kennaway ([EMAIL PROTECTED]) wrote: > > >> Excellent. I've been seeing this behavior for a long time, mostly on > >> backup runs (RAID-1 amr SATA -> 1 disk Marvell ata). It's pretty odd > >> seeing a system with 8G of memory, 60% of which is just cache, swap > >> out half a dozen things for no apparant reason. And to think, people > >> on FreeNode ##freebsd just insisted I had a misconfigured system ;) > > 7 does indeed seem to resolve this. Also, no sign of corruption on my > Marvell 88SX6081, at least during serial read tests. I'll test with > a level 0 dump tonight; my writes go via tee to the filesystem and > sha1(1) so I should pick up any silent corruption. Doh: http://voi.aagh.net/voi.nightsdawn.sf-swap-day.png Making a 100GB dump of /usr, then making a par2 set results in minor paging activity all through it. Running sha1 over the dump has a similar effect. It settles around 2.5MB swap now, though; 6 would quickly settle around 10MB, so it does appear to be improved somewhat; top isn't showing fetchmail and friends as for instance. This is off a single ata(4) disk, which peaks around 65MB/s. sha1 file results match that from the dump|tee file|sha1 > file.sha1 at least :) -- Thomas 'Freaky' Hurst http://hur.st/ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Swapping caused by very large (regular) file size
On Sat, Jan 12, 2008 at 12:05:38AM -0500, John Baldwin wrote: On Friday 11 January 2008 10:31:47 pm Peter Jeremy wrote: > On Fri, Jan 11, 2008 at 06:44:20PM +0100, Kris Kennaway wrote: > >Ian West wrote: > >> dd if=/dev/zero bs=32768 of=junkfile count=10 seems to do it quite > >> reliably on all the boxes I have tested ? > > > >I am unable to reproduce this on 7.0. > > I can't reproduce it on 6.3-PRERELEASE/amd64 with 1GB RAM. > > vmstat -s;dd if=/dev/zero bs=32768 of=junkfile count=10;vmstat -s > shows the following changes: > 2 swap pager pageins > 2 swap pager pages paged in > 4 swap pager pageouts > 5 swap pager pages paged out >24 vnode pager pageins >78 vnode pager pages paged in > 0 vnode pager pageouts > 0 vnode pager pages paged out You may not have a fast enough disk. We have noticed an issue at work but only on faster controllers (e.g. certain mfi(4) drive configurations) when doing I/O to a single file like the dd command mentioned causes the buffer cache to fill up. The problem being that we can't lock the vm object to recycle pages when we hit the limit that is supposed to prevent this because all the pages in the cache are for the file (vm object) we are working on. Stephan (ups@) says this is fixed in 7. The tell-tale sign that we see is pagedaemon starts chewing up lots of CPU as the kernel tries to realign the page queues along with I/O throughput going down the toilet and being very erratic. -- John Baldwin These are the same symptoms as on a friend's system a little while back with 6.x. I forwarded him a message from this thread and he agreed, and he confirmed having an mfi. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Swapping caused by very large (regular) file size
* Kris Kennaway ([EMAIL PROTECTED]) wrote: >> Excellent. I've been seeing this behavior for a long time, mostly on >> backup runs (RAID-1 amr SATA -> 1 disk Marvell ata). It's pretty odd >> seeing a system with 8G of memory, 60% of which is just cache, swap >> out half a dozen things for no apparant reason. And to think, people >> on FreeNode ##freebsd just insisted I had a misconfigured system ;) 7 does indeed seem to resolve this. Also, no sign of corruption on my Marvell 88SX6081, at least during serial read tests. I'll test with a level 0 dump tonight; my writes go via tee to the filesystem and sha1(1) so I should pick up any silent corruption. >> My other IO related issue with 6 is IO to independent amr arrays >> blocking each other; e.g. daily run's find over amrd0 will invariably >> cause reads from amrd1 and amrd2 to freeze for seconds at a time (pr >> 114438). I'll be very interested to see if 7 fixes that. > > Sounds like it might be due to Giant contention, which I think is > indeed fixed in 7. Can you try it? I've updated to test and it seems that no, it's not fixed. Doing a find /usr >/dev/null and a dd if=/dump/bla/bigfile of=/dev/null bs=64k results in normal behavior, followed by a burst of writes and an IO queue depth on the order of 1,000, which freezes reads on all devices on the card while it clears. I'm not sure why I've not noticed the writes before, but they do make for one obvious culprit: atime updates. Presumably by the time the IO's get to amr(4), they're queued per-card, not per-array, so the pile of extra work for one array starves the rest too. Again, a dd from the Marvell ata(4) controller is completely oblivious, and maintains a steady 55MB/s while amrd0's servicing 1,000 write requests; amrd1/2 freeze. noatime would be the obvious solution, but it'd be nice if either the driver could keep the work queues more isolated, or the IO scheduler could avoid letting them starve out everything else. Or the system could emmit a cute fluffy kitten, and a pony. -- Thomas 'Freaky' Hurst http://hur.st/ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Swapping caused by very large (regular) file size
Thomas Hurst wrote: * John Baldwin ([EMAIL PROTECTED]) wrote: We have noticed an issue at work but only on faster controllers (e.g. certain mfi(4) drive configurations) when doing I/O to a single file like the dd command mentioned causes the buffer cache to fill up. The problem being that we can't lock the vm object to recycle pages when we hit the limit that is supposed to prevent this because all the pages in the cache are for the file (vm object) we are working on. Stephan (ups@) says this is fixed in 7. Excellent. I've been seeing this behavior for a long time, mostly on backup runs (RAID-1 amr SATA -> 1 disk Marvell ata). It's pretty odd seeing a system with 8G of memory, 60% of which is just cache, swap out half a dozen things for no apparant reason. And to think, people on FreeNode ##freebsd just insisted I had a misconfigured system ;) My other IO related issue with 6 is IO to independent amr arrays blocking each other; e.g. daily run's find over amrd0 will invariably cause reads from amrd1 and amrd2 to freeze for seconds at a time (pr 114438). I'll be very interested to see if 7 fixes that. Sounds like it might be due to Giant contention, which I think is indeed fixed in 7. Can you try it? Kris ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Swapping caused by very large (regular) file size
* John Baldwin ([EMAIL PROTECTED]) wrote: > We have noticed an issue at work but only on faster controllers (e.g. > certain mfi(4) drive configurations) when doing I/O to a single file > like the dd command mentioned causes the buffer cache to fill up. The > problem being that we can't lock the vm object to recycle pages when > we hit the limit that is supposed to prevent this because all the > pages in the cache are for the file (vm object) we are working on. > Stephan (ups@) says this is fixed in 7. Excellent. I've been seeing this behavior for a long time, mostly on backup runs (RAID-1 amr SATA -> 1 disk Marvell ata). It's pretty odd seeing a system with 8G of memory, 60% of which is just cache, swap out half a dozen things for no apparant reason. And to think, people on FreeNode ##freebsd just insisted I had a misconfigured system ;) My other IO related issue with 6 is IO to independent amr arrays blocking each other; e.g. daily run's find over amrd0 will invariably cause reads from amrd1 and amrd2 to freeze for seconds at a time (pr 114438). I'll be very interested to see if 7 fixes that. -- Thomas 'Freaky' Hurst http://hur.st/ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Swapping caused by very large (regular) file size
On Friday 11 January 2008 10:31:47 pm Peter Jeremy wrote: > On Fri, Jan 11, 2008 at 06:44:20PM +0100, Kris Kennaway wrote: > >Ian West wrote: > >> dd if=/dev/zero bs=32768 of=junkfile count=10 seems to do it quite > >> reliably on all the boxes I have tested ? > > > >I am unable to reproduce this on 7.0. > > I can't reproduce it on 6.3-PRERELEASE/amd64 with 1GB RAM. > > vmstat -s;dd if=/dev/zero bs=32768 of=junkfile count=10;vmstat -s > shows the following changes: > 2 swap pager pageins > 2 swap pager pages paged in > 4 swap pager pageouts > 5 swap pager pages paged out >24 vnode pager pageins >78 vnode pager pages paged in > 0 vnode pager pageouts > 0 vnode pager pages paged out You may not have a fast enough disk. We have noticed an issue at work but only on faster controllers (e.g. certain mfi(4) drive configurations) when doing I/O to a single file like the dd command mentioned causes the buffer cache to fill up. The problem being that we can't lock the vm object to recycle pages when we hit the limit that is supposed to prevent this because all the pages in the cache are for the file (vm object) we are working on. Stephan (ups@) says this is fixed in 7. The tell-tale sign that we see is pagedaemon starts chewing up lots of CPU as the kernel tries to realign the page queues along with I/O throughput going down the toilet and being very erratic. -- John Baldwin ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Swapping caused by very large (regular) file size
On Fri, Jan 11, 2008 at 06:44:20PM +0100, Kris Kennaway wrote: >Ian West wrote: >> dd if=/dev/zero bs=32768 of=junkfile count=10 seems to do it quite >> reliably on all the boxes I have tested ? > >I am unable to reproduce this on 7.0. I can't reproduce it on 6.3-PRERELEASE/amd64 with 1GB RAM. vmstat -s;dd if=/dev/zero bs=32768 of=junkfile count=10;vmstat -s shows the following changes: 2 swap pager pageins 2 swap pager pages paged in 4 swap pager pageouts 5 swap pager pages paged out 24 vnode pager pageins 78 vnode pager pages paged in 0 vnode pager pageouts 0 vnode pager pages paged out -- Peter Jeremy Please excuse any delays as the result of my ISP's inability to implement an MTA that is either RFC2821-compliant or matches their claimed behaviour. pgphFm24TLtfp.pgp Description: PGP signature
Re: Swapping caused by very large (regular) file size
Ian West wrote: Hello, I have noticed while benchmarking a system with a fair bit of ram (3G usable of 4G installed) that when using a very large file (3G upwards) in a simple benchmark it will cause the system to swap, even though the actual process does not show in top to be using a lot of memory, as soon as the swapping starts the throughput degrades dramatically. The 'inactive' ram shown in top increases rapidly and 'free' ram reduces, this seems fair and sensible, but allowing it to then page to possibly the same spindle/array seems like a bad idea ? I have tested this on a 4.11 system with 512M of ram as well as a RELENG-6 system with an areca raid controller, both behave in the same way, once the file gets to a certain size the system starts paging. Is there any way to tune this behaviour ? The test I have been doing is just generating a big file full of nulls, but bonnie++ causes the same behaviour with very large file sizes. dd if=/dev/zero bs=32768 of=junkfile count=10 seems to do it quite reliably on all the boxes I have tested ? Using cp to copy the file doesnt appear to cause the problem. Any thoughts or suggestions would be much appreciated ? I am unable to reproduce this on 7.0. On a system with 4GB of RAM, creating an 8GB file causes almost all of memory to be allocated to 'inactive' (as expected; these are cached pages in case something accesses the file again). However when free memory gets down to about 110M the inactive pages are replaced and free memory does not drop below this. Can you confirm whether 7.0 continues to manifest the problem for you? kris ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Swapping caused by very large (regular) file size
Hello, I have noticed while benchmarking a system with a fair bit of ram (3G usable of 4G installed) that when using a very large file (3G upwards) in a simple benchmark it will cause the system to swap, even though the actual process does not show in top to be using a lot of memory, as soon as the swapping starts the throughput degrades dramatically. The 'inactive' ram shown in top increases rapidly and 'free' ram reduces, this seems fair and sensible, but allowing it to then page to possibly the same spindle/array seems like a bad idea ? I have tested this on a 4.11 system with 512M of ram as well as a RELENG-6 system with an areca raid controller, both behave in the same way, once the file gets to a certain size the system starts paging. Is there any way to tune this behaviour ? The test I have been doing is just generating a big file full of nulls, but bonnie++ causes the same behaviour with very large file sizes. dd if=/dev/zero bs=32768 of=junkfile count=10 seems to do it quite reliably on all the boxes I have tested ? Using cp to copy the file doesnt appear to cause the problem. Any thoughts or suggestions would be much appreciated ? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
