Re: Scrub priority, am I using it wrong?
Yeah, RAID5. I'm now doing pause and resume on it to let it take multiple nights, the idle should let other processes complete in reasonable time. On Wed, Apr 6, 2016 at 3:34 AM, Henk Slagerwrote: > On Tue, Apr 5, 2016 at 4:37 AM, Duncan <1i5t5.dun...@cox.net> wrote: >> Gareth Pye posted on Tue, 05 Apr 2016 09:36:48 +1000 as excerpted: >> >>> I've got a btrfs file system set up on 6 drbd disks running on 2Tb >>> spinning disks. The server is moderately loaded with various regular >>> tasks that use a fair bit of disk IO, but I've scheduled my weekly btrfs >>> scrub for the best quiet time in the week. >>> >>> The command that is run is: >>> /usr/local/bin/btrfs scrub start -Bd -c idle /data >>> >>> Which is my best attempt to try and get it to have a low impact on user >>> operations >>> >>> But iotop shows me: >>> >>> 1765 be/4 root 14.84 M/s0.00 B/s 0.00 % 96.65 % btrfs scrub >>> start -Bd -c idle /data >>> 1767 be/4 root 14.70 M/s0.00 B/s 0.00 % 95.35 % btrfs >>> scrub start -Bd -c idle /data >>> 1768 be/4 root 13.47 M/s0.00 B/s 0.00 % 92.59 % btrfs >>> scrub start -Bd -c idle /data >>> 1764 be/4 root 12.61 M/s0.00 B/s 0.00 % 88.77 % btrfs >>> scrub start -Bd -c idle /data >>> 1766 be/4 root 11.24 M/s0.00 B/s 0.00 % 85.18 % btrfs >>> scrub start -Bd -c idle /data >>> 1763 be/4 root7.79 M/s0.00 B/s 0.00 % 63.30 % btrfs >>> scrub start -Bd -c idle /data >>> 28858 be/4 root0.00 B/s 810.50 B/s 0.00 % 61.32 % [kworker/ >> u16:25] >>> >>> >>> Which doesn't look like an idle priority to me. And the system sure >>> feels like a system with a lot of heavy io going on. Is there something >>> I'm doing wrong? > > When I see the throughput numbers, it lets me think that the > filesystem is raid5 or raid6. On single or raid1 or raid10 one easily > gets around 100M/s without the notice/feeling of heavy IO ongoing, > mostly independent of scrub options. > >> Two points: >> >> 1) It appears btrfs scrub start's -c option only takes numeric class, so >> try -c3 instead of -c idle. > > Thanks to Duncan for pointing this out. I don't remember exactly, but > I think I also had issues with this in the past, but did not realize > or have a further look at it. > >> Works for me with the numeric class (same results as you with spelled out >> class), tho I'm on ssd with multiple independent btrfs on partitions, the >> biggest of which is 24 GiB, 18.something GiB used, which scrubs in all of >> 20 seconds, so I don't need and hadn't tried the -c option at all until >> now. >> >> 2) What a difference an ssd makes! >> >> $$ sudo btrfs scrub start -c3 /p >> scrub started on /p, [...] >> >> $$ sudo iotop -obn1 >> Total DISK READ : 626.53 M/s | Total DISK WRITE : 0.00 B/s >> Actual DISK READ: 596.93 M/s | Actual DISK WRITE: 0.00 B/s >> TID PRIO USER DISK READ DISK WRITE SWAPIN IOCOMMAND >> 872 idle root 268.40 M/s0.00 B/s 0.00 % 0.00 % btrfs scrub >> start -c3 /p >> 873 idle root 358.13 M/s0.00 B/s 0.00 % 0.00 % btrfs scrub >> start -c3 /p >> >> CPU bound, 0% IOWait even at idle IO priority, in addition to the >> hundreds of M/s values per thread/device, here. You OTOH are showing >> under 20 M/s per thread/device on spinning rust, with an IOWait near 90%, >> thus making it IO bound. > > This low M/s and high IOWait is the kind of behavior I noticed with 3x > 2TB raid5 when scrubbing or balancing (no bcache activated, kernel > 4.3.3). > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Gareth Pye - blog.cerberos.id.au Level 2 MTG Judge, Melbourne, Australia -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Scrub priority, am I using it wrong?
On Tue, Apr 5, 2016 at 4:37 AM, Duncan <1i5t5.dun...@cox.net> wrote: > Gareth Pye posted on Tue, 05 Apr 2016 09:36:48 +1000 as excerpted: > >> I've got a btrfs file system set up on 6 drbd disks running on 2Tb >> spinning disks. The server is moderately loaded with various regular >> tasks that use a fair bit of disk IO, but I've scheduled my weekly btrfs >> scrub for the best quiet time in the week. >> >> The command that is run is: >> /usr/local/bin/btrfs scrub start -Bd -c idle /data >> >> Which is my best attempt to try and get it to have a low impact on user >> operations >> >> But iotop shows me: >> >> 1765 be/4 root 14.84 M/s0.00 B/s 0.00 % 96.65 % btrfs scrub >> start -Bd -c idle /data >> 1767 be/4 root 14.70 M/s0.00 B/s 0.00 % 95.35 % btrfs >> scrub start -Bd -c idle /data >> 1768 be/4 root 13.47 M/s0.00 B/s 0.00 % 92.59 % btrfs >> scrub start -Bd -c idle /data >> 1764 be/4 root 12.61 M/s0.00 B/s 0.00 % 88.77 % btrfs >> scrub start -Bd -c idle /data >> 1766 be/4 root 11.24 M/s0.00 B/s 0.00 % 85.18 % btrfs >> scrub start -Bd -c idle /data >> 1763 be/4 root7.79 M/s0.00 B/s 0.00 % 63.30 % btrfs >> scrub start -Bd -c idle /data >> 28858 be/4 root0.00 B/s 810.50 B/s 0.00 % 61.32 % [kworker/ > u16:25] >> >> >> Which doesn't look like an idle priority to me. And the system sure >> feels like a system with a lot of heavy io going on. Is there something >> I'm doing wrong? When I see the throughput numbers, it lets me think that the filesystem is raid5 or raid6. On single or raid1 or raid10 one easily gets around 100M/s without the notice/feeling of heavy IO ongoing, mostly independent of scrub options. > Two points: > > 1) It appears btrfs scrub start's -c option only takes numeric class, so > try -c3 instead of -c idle. Thanks to Duncan for pointing this out. I don't remember exactly, but I think I also had issues with this in the past, but did not realize or have a further look at it. > Works for me with the numeric class (same results as you with spelled out > class), tho I'm on ssd with multiple independent btrfs on partitions, the > biggest of which is 24 GiB, 18.something GiB used, which scrubs in all of > 20 seconds, so I don't need and hadn't tried the -c option at all until > now. > > 2) What a difference an ssd makes! > > $$ sudo btrfs scrub start -c3 /p > scrub started on /p, [...] > > $$ sudo iotop -obn1 > Total DISK READ : 626.53 M/s | Total DISK WRITE : 0.00 B/s > Actual DISK READ: 596.93 M/s | Actual DISK WRITE: 0.00 B/s > TID PRIO USER DISK READ DISK WRITE SWAPIN IOCOMMAND > 872 idle root 268.40 M/s0.00 B/s 0.00 % 0.00 % btrfs scrub > start -c3 /p > 873 idle root 358.13 M/s0.00 B/s 0.00 % 0.00 % btrfs scrub > start -c3 /p > > CPU bound, 0% IOWait even at idle IO priority, in addition to the > hundreds of M/s values per thread/device, here. You OTOH are showing > under 20 M/s per thread/device on spinning rust, with an IOWait near 90%, > thus making it IO bound. This low M/s and high IOWait is the kind of behavior I noticed with 3x 2TB raid5 when scrubbing or balancing (no bcache activated, kernel 4.3.3). -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Scrub priority, am I using it wrong?
On 2016-04-05 00:19, Duncan wrote: Gareth Pye posted on Tue, 05 Apr 2016 13:44:05 +1000 as excerpted: On Tue, Apr 5, 2016 at 12:37 PM, Duncan <1i5t5.dun...@cox.net> wrote: 1) It appears btrfs scrub start's -c option only takes numeric class, so try -c3 instead of -c idle. Does it count as a bug if it silently accepts the way I was doing it? I've switched to -c3 and at least now the idle class listed in iotop is idle, so I hope that means it will be more friendly to other processes. I'd say yes, particularly given that the value must be the numeric class isn't documented in the manpage at all. Whether the bug is then one of documentation (say it must be numeric) or of implementation (take the class name as well) is then up for debate. I'd call fixing either one a fix. If it must be numeric, document that (and optionally change the implementation to error in some way if a numeric parameter isn't supplied for -c), else change the implementation so the name can be taken as well (and optionally change the documentation to explicitly mention that either one can be used). Doesn't matter to me which. In general, I'd say: 1. The documentation needs to be updated. Even if it's intended to accept class names eventually, it should match the implementation. 2. The implementation needs error checking added. At the very least, it should say something to the effect of "Hey, this doesn't work the way you appear to think it does.", and possibly bail. That said, I don't think there's a huge amount of value in adding the ability to accept symbolic names. Nothing that I know of accepts them on the command line, because most things pass the integer directly to the syscall, thus allowing for people to use new classes (if any are ever introduced) with old tools. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Scrub priority, am I using it wrong?
Gareth Pye posted on Tue, 05 Apr 2016 13:45:11 +1000 as excerpted: > On Tue, Apr 5, 2016 at 12:37 PM, Duncan <1i5t5.dun...@cox.net> wrote: >> CPU bound, 0% IOWait even at idle IO priority, in addition to the >> hundreds of M/s values per thread/device, here. You OTOH are showing >> under 20 M/s per thread/device on spinning rust, with an IOWait near >> 90%, >> thus making it IO bound. > > > And yes I'd love to switch to SSD, but 12 2TB drives is a bit pricey > still No kidding. That's why my media partition remains spinning rust. (Tho FWIW, not btrfs, I use btrfs only on my ssds, and still use the old and stable reiserfs on my spinning rust.) But my media partition is small enough, and ssd prices now low enough up to the 1 TB level, that when I upgrade I'll probably switch to ssd for the media partition as well, and leave spinning rust only as second or third level backups. But that's because it all, including first level backups, fits in under a TB (and if pressed I could do it under a half TB). Multi-TB, as you have, definitely still spinning rust, for me too. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Scrub priority, am I using it wrong?
Gareth Pye posted on Tue, 05 Apr 2016 13:44:05 +1000 as excerpted: > On Tue, Apr 5, 2016 at 12:37 PM, Duncan <1i5t5.dun...@cox.net> wrote: >> 1) It appears btrfs scrub start's -c option only takes numeric class, >> so try -c3 instead of -c idle. > > > Does it count as a bug if it silently accepts the way I was doing it? > > I've switched to -c3 and at least now the idle class listed in iotop is > idle, so I hope that means it will be more friendly to other processes. I'd say yes, particularly given that the value must be the numeric class isn't documented in the manpage at all. Whether the bug is then one of documentation (say it must be numeric) or of implementation (take the class name as well) is then up for debate. I'd call fixing either one a fix. If it must be numeric, document that (and optionally change the implementation to error in some way if a numeric parameter isn't supplied for -c), else change the implementation so the name can be taken as well (and optionally change the documentation to explicitly mention that either one can be used). Doesn't matter to me which. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Scrub priority, am I using it wrong?
On Tue, Apr 5, 2016 at 12:37 PM, Duncan <1i5t5.dun...@cox.net> wrote: > CPU bound, 0% IOWait even at idle IO priority, in addition to the > hundreds of M/s values per thread/device, here. You OTOH are showing > under 20 M/s per thread/device on spinning rust, with an IOWait near 90%, > thus making it IO bound. And yes I'd love to switch to SSD, but 12 2TB drives is a bit pricey still -- Gareth Pye - blog.cerberos.id.au Level 2 MTG Judge, Melbourne, Australia -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Scrub priority, am I using it wrong?
On Tue, Apr 5, 2016 at 12:37 PM, Duncan <1i5t5.dun...@cox.net> wrote: > 1) It appears btrfs scrub start's -c option only takes numeric class, so > try -c3 instead of -c idle. Does it count as a bug if it silently accepts the way I was doing it? I've switched to -c3 and at least now the idle class listed in iotop is idle, so I hope that means it will be more friendly to other processes. -- Gareth Pye - blog.cerberos.id.au Level 2 MTG Judge, Melbourne, Australia -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Scrub priority, am I using it wrong?
Gareth Pye posted on Tue, 05 Apr 2016 09:36:48 +1000 as excerpted: > I've got a btrfs file system set up on 6 drbd disks running on 2Tb > spinning disks. The server is moderately loaded with various regular > tasks that use a fair bit of disk IO, but I've scheduled my weekly btrfs > scrub for the best quiet time in the week. > > The command that is run is: > /usr/local/bin/btrfs scrub start -Bd -c idle /data > > Which is my best attempt to try and get it to have a low impact on user > operations > > But iotop shows me: > > 1765 be/4 root 14.84 M/s0.00 B/s 0.00 % 96.65 % btrfs scrub > start -Bd -c idle /data > 1767 be/4 root 14.70 M/s0.00 B/s 0.00 % 95.35 % btrfs > scrub start -Bd -c idle /data > 1768 be/4 root 13.47 M/s0.00 B/s 0.00 % 92.59 % btrfs > scrub start -Bd -c idle /data > 1764 be/4 root 12.61 M/s0.00 B/s 0.00 % 88.77 % btrfs > scrub start -Bd -c idle /data > 1766 be/4 root 11.24 M/s0.00 B/s 0.00 % 85.18 % btrfs > scrub start -Bd -c idle /data > 1763 be/4 root7.79 M/s0.00 B/s 0.00 % 63.30 % btrfs > scrub start -Bd -c idle /data > 28858 be/4 root0.00 B/s 810.50 B/s 0.00 % 61.32 % [kworker/ u16:25] > > > Which doesn't look like an idle priority to me. And the system sure > feels like a system with a lot of heavy io going on. Is there something > I'm doing wrong? Two points: 1) It appears btrfs scrub start's -c option only takes numeric class, so try -c3 instead of -c idle. Works for me with the numeric class (same results as you with spelled out class), tho I'm on ssd with multiple independent btrfs on partitions, the biggest of which is 24 GiB, 18.something GiB used, which scrubs in all of 20 seconds, so I don't need and hadn't tried the -c option at all until now. 2) What a difference an ssd makes! $$ sudo btrfs scrub start -c3 /p scrub started on /p, [...] $$ sudo iotop -obn1 Total DISK READ : 626.53 M/s | Total DISK WRITE : 0.00 B/s Actual DISK READ: 596.93 M/s | Actual DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IOCOMMAND 872 idle root 268.40 M/s0.00 B/s 0.00 % 0.00 % btrfs scrub start -c3 /p 873 idle root 358.13 M/s0.00 B/s 0.00 % 0.00 % btrfs scrub start -c3 /p CPU bound, 0% IOWait even at idle IO priority, in addition to the hundreds of M/s values per thread/device, here. You OTOH are showing under 20 M/s per thread/device on spinning rust, with an IOWait near 90%, thus making it IO bound. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Scrub priority, am I using it wrong?
I've got a btrfs file system set up on 6 drbd disks running on 2Tb spinning disks. The server is moderately loaded with various regular tasks that use a fair bit of disk IO, but I've scheduled my weekly btrfs scrub for the best quiet time in the week. The command that is run is: /usr/local/bin/btrfs scrub start -Bd -c idle /data Which is my best attempt to try and get it to have a low impact on user operations But iotop shows me: 1765 be/4 root 14.84 M/s0.00 B/s 0.00 % 96.65 % btrfs scrub start -Bd -c idle /data 1767 be/4 root 14.70 M/s0.00 B/s 0.00 % 95.35 % btrfs scrub start -Bd -c idle /data 1768 be/4 root 13.47 M/s0.00 B/s 0.00 % 92.59 % btrfs scrub start -Bd -c idle /data 1764 be/4 root 12.61 M/s0.00 B/s 0.00 % 88.77 % btrfs scrub start -Bd -c idle /data 1766 be/4 root 11.24 M/s0.00 B/s 0.00 % 85.18 % btrfs scrub start -Bd -c idle /data 1763 be/4 root7.79 M/s0.00 B/s 0.00 % 63.30 % btrfs scrub start -Bd -c idle /data 28858 be/4 root0.00 B/s 810.50 B/s 0.00 % 61.32 % [kworker/u16:25] Which doesn't look like an idle priority to me. And the system sure feels like a system with a lot of heavy io going on. Is there something I'm doing wrong? System details: # uname -a Linux emile 4.4.3-040403-generic #201602251634 SMP Thu Feb 25 21:36:25 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux # /usr/local/bin/btrfs --version btrfs-progs v4.4.1 I'm waiting on the ppa version of 4.5.1 before upgrading, that is my usual kernel update strategy. # cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=14.04 DISTRIB_CODENAME=trusty DISTRIB_DESCRIPTION="Ubuntu 14.04.4 LTS" Any other details that people would like to see that are relevant to this question? -- Gareth Pye - blog.cerberos.id.au Level 2 MTG Judge, Melbourne, Australia -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html