Hi, thanks for the answer, I will answer between the lines.
On 15.12.2014 08:45, Robert White wrote: > On 12/14/2014 08:50 PM, Nick Dimov wrote: >> Hello everyone! >> >> First, thanks for amazing work on btrfs filesystem! >> >> Now the problem: >> I use a ssd as my system drive (/dev/sda2) and use daily snapshots on >> it. Then, from time to time, i sync those on HDD (/dev/sdb4) by using >> btrfs send / receive like this: >> >> ionice -c3 btrfs send -p /ssd/previously_synced_snapshot /ssd/snapshot-X >> | pv | btrfs receive /hdd/snapshots >> >> I use pv to measure speed and i get ridiculos speeds like 5-200kiB/s! >> (rarely it goes over 1miB). However if i replace the btrfs receive with >> cat >/dev/null - the speed is 400-500MiB/s (almost full SSD speed) so I >> understand that the problem is the fs on the HDD... Do you have any idea >> of how to trace this problem down? > > > You have _lots_ of problems with that above... > > (1) your ionice is causing the SSD to stall the send every time the > receiver does _anything_. I will try to remove completely ionice - but them my system becomes irresponsive :( > > (1a) The ionice doesn't apply to the pipeline, it only applies to the > command it proceeds. So it's "ionice -c btrfs send..." then pipeline > then "btrfs receive" at the default io scheduling class. You need to > specify it twice, or wrap it all in a script. > > ionice -c 3 btrfs send -p /ssd/parent /ssd/snapshot-X | > ionice -c 3 btrfs receive /hdd/snapshots This is usually what i do but I wanted to show that there is no throtle on the receiver. (i tested it with and without - the result is the same) > > (1b) Your comparison case is flawed because cat >/dev/null results in > no actual IO (e.g. writing to dev-null doesn't transfer any data > anywhere, it just gets rubber-stamped okay at the kernel method level). This was only an intention to show that the sender itself is OK. > > (2) You probably get negative-to-no value from using ionice on the > sending side, particularly since SSDs don't have physical heads to > seek around. yeah in theory it should be like this, but in practice on my system - when i use no ionice my system becomes very unresponsive (ubuntu 14.10). > > (2a) The value of nicing your IO is trivial on the actual SATA buss, > the real value is only realized on a rotating media where the cost of > interrupting other-normal-process is very high because you have to > seek the heads way over there---> when other-normal-process needs them > right here. > > (2b) Any pipeline will naturally throttle a more-left-end process to > wait for a more-right-end process to read the data. The default buffer > is really small, living in the neighborhood of "MAX_PIPE" so like 5k > last time I looked. If you want to throttle this sort of transfer just > throttle the writer. So.. > > btrfs send -p /ssd/parent /ssd/snapshot-X | > ionice -c 3 btrfs receive /hdd/snapshots > > (3) You need to find out if you are treating things nicely on > /hdd/snapshots. If you've hoarded a lot of snapshots on there, or your > snapshot history is really deep, or you are using the drive for other > purposes that are unfriendly to large allocations, or you've laid out > a multi-drive array poorly, then it may need some maintenance. Yes this is what I suspect too, that the system is too fragmented. I have about 15 snapshots now, but new snapshots are created and older ones are deleted, is it possible that this caused the problem? is there a way to tell how badly the file system is fragmented? > > (3a) Test the raw receive throughput if you have the space to share. > To do this save the output of btrfs send to a file with -f some_file. > Then run the receive with -f some_file. Ideally some_file will be on > yet a third media, but it's okay if its on /hdd/snapshot somewhere. > Watch the output of iotop or your favorite graphical monitoring > daemon. If the write throughput is suspiciously low you may be dealing > with a fragmented filesystem. Great idea. Will try this. > > (3b) If /hdd/snapshots is a multi-device filesystem and you are using > "single" for data extents, try switching to RAID0. It's just as > _unsafe_ as "single" but its distributed write layout will speed up > your storage. Its single device. > > (4) If your system is busy enough that you really need the ionice, you > likely just need to really re-think your storage layouts and whatnot. Well, its a laptop :) and i'm not doing anything when the sync happens. I do ionice because without it - the system becomes unresponsive and I can't even browse the internet (it just freezes for 20 seconds or so). But this has to do with the sender somehow... (probably saturates the SATA throughput at 500mb/s?) > > (5) Remember to evaluate the secondary system effects. For example If > /hdd/snapshots is really a USB attached external storage unit, make > sure it's USB3 and so is the port you're plugging it into. Make sure > you aren't paging/swapping in the general sense (or just on the cusp > of doing so) as both of the programs are going to start competing with > your system for buffer cache and whatnot. A "nearly busy" system can > be pushed over the edge into thrashing by adding two large-IO tasks > like these. > > (5b) Give /hdd/snapshots the once-over with smartmontools, especially > if it's old, to make sure its not starting to have read/write retry > delays. Old disks can get "slower" before the get all "failed". > > And remember, you may not be able to do squat about your results. If > you are I/O bound, and you just _can't_ bear to part wiht some subset > of your snapshot hoard for any reason (e.g. I know some government > programs with hellish retention policies), then you might just be > living the practical outcome of your policies and available hardware. > > Thanks again for the answer I will try to do the tests described here and get back. Cheers! -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
