On 12/14/2014 08:50 PM, Nick Dimov wrote:
Hello everyone!
First, thanks for amazing work on btrfs filesystem!
Now the problem:
I use a ssd as my system drive (/dev/sda2) and use daily snapshots on
it. Then, from time to time, i sync those on HDD (/dev/sdb4) by using
btrfs send / receive like this:
ionice -c3 btrfs send -p /ssd/previously_synced_snapshot /ssd/snapshot-X
| pv | btrfs receive /hdd/snapshots
I use pv to measure speed and i get ridiculos speeds like 5-200kiB/s!
(rarely it goes over 1miB). However if i replace the btrfs receive with
cat >/dev/null - the speed is 400-500MiB/s (almost full SSD speed) so I
understand that the problem is the fs on the HDD... Do you have any idea
of how to trace this problem down?
You have _lots_ of problems with that above...
(1) your ionice is causing the SSD to stall the send every time the
receiver does _anything_.
(1a) The ionice doesn't apply to the pipeline, it only applies to the
command it proceeds. So it's "ionice -c btrfs send..." then pipeline
then "btrfs receive" at the default io scheduling class. You need to
specify it twice, or wrap it all in a script.
ionice -c 3 btrfs send -p /ssd/parent /ssd/snapshot-X |
ionice -c 3 btrfs receive /hdd/snapshots
(1b) Your comparison case is flawed because cat >/dev/null results in no
actual IO (e.g. writing to dev-null doesn't transfer any data anywhere,
it just gets rubber-stamped okay at the kernel method level).
(2) You probably get negative-to-no value from using ionice on the
sending side, particularly since SSDs don't have physical heads to seek
around.
(2a) The value of nicing your IO is trivial on the actual SATA buss, the
real value is only realized on a rotating media where the cost of
interrupting other-normal-process is very high because you have to seek
the heads way over there---> when other-normal-process needs them right
here.
(2b) Any pipeline will naturally throttle a more-left-end process to
wait for a more-right-end process to read the data. The default buffer
is really small, living in the neighborhood of "MAX_PIPE" so like 5k
last time I looked. If you want to throttle this sort of transfer just
throttle the writer. So..
btrfs send -p /ssd/parent /ssd/snapshot-X |
ionice -c 3 btrfs receive /hdd/snapshots
(3) You need to find out if you are treating things nicely on
/hdd/snapshots. If you've hoarded a lot of snapshots on there, or your
snapshot history is really deep, or you are using the drive for other
purposes that are unfriendly to large allocations, or you've laid out a
multi-drive array poorly, then it may need some maintenance.
(3a) Test the raw receive throughput if you have the space to share. To
do this save the output of btrfs send to a file with -f some_file. Then
run the receive with -f some_file. Ideally some_file will be on yet a
third media, but it's okay if its on /hdd/snapshot somewhere. Watch the
output of iotop or your favorite graphical monitoring daemon. If the
write throughput is suspiciously low you may be dealing with a
fragmented filesystem.
(3b) If /hdd/snapshots is a multi-device filesystem and you are using
"single" for data extents, try switching to RAID0. It's just as _unsafe_
as "single" but its distributed write layout will speed up your storage.
(4) If your system is busy enough that you really need the ionice, you
likely just need to really re-think your storage layouts and whatnot.
(5) Remember to evaluate the secondary system effects. For example If
/hdd/snapshots is really a USB attached external storage unit, make sure
it's USB3 and so is the port you're plugging it into. Make sure you
aren't paging/swapping in the general sense (or just on the cusp of
doing so) as both of the programs are going to start competing with your
system for buffer cache and whatnot. A "nearly busy" system can be
pushed over the edge into thrashing by adding two large-IO tasks like these.
(5b) Give /hdd/snapshots the once-over with smartmontools, especially if
it's old, to make sure its not starting to have read/write retry delays.
Old disks can get "slower" before the get all "failed".
And remember, you may not be able to do squat about your results. If you
are I/O bound, and you just _can't_ bear to part wiht some subset of
your snapshot hoard for any reason (e.g. I know some government programs
with hellish retention policies), then you might just be living the
practical outcome of your policies and available hardware.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
