On Wed, Sep 13, 2017 at 08:21:01AM -0400, Austin S. Hemmelgarn wrote: > On 2017-09-12 17:13, Adam Borowski wrote: > > On Tue, Sep 12, 2017 at 04:12:32PM -0400, Austin S. Hemmelgarn wrote: > > > On 2017-09-12 16:00, Adam Borowski wrote: > > > > Noted. Both Marat's and my use cases, though, involve VMs that are off > > > > most > > > > of the time, and at least for me, turned on only to test something. > > > > Touching mtime makes rsync run again, and it's freaking _slow_: worse > > > > than > > > > 40 minutes for a 40GB VM (source:SSD target:deduped HDD). > > > 40 minutes for 40GB is insanely slow (that's just short of 18 MB/s) if > > > you're going direct to a hard drive. I get better performance than that > > > on > > > my somewhat pathetic NUC based storage cluster (I get roughly 20 MB/s > > > there, > > > but it's for archival storage so I don't really care). I'm actually > > > curious > > > what the exact rsync command you are using is (you can obviously redact > > > paths as you see fit), as the only way I can think of that it should be > > > that > > > slow is if you're using both --checksum (but if you're using this, you can > > > tell rsync to skip the mtime check, and that issue goes away) and > > > --inplace, > > > _and_ your HDD is slow to begin with. > > > > rsync -axX --delete --inplace --numeric-ids /mnt/btr1/qemu/ > > mordor:$BASE/qemu > > The target is single, compress=zlib SAMSUNG HD204UI, 34976 hours old but > > with nothing notable on SMART, in a Qnap 253a, kernel 4.9. > compress=zlib is probably your biggest culprit. As odd as this sounds, I'd > suggest switching that to lzo (seriously, the performance difference is > ludicrous), and then setting up a cron job (or systemd timer) to run defrag > over things to switch to zlib. As a general point of comparison, we do > archival backups to a file server running BTRFS where I work, and the > archiving process runs about four to ten times faster if we take this > approach (LZO for initial compression, then recompress using defrag once the > initial transfer is done) than just using zlib directly.
Turns out that lzo is actually the slowest, but only by a bit. I tried a different disk, in the same Qnap; also an old disk but 7200 rpm rather than 5400. Mostly empty, only a handful subvolumes, not much reflinking. I made three separate copies, fallocated -d, upgraded Windows inside the VM, then: [/mnt/btr1/qemu]$ for x in none lzo zlib;do time rsync -axX --delete --inplace --numeric-ids win10.img mordor:/SOME/DIR/$x/win10.img;done real 31m37.459s user 27m21.587s sys 2m16.210s real 33m28.258s user 27m19.745s sys 2m17.642s real 32m57.058s user 27m24.297s sys 2m17.640s Note the "user" values. So rsync does something bad on the source side. Despite fragmentation, reads on the source are not a problem: [/mnt/btr1/qemu]$ time cat <win10.img >/dev/null real 1m28.815s user 0m0.061s sys 0m48.094s [/mnt/btr1/qemu]$ /usr/sbin/filefrag win10.img win10.img: 63682 extents found [/mnt/btr1/qemu]$ btrfs fi def win10.img [/mnt/btr1/qemu]$ /usr/sbin/filefrag win10.img win10.img: 18015 extents found [/mnt/btr1/qemu]$ time cat <win10.img >/dev/null real 1m17.879s user 0m0.076s sys 0m37.757s > `--inplace` is probably not helping (especially if most of the file changed, > on BTRFS, it actually is marginally more efficient to just write out a whole > new file and then replace the old one with a rename if you're rewriting most > of the file), but is probably not as much of an issue as compress=zlib. Yeah, scp + dedupe would run faster. For deduplication, instead of duperemove it'd be better to call file_extent_same on the first 128K, then the second, ... -- without even hashing the blocks beforehand. Not that this particular VM takes enough backup space to make spending too much time worthwhile, but it's a good test case for performance issues like this. Meow! -- ⢀⣴⠾⠻⢶⣦⠀ I've read an article about how lively happy music boosts ⣾⠁⢰⠒⠀⣿⡁ productivity. You can read it, too, you just need the ⢿⡄⠘⠷⠚⠋⠀ right music while doing so. I recommend Skepticism ⠈⠳⣄⠀⠀⠀⠀ (funeral doom metal). -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
