Hi again. On 1/29/21 2:09 AM, backu...@kosowsky.org wrote:
Thanks Alexander -- REALLY helpful, REALLY thoughtful. Comments below Alexander Kobel wrote at about 18:04:54 +0100 on Thursday, January 28, 2021:> For initial backups and changes, it depends on your BackupPC server CPU. > The zlib compression in BackupPC is *way* more resource hungry than lzop > or zstd. You probably want to make sure that the network bandwidth is > the bottleneck rather than compressor throughput: > > gzip -c $somebigfile | pv > /dev/null > zstd -c $somebigfile | pv > /dev/null > lzop -c $somebigfile | pv > /dev/null > > I get the following where I stored the file on a ram disk to minimize the file read time effect... 1] Highly compressible 6GB text file Compress: gzip: 207MiB 0:00:47 [4.39MiB/s] lzop: 355MiB 0:00:05 [70.2MiB/s] zstd: 177MiB 0:00:07 [22.2MiB/s] Uncompress: gzip: 5.90GiB 0:00:21 [ 287MiB/s] lzop: 5.90GiB 0:00:06 [ 946MiB/s] zstd: 5.90GiB 0:00:04 [1.40GiB/s] 2] 1GB highly non-compressible file (created by /dev/urandom) Compress: gzip: 987MiB 0:00:31 [31.6MiB/s] lzop: 987MiB 0:00:00 [1.24GiB/s] zstd: 986MiB 0:00:01 [ 857MiB/s] Note: I used the default compression levels for each. So, focusing on the compressible file: - gzip/zlib is slower than lzop and zstd and less compressible than zstd (but more than lzop) - lzop is fastest but least compressible - zstd is most compressible but slower than lzop, especially on compressionMy concern with zstd though is that on compression, it is more than 3xslower than lzo -- and is slower than even a standard hard disk, meaning that it may be system performance limiting on writes. Are your numbers similar?
Yes, they are, roughly. (On my laptop, didn't check on my server.)However, I should have been a bit more careful here. What's actually important (to me) is whether my server can handle the *input* stream via network in time, in typical cases, so that the server side doesn't delay backup speed.
So the more interesting number is whether the server can consume, say, 1 Gbit/s (or whatever your link speed is) in real-time. In your case, lzop and zstd deal with 1.2 GiB/s and 750 MiB/s *input* from the compressible file, whereas gzip does only 125 MiB/s. That's enough for Gigabit ethernet, theoretically, but just.
However, gzip is a severe bottleneck for the non-compressible data in your case.
And, of course, things get only worse if your CPU is also serving other tasks besides compression.
Either way, it seems that btrfs block level compression using either lzo or zstd would be about an order of magnitude *faster* than BackupPC compression using zlib. Right??
Correct. zlib is excessively slow compared to both, and the compression ratio does not make up for that. I quit gzip and bzip2 entirely and only use zstd, lz4/lzop or xz whenever I can. Of course, xz is abysmal in terms of speed, but it's still in the lead if every byte counts.
> Unchanged files are essentially for free with both cpool and > pool+btrfs-comp for incrementals, but require decryption for full > backups except for rsync (as the hashes are always built over the > uncompressed content). I assume you are referring to '--checksum'
Yes. IIUC, non-rsync-based transfers require 1:1-checking for full backups. (I have no such clients, thus I'm not sure.)
> Same for nightlies, where integrity checks over > your pool data is done. I don't believe that BackupPC_nightly does any integrity check of the content, but rather just checks the integrity of the refcounts. As such, I don't believe that it actually reads any files.
I was refering to $Conf{PoolNightlyDigestCheckPercent}, available since v4.4.0...
I did write my own perl script to check pool integrity in case anybody is interested (it is about 100x faster than using a bash script to iterate through the c-pool and pipe files to BackupPC_zcat followed by md5sum)
... which has been superseded by the above and would again be superfluous with overall btrfs-scrub...
> > 2. Storage efficiency, including: > > - Raw compression efficiency of each file > > Cpool does file-level compression, btrfs does block-level compression. > The difference is measurable, but not huge (~ 1 to 2% compression ratio > in my experience for the same algorithm, i.e. zstd on block vs. file > level). I assume compression is better on the whole file level, right?
Yes. As a rule, the better a file compresses,the more overhead the block-level compression scheme is. To compare the effects, compare the output of `compsize noncompressed-file` with the compression ratio of the full file, or better `du compressed-file`.
For a chunk of kernel sources in a tarball, zstd -3 compresses to 16% file-level, but only to 21% on block-level. But I'd consider that pretty artificial already. (Note that if you compress the files individually, the file system still needs to allocate full blocks, so you'll also waste some bytes.)
But, frankly, that's somewhat academic; you'd need to check for your typical data.
But deduplication would likely more than make up for this difference, right?
It's orthogonal, but I happily trade a slightly larger pool for increased performance.
> [...] Block-level dedup is difficult, only > available as offline dedup in btrfs, You can use 'duperemove' online though it is a memory hog for sure.
Right.
> and you risk that all your backups > are destroyed if the one copy of the common block in there gets > corrupted. This is true for btrfs snapshots generally of course...
Thing is, if you have v1, v2 and v3 of a file, and your drive cannot recover one sector, you'll lose exactly one of v1, v2, v3. And you'll still be left two good copies, perhaps older than you like.
If extents are deduplicated, and the bad sector happens to be exactly on one of the blocks shared between all versions, then the one bad sector kills *all* your copies.
> For me a no-go, but YMMV, in particular with a RAID-1. > [...] > > > 3. Robustness in case of disk crashes, file corruption, file system > > corruption, other types of "bit rot" etc. > > (note my btrfs filesystem is in a btrfs-native Raid-1 > > configuration) [...] Luckily haven't seen any scrub errors yet. But what I like about btrfs Raid-1 is that btrfs can use checksums to decide which extent to choose from when there is a lack of concordance which is not true with standard RAID-1.
Right, that's my understanding, too.
> On a side note, theoretically you won't need nightly pool checks if you > run btrfs scrub at the same rate. I don't think the standard BackupPC_nightly does file integrity checks. See above.
I think you're not up-to-date, see above. ;-)
> With kernel 5.10 being a LTS release, we even have a stable kernel + > fallback supporting xxhash/blake2/sha256 checksums, which is great at > least from a theoretical perspective. > Can you elaborate here? - What advantages do you get?
Practically, at some point: if you want to do block-level dedup, you could rely on comparing block checksums instead of block contents (and double-check before dedup). This could be dramatically faster than current dedup operations. Classic CRC checksums are not foolproof enough, you'll get too many false matches.
Theoretically, rsync --checksum wouldn't need to look at the actual content, it could just read the checksums on both ends. If you trust the regular scrubs that you should do anyway.
Also theoretically, blake and sha are cryptographic hashes, so you're protected against malicious attackers that change the content of your file by flipping bits on-disk, but forget to modify the checksums as well. ;-)
For more details, see https://kdave.github.io/btrfs-hilights-5.5-new-hashes/
- Is it enough to install the kernel or do you need to update user-space programs too? Note: I am running kernel 5.10.7 on Ubuntu 18.04.
You'll have to start fresh - all checksums have to be in the new format, and there are no plans to have in-place conversions of an existing file system.
> In case there *is* a defect, however, there's not a whole lot of > recovery options on btrfs systems. I wasn't able to recover from any of > the above scrub errors, I had to delete the affected files. > > > > In the past, it seems like the tradeoffs were not always clear so > > hoping the above outline will help flesh out the details... VERY HELPFUL > > Looking for both real-world experience as well as theoretical > > observations :) > > Theoretically, pool+btrfs-comp with zstd is hard to beat. You won't find > a better trade-off between resource usage and compression ratio these days. > > Also, I believe it's more elegant and clean to keep compression apart > from BackupPC. Storing and retrieving files efficiently is what > filesystems are there for; BackupPC is busy enough already with rotating > backups, deduplication and transfer. AGREE - plus would be nice to have to use BackupPC_zcat every time I want to look at a pool file.
Insert *not*, but: Exactly.
Is there any way to convert your backups from cpool to pool, short of
manually doing the following:
1. Loop through cpool tree, decompress each file, and move to pool
2. Loop through pc tree, edit each attrib file to change each file
attrib from compress=3 to compress=1
I have done worse, but this would take quite a while to run...
I'm not aware of anything better, sorry.
> Practically, it depends on how much trust you put into btrfs. > > Definitive answers for that one are expected to be available immediately > after the emacs-vs-vi question is settled for good. Oh, that hasn't been resolved yet? :)
;-) Cheers, Alex
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
