Re: [BackupPC-users] Cpool vs. filesystem level compression
Thanks Alexander -- REALLY helpful, REALLY thoughtful. Comments below Alexander Kobel wrote at about 18:04:54 +0100 on Thursday, January 28, 2021: > For initial backups and changes, it depends on your BackupPC server CPU. > The zlib compression in BackupPC is *way* more resource hungry than lzop > or zstd. You probably want to make sure that the network bandwidth is > the bottleneck rather than compressor throughput: > >gzip -c $somebigfile | pv > /dev/null >zstd -c $somebigfile | pv > /dev/null >lzop -c $somebigfile | pv > /dev/null > > I get the following where I stored the file on a ram disk to minimize the file read time effect... 1] Highly compressible 6GB text file Compress: gzip: 207MiB 0:00:47 [4.39MiB/s] lzop: 355MiB 0:00:05 [70.2MiB/s] zstd: 177MiB 0:00:07 [22.2MiB/s] Uncompress: gzip: 5.90GiB 0:00:21 [ 287MiB/s] lzop: 5.90GiB 0:00:06 [ 946MiB/s] zstd: 5.90GiB 0:00:04 [1.40GiB/s] 2] 1GB highly non-compressible file (created by /dev/urandom) Compress: gzip: 987MiB 0:00:31 [31.6MiB/s] lzop: 987MiB 0:00:00 [1.24GiB/s] zstd: 986MiB 0:00:01 [ 857MiB/s] Note: I used the default compression levels for each. So, focusing on the compressible file: - gzip/zlib is slower than lzop and zstd and less compressible than zstd (but more than lzop) - lzop is fastest but least compressible - zstd is most compressible but slower than lzop, especially on compression My concern with zstd though is that on compression, it is more than 3x slower than lzo -- and is slower than even a standard hard disk, meaning that it may be system performance limiting on writes. Are your numbers similar? Either way, it seems that btrfs block level compression using either lzo or zstd would be about an order of magnitude *faster* than BackupPC compression using zlib. Right?? > +/- multithreading, check for yourself. > > Unchanged files are essentially for free with both cpool and > pool+btrfs-comp for incrementals, but require decryption for full > backups except for rsync (as the hashes are always built over the > uncompressed content). I assume you are referring to '--checksum' > Same for nightlies, where integrity checks over > your pool data is done. I don't believe that BackupPC_nightly does any integrity check of the content, but rather just checks the integrity of the refcounts. As such, I don't believe that it actually reads any files. I did write my own perl script to check pool integrity in case anybody is interested (it is about 100x faster than using a bash script to iterate through the c-pool and pipe files to BackupPC_zcat followed by md5sum) > Decryption is significantly faster, of course, > but still vastly different between the three algorithms. > For fast full backups, you might want to ensure that you can decrypt even > several > times faster than network throughput. > > > 2. Storage efficiency, including: > > - Raw compression efficiency of each file > > Cpool does file-level compression, btrfs does block-level compression. > The difference is measurable, but not huge (~ 1 to 2% compression ratio > in my experience for the same algorithm, i.e. zstd on block vs. file > level). I assume compression is better on the whole file level, right? But deduplication would likely more than make up for this difference, right? > Btrfs also includes a logic to not even attempt further > compression if a block looks like it's not going to compress well. In my > experience, that's hardly ever an issue. > > So, yes, using zlib at the same compression level, btrfs compresses > slightly worse than BackupPC. But for btrfs there's also lzop and zstd. > > > - Ability to take advantage of btrfs extent deduplication for 2 > > distinct files that share some or all of the same (uncompressed) > > content > > Won't work with cpool compression. > For pool+btrfs-comp, it's hard to assess - depends on how your data > changes. Effectively, this only helps with large files that are mostly > identical, such as VM images. Block-level dedup is difficult, only > available as offline dedup in btrfs, You can use 'duperemove' online though it is a memory hog for sure. > and you risk that all your backups > are destroyed if the one copy of the common block in there gets > corrupted. This is true for btrfs snapshots generally of course... > For me a no-go, but YMMV, in particular with a RAID-1. > > File level deduplication is irrelevant, because BackupPC takes care of > that by pooling. > > > 3. Robustness in case of disk crashes, file corruption, file system > > corruption, other types of "bit rot" etc. > > (note my btrfs filesystem is in a btrfs-native Raid-1 > > configuration) > > DISCLAIMER: These are instances for personal data of few people. I care > about the data, but there are no lives or jobs at stake. > > > Solid in my experience. Make sure to perform regular scrubs and
Re: [BackupPC-users] Cpool vs. filesystem level compression
Hi, On 1/27/21 10:58 PM, backu...@kosowsky.org wrote: I know this question has been asked in the more distant past, but I would like to get the latest views, as relevant to backuppc 4.x I have my TopDir on a btrfs filesystem which has file-level compression capabilities (using the mount option -o compress=lzo for example). I use the same, both as a daily driver on my machines and for my BackupPC pool. And I've been an early adopter of zstd instead of lzop, which I cannot praise highly enough. I can do either: 1. Cpool with no btrfs compression 2. Pool with btrfs compression 3. Cpool plus btrfs compression (presumably no advantage) Correct IMHO. The compressed cpool data will not compress any further. So I'll only comment on scenarios 1 and 2. Throughout, I'll assume rsync transfers. Educated guess: the arguments hold for tar and rsyncd. For smb, no idea; decompression speed could be even more relevant. I would like to understand the pros/cons of #1 vs. #2, considering among other things: 1. Backup speed, including: - Initial backup of new files - Subsequent incremental/full backups of the same (unchanged) file - Subsequent incremental/full backups of the same changed file For initial backups and changes, it depends on your BackupPC server CPU. The zlib compression in BackupPC is *way* more resource hungry than lzop or zstd. You probably want to make sure that the network bandwidth is the bottleneck rather than compressor throughput: gzip -c $somebigfile | pv > /dev/null zstd -c $somebigfile | pv > /dev/null lzop -c $somebigfile | pv > /dev/null +/- multithreading, check for yourself. Unchanged files are essentially for free with both cpool and pool+btrfs-comp for incrementals, but require decryption for full backups except for rsync (as the hashes are always built over the uncompressed content). Same for nightlies, where integrity checks over your pool data is done. Decryption is significantly faster, of course, but still vastly different between the three algorithms. For fast full backups, you might want to ensure that you can decrypt even several times faster than network throughput. 2. Storage efficiency, including: - Raw compression efficiency of each file Cpool does file-level compression, btrfs does block-level compression. The difference is measurable, but not huge (~ 1 to 2% compression ratio in my experience for the same algorithm, i.e. zstd on block vs. file level). Btrfs also includes a logic to not even attempt further compression if a block looks like it's not going to compress well. In my experience, that's hardly ever an issue. So, yes, using zlib at the same compression level, btrfs compresses slightly worse than BackupPC. But for btrfs there's also lzop and zstd. - Ability to take advantage of btrfs extent deduplication for 2 distinct files that share some or all of the same (uncompressed) content Won't work with cpool compression. For pool+btrfs-comp, it's hard to assess - depends on how your data changes. Effectively, this only helps with large files that are mostly identical, such as VM images. Block-level dedup is difficult, only available as offline dedup in btrfs, and you risk that all your backups are destroyed if the one copy of the common block in there gets corrupted. For me a no-go, but YMMV, in particular with a RAID-1. File level deduplication is irrelevant, because BackupPC takes care of that by pooling. 3. Robustness in case of disk crashes, file corruption, file system corruption, other types of "bit rot" etc. (note my btrfs filesystem is in a btrfs-native Raid-1 configuration) DISCLAIMER: These are instances for personal data of few people. I care about the data, but there are no lives or jobs at stake. Solid in my experience. Make sure to perform regular scrubs and check that you get informed about problems. On my backup system, I only ever saw problems once, when the HDD was about to die. No RAID to help, so this was fatal for a dozen files, which I had to recover from a second off-site BackupPC server. On my laptops, I saw scrub errors five, six after power losses during heavy duty. That's less than one occasion per year, but still, it happened. On a side note, theoretically you won't need nightly pool checks if you run btrfs scrub at the same rate. With kernel 5.10 being a LTS release, we even have a stable kernel + fallback supporting xxhash/blake2/sha256 checksums, which is great at least from a theoretical perspective. In case there *is* a defect, however, there's not a whole lot of recovery options on btrfs systems. I wasn't able to recover from any of the above scrub errors, I had to delete the affected files. In the past, it seems like the tradeoffs were not always clear so hoping the above outline will help flesh out the details... Looking for both real-world experience as well as theoretical observations :)
