Re: [BackupPC-users] Cpool vs. filesystem level compression
Hi again.
On 1/29/21 2:09 AM, backu...@kosowsky.org wrote:
Thanks Alexander -- REALLY helpful, REALLY thoughtful.
Comments below
Alexander Kobel wrote at about 18:04:54 +0100 on Thursday, January 28, 2021:
> For initial backups and changes, it depends on your BackupPC server CPU.
> The zlib compression in BackupPC is *way* more resource hungry than lzop
> or zstd. You probably want to make sure that the network bandwidth is
> the bottleneck rather than compressor throughput:
>
>gzip -c $somebigfile | pv > /dev/null
>zstd -c $somebigfile | pv > /dev/null
>lzop -c $somebigfile | pv > /dev/null
>
>
I get the following where I stored the file on a ram disk to minimize
the file read time effect...
1] Highly compressible 6GB text file
Compress:
gzip: 207MiB 0:00:47 [4.39MiB/s]
lzop: 355MiB 0:00:05 [70.2MiB/s]
zstd: 177MiB 0:00:07 [22.2MiB/s]
Uncompress:
gzip: 5.90GiB 0:00:21 [ 287MiB/s]
lzop: 5.90GiB 0:00:06 [ 946MiB/s]
zstd: 5.90GiB 0:00:04 [1.40GiB/s]
2] 1GB highly non-compressible file (created by /dev/urandom)
Compress:
gzip: 987MiB 0:00:31 [31.6MiB/s]
lzop: 987MiB 0:00:00 [1.24GiB/s]
zstd: 986MiB 0:00:01 [ 857MiB/s]
Note: I used the default compression levels for each.
So, focusing on the compressible file:
- gzip/zlib is slower than lzop and zstd and less compressible than
zstd (but more than lzop)
- lzop is fastest but least compressible
- zstd is most compressible but slower than lzop, especially on
compression
My concern with zstd though is that on compression, it is more than 3x
slower than lzo -- and is slower than even a standard hard disk,
meaning that it may be system performance limiting on writes.
Are your numbers similar?
Yes, they are, roughly. (On my laptop, didn't check on my server.)
However, I should have been a bit more careful here. What's actually
important (to me) is whether my server can handle the *input* stream via
network in time, in typical cases, so that the server side doesn't delay
backup speed.
So the more interesting number is whether the server can consume, say, 1
Gbit/s (or whatever your link speed is) in real-time. In your case, lzop
and zstd deal with 1.2 GiB/s and 750 MiB/s *input* from the compressible
file, whereas gzip does only 125 MiB/s. That's enough for Gigabit
ethernet, theoretically, but just.
However, gzip is a severe bottleneck for the non-compressible data in
your case.
And, of course, things get only worse if your CPU is also serving other
tasks besides compression.
Either way, it seems that btrfs block level compression using either
lzo or zstd would be about an order of magnitude *faster* than
BackupPC compression using zlib. Right??
Correct. zlib is excessively slow compared to both, and the compression
ratio does not make up for that. I quit gzip and bzip2 entirely and only
use zstd, lz4/lzop or xz whenever I can. Of course, xz is abysmal in
terms of speed, but it's still in the lead if every byte counts.
> Unchanged files are essentially for free with both cpool and
> pool+btrfs-comp for incrementals, but require decryption for full
> backups except for rsync (as the hashes are always built over the
> uncompressed content).
I assume you are referring to '--checksum'
Yes. IIUC, non-rsync-based transfers require 1:1-checking for full
backups. (I have no such clients, thus I'm not sure.)
> Same for nightlies, where integrity checks over
> your pool data is done.
I don't believe that BackupPC_nightly does any integrity check of the
content, but rather just checks the integrity of the refcounts. As
such, I don't believe that it actually reads any files.
I was refering to $Conf{PoolNightlyDigestCheckPercent}, available since
v4.4.0...
I did write my own perl script to check pool integrity in case anybody
is interested (it is about 100x faster than using a bash script to
iterate through the c-pool and pipe files to BackupPC_zcat followed by
md5sum)
... which has been superseded by the above and would again be
superfluous with overall btrfs-scrub...
> > 2. Storage efficiency, including:
> > - Raw compression efficiency of each file
>
> Cpool does file-level compression, btrfs does block-level compression.
> The difference is measurable, but not huge (~ 1 to 2% compression ratio
> in my experience for the same algorithm, i.e. zstd on block vs. file
> level).
I assume compression is better on the whole file level, right?
Yes. As a rule, the better a file compresses,
the more overhead the block-level compression scheme is. To compare the
effects, compare the output of `compsize noncompressed-file` with the
compression ratio of the full file, or better `du compressed-file`.
For a chunk of kernel sources in a tarball, zstd -3 compresses to 16%
file-level, but only to 21% on block-level.
But I'd consider that pretty artificial already. (Note that if you
compress the files individually, the file system still
Re: [BackupPC-users] Cpool vs. filesystem level compression
Hi, On 1/27/21 10:58 PM, backu...@kosowsky.org wrote: I know this question has been asked in the more distant past, but I would like to get the latest views, as relevant to backuppc 4.x I have my TopDir on a btrfs filesystem which has file-level compression capabilities (using the mount option -o compress=lzo for example). I use the same, both as a daily driver on my machines and for my BackupPC pool. And I've been an early adopter of zstd instead of lzop, which I cannot praise highly enough. I can do either: 1. Cpool with no btrfs compression 2. Pool with btrfs compression 3. Cpool plus btrfs compression (presumably no advantage) Correct IMHO. The compressed cpool data will not compress any further. So I'll only comment on scenarios 1 and 2. Throughout, I'll assume rsync transfers. Educated guess: the arguments hold for tar and rsyncd. For smb, no idea; decompression speed could be even more relevant. I would like to understand the pros/cons of #1 vs. #2, considering among other things: 1. Backup speed, including: - Initial backup of new files - Subsequent incremental/full backups of the same (unchanged) file - Subsequent incremental/full backups of the same changed file For initial backups and changes, it depends on your BackupPC server CPU. The zlib compression in BackupPC is *way* more resource hungry than lzop or zstd. You probably want to make sure that the network bandwidth is the bottleneck rather than compressor throughput: gzip -c $somebigfile | pv > /dev/null zstd -c $somebigfile | pv > /dev/null lzop -c $somebigfile | pv > /dev/null +/- multithreading, check for yourself. Unchanged files are essentially for free with both cpool and pool+btrfs-comp for incrementals, but require decryption for full backups except for rsync (as the hashes are always built over the uncompressed content). Same for nightlies, where integrity checks over your pool data is done. Decryption is significantly faster, of course, but still vastly different between the three algorithms. For fast full backups, you might want to ensure that you can decrypt even several times faster than network throughput. 2. Storage efficiency, including: - Raw compression efficiency of each file Cpool does file-level compression, btrfs does block-level compression. The difference is measurable, but not huge (~ 1 to 2% compression ratio in my experience for the same algorithm, i.e. zstd on block vs. file level). Btrfs also includes a logic to not even attempt further compression if a block looks like it's not going to compress well. In my experience, that's hardly ever an issue. So, yes, using zlib at the same compression level, btrfs compresses slightly worse than BackupPC. But for btrfs there's also lzop and zstd. - Ability to take advantage of btrfs extent deduplication for 2 distinct files that share some or all of the same (uncompressed) content Won't work with cpool compression. For pool+btrfs-comp, it's hard to assess - depends on how your data changes. Effectively, this only helps with large files that are mostly identical, such as VM images. Block-level dedup is difficult, only available as offline dedup in btrfs, and you risk that all your backups are destroyed if the one copy of the common block in there gets corrupted. For me a no-go, but YMMV, in particular with a RAID-1. File level deduplication is irrelevant, because BackupPC takes care of that by pooling. 3. Robustness in case of disk crashes, file corruption, file system corruption, other types of "bit rot" etc. (note my btrfs filesystem is in a btrfs-native Raid-1 configuration) DISCLAIMER: These are instances for personal data of few people. I care about the data, but there are no lives or jobs at stake. Solid in my experience. Make sure to perform regular scrubs and check that you get informed about problems. On my backup system, I only ever saw problems once, when the HDD was about to die. No RAID to help, so this was fatal for a dozen files, which I had to recover from a second off-site BackupPC server. On my laptops, I saw scrub errors five, six after power losses during heavy duty. That's less than one occasion per year, but still, it happened. On a side note, theoretically you won't need nightly pool checks if you run btrfs scrub at the same rate. With kernel 5.10 being a LTS release, we even have a stable kernel + fallback supporting xxhash/blake2/sha256 checksums, which is great at least from a theoretical perspective. In case there *is* a defect, however, there's not a whole lot of recovery options on btrfs systems. I wasn't able to recover from any of the above scrub errors, I had to delete the affected files. In the past, it seems like the tradeoffs were not always clear so hoping the above outline will help flesh out the details... Looking for both real-world experience as well as theoretical observations :)
