Hi, On 27.11.18 13:40, Guillermo Rozas wrote:
Pigz doesn't correctly support BackupPC compressed files, althoughit will in some cases. The reported error is likely the problem. Please use BackupPC_zcat instead and report back.Of course, you're right :) (although pigz failed only in 2 files out of several thousands).
oh well, I was wondering about that. I've yet to see such a file (and probably never will, because I disabled pool compression for good and now use btrfs' lzop filesystem-based compression), but...
BackupPC_zcat decompresses both files correctly and their checksums are correct now. However, at least with one of the files there is something fishy going on because the compressed version is 60KB, the decompressed is 7GB!
... from what I understand, the main difference between "standard" gzip/pigz and BackupPC's variant is that the latter adds additional "sync" operations when a (e.g. sparse) file compresses way better than usual. In that case, BackupPC's zipping mechanism ensures that decompression only requires a fixed amount of memory, at the expense that extremely compressible data does not compress to 0.000001%, but only to 0.001% or something. (I'm lacking the details, sorry.)
I'd bet that those two files are extremely sparse.There are good reasons for such a file to be generated: e.g., from a ddrescue run that skipped lots of bad areas on a drive, or a VM disk image with a recently formatted partition, or similar. On many modern file systems supporting sparse files, the overhead for the holes in the file is negligible, so it's easier from a user perspective to allocate the "full" file and rely on the filesystem's abilities to optimize storage and access. However, some of BackupPC's transfer methods (in particular, rsync) cannot treat sparse files natively, but since they compress so well, that's hardly an issue for transfer nor storage on the server.
The reason why I recommended pigz (unfortunately without an appropriate disclaimer) is that it
- never failed on me, for the files I had around at that time, and - it was *magnitudes* faster than BackupPC_zcat. But I had a severely CPU-limited machine; YMMV with a more powerful CPU.Depending on your use case (and performance experience), it might still be clever to run pigz first and only run BackupPC_zcat if there is a mismatch. If a pigz-decompressed file matches the expected hash, I'd bet at approximately 1 : 2^64 that no corruption happened.
Which brings me to:
I added a guide to the Wiki to find out where a pool file is
referenced
<https://github.com/backuppc/backuppc/wiki/How-to-find-which-backups-reference-a-particular-pool-file>.
That's great, thanks!
Indeed, thanks!
I'll check those 2 files tonight, and hopefully have a script working by the weekend.
Cool! If you don't mind and are allowed to, please share here... Cheers, Alex
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
