themanwittaplan17 wrote:
tar -c<arguments> | tarlz -z9 --no-solid -o /mnt/backup/datadir.tar.lz
This works perfectly in my setup. I am curious...does tarlz discern between
members in this way by reading the tar headers and compressing each member
accordingly?
Yes. This is why the manual warns about global headers:
http://www.nongnu.org/lzip/manual/tarlz_manual.html#Invoking-tarlz
"Note that tarlz only works reliably on archives without global headers, or
with global headers whose content can be ignored."
For me personally, --full-time is the most important one, as nanosecond
precision for my mtimes is necessary.
I'll give priority to --full-time then.
I see FEC support in lziprecover. Do these files essentially function the same
as Parchive[1] files?
Yes, but lziprecover's FEC is simpler in general and more efficient than
Parchive for small files. Also lziprecover's ability to repair lzip files
using FEC is greater than for other kinds of files.
I noticed on tarlz manual that POSIX.1-2008 backward compatibility is removed
(zeroing truncated filenames etc) on tarballs it creates. Can it do this to
tarballs I pass to it with GNU tar? Can it also calculate and record CRC32s?
Currently no and no, but adding CRCs to the extended records is in my TODO
list, and zeroing truncated filenames can be implemented at the same time.
Stay tuned. :-)
Lastly, have you considered hashing algorithms like XXH32 (of xxHash[2]) or
BLAKE3[3]? I suggest these because of known collisions sometimes occurring in
CRC32 (example: many microsoft ISO's have FFFF:FFFF crc32[4]).
Those are not collisions[1], and are 0xFFFF_FFFF instead of zero because of
the final inversion in the computation of the CRC. The integrity of those
ISO files can be checked; a corrupt file will produce a combined CRC
different from 0xFFFF_FFFF. In fact, CRC32(-C) is optimum for uses like
those of tarlz and lzip.
[1]
https://en.wikipedia.org/wiki/Computation_of_cyclic_redundancy_checks#One-pass_checking
"When the CRC is transmitted with the correct byte order (matching the
chosen bit-ordering convention), a receiver can compute an overall CRC, over
the message and the CRC, and if they are correct, the result will be zero."
Best regards,
Antonio.
