Re: plzip: manual gives very false numbers, real defaults are huge!
Hi Steffen, Steffen Nurpmeso wrote: The package system (no info, no README or the like are installed, only manuals) has a readme Print the port’s README file if it exists; if set, uses $PAGER option, it prints a port-specific README file; would it be ok for you if i note the presence of this non-upstream patch in such a README file, or do i have to change the program name as such? I think you should modify at least the output of 'plzip --version' to make clear that it is a modified version without having to read a separate README file. Most probably, any user having trouble with the modified plzip will look at 'plzip --version' to find out what version he is running. As the preamble of the GPL says: "If the software is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introduced by others will not reflect on the original authors' reputations." I could also patch the manual to note the presence of the non-upstream patch within the installed package. You should. Or else the users wouldn't know how to use the feature. ;-) Best regards, Antonio.
Re: plzip: manual gives very false numbers, real defaults are huge!
Hi Steffen, Steffen Nurpmeso wrote: #?0|kent:plzip-1.11$ cp /x/balls/gcc-13.2.0.tar.xz X1 #?0|kent:plzip-1.11$ cp X1 X2 [...] -rw-r- 1 steffen steffen 89049959 May 7 22:14 X1.lz -rw-r- 1 steffen steffen 89079463 May 7 22:14 X2.lz Note that if you use uncompressible files as input, you'll always obtain similar compressed sizes, no matter the compression level or the dictionary size. Try the test with gcc-13.2.0.tar and you'll see the difference. (As in your other test with /x/doc/coding/austin-group/202x_d4.txt). I think dynamically scalling according to the processors, talking into account the dictionary size, as you said above, is the sane approach for "saturating" with plzip, in the above job there are quite a lot of files, of varying size (the spam DB being very large), and one recipe is not good for them all. Maybe there is a better way (almost optimal for many files) to compress the spam DB that does not require a parallel compressor, but uses all the processors in your machine. (And, as a bonus, achieves maximum compression on files of any size and produces reproducible files). ls | xargs -n1 -P4 lzip -9 The command above should produce better results than a saturated plzip. 'ls' may be replaced by any way to generate a list of the files to be compressed. See http://www.gnu.org/software/findutils/manual/html_node/find_html/xargs-options.html Hope this helps, Antonio.
Re: plzip: manual gives very false numbers, real defaults are huge!
Steffen Nurpmeso wrote: The above means "on files of only a few MB plzip can't be faster than lzip, no matter what options you use". Of course, at high compression levels the "few MB" become "several tens of MB". I think i now have understood your approach. But i claim it is not what people would expect. People tends to expect contradictory things. Like using all the processors on any machine while at the same time producing the same compressed output on all machines. For example, if i hack just a little bit i get on my i5 laptop #?0|kent:plzip-1.11$ time ./plzip -9 -n4 x1 instat=0x7fff32eb6800 inreg=1 sat=0 cf=680412 x=170103 tmp=67108864 USING 67108864 real0m37.743s user0m37.737s sys 0m0.273s [...] #?0|kent:plzip-1.11$ time ./plzip -9 -n0 x1 instat=0x7ffe538049d0 inreg=1 sat=1 cf=680412 x=170103 tmp=67108864 USING 170103 real0m3.157s user0m12.415s sys 0m0.087s Note that the above does not run 12 times faster because you have 12 processors, but because you are using a dictionary size almost 200 times smaller (which I guess will give a compression ratio between levels 0 and 1 instead of the level 9 requested). I realized for the first time that standard input is treated in a different way via the "one_to_one" mapping of yours. Ie while doing time ./plzip -9 -n0 -c < /tmp/t.tar.xz > x1.lz it occurred to me that the "struct stat" is not used at all for stdin, which is a pity imho, especially since S_ISREG() is tested. S_ISREG is not tested for stdin. But if you want to reproduce the metadata of the input file in an output file with a different name, you can use time ./plzip -9 /tmp/t.tar.xz -o x1.lz That is true. If not a regular file, then the above saturation will unfortunately not work out. Yet, i thought, limiting a data size that the user did not explicitly set in the user required saturation mode could at least minimize the damage a bit: Or can cause the opposite damage by splitting a huge file into twice as many members as now. And i hope the people of reproducible-builds.org now always check their environment before penaltizing aka flagging other people's work. Reproducible builds are a set of software development practices that create an independently-verifiable path from source to binary code. They have nothing to do with reproducible compression. (Obtaining identical compressed output from identical uncompressed input read from anywhere). i find myself using ZEXE='plzip -9 -B16MiB -n'"$NPROC"' -c' ZEXT=lz for this to not end up taking dozens of minutes. I already gave you a solution; use -m or -s: ZEXE='plzip -m273 -n'"$NPROC"' -c' ZEXT=lz or ZEXE='plzip -9 -s8MiB -n'"$NPROC"' -c' ZEXT=lz The above would at least half the necessary time. Sure. The above is old and maybe totally useless when using things like -k and -f. Hm. I guess you could simplify it to something like this ZEXE='plzip -m273 -n"$NPROC"' $ZEXE -kf FILE || exit 5 Best regards, Antonio.
Re: plzip: manual gives very false numbers, real defaults are huge!
Steffen Nurpmeso wrote: Thanks for the quick response on a Saturday. You are welcome. :-) Note that the number of usable threads is limited by file size; on files larger than a few GB plzip can use hundreds of processors, but on files of only a few MB plzip is no faster than lzip. Ok "you get scaling effects", but 70 MiB is not "a few MiB". The above means "on files of only a few MB plzip can't be faster than lzip, no matter what options you use". Of course, at high compression levels the "few MB" become "several tens of MB". 67 megabytes per processor! (How about doing a stat and somehow taking into account st_size? Or fstat, after the file was opened? This would break reproducibility (obtaining identical compressed output from identical input) because the size of uncompressed data read from standard input (not from a file) can't be known in advance. A single sentence that the "defaults" are (of course?!!?!) dependent on the compression level would have shed some enlightening. I'll try to document it better in the manual and in the man page. (Having read the referenced section in the .info file in the source tarball i would open an issue as "wishlist" asking for an option that would scale-to-"a-reasonable-number-of"-cpus. As I said above, such an option would not work with data read from standard input, and would break reproducibility. Best regards, Antonio.
Re: plzip: manual gives very false numbers, real defaults are huge!
Hello Steffen, Steffen Nurpmeso wrote: plzip: manual gives very false numbers, real defaults are huge! But if you pass options to plzip (-9 -n4) then you are no longer using the defaults. ;-) but while compressing a 70MB file i realized it was not multithreaded. At compression level 9 and 4 threads you need at least a 256 MiB file. See http://www.nongnu.org/lzip/manual/plzip_manual.html#Minimum-file-sizes (-n4 i never did; the manual however says two is default, which is not true, mind you) The man page (plzip.1) is a short reference created automatically with help2man from 'plzip --help', and reports the number of processors in my machine. You need to run 'plzip --help' yourself to see the number of processors in your machine. The real manual can be accessed with 'info plzip' or at http://www.nongnu.org/lzip/manual/plzip_manual.html But the thing is, if i do plzip -9 -n4 -B1600 -c < 76-MiB-file > au.lz aka use the values the manual describes, then i get 263 and then 400 until the end, in my poor man's top(1). Correct. The command above first sets a --dictionary-size of 32 MiB and a --data-size of 64 MiB (with -9), then reduces the --data-size to 16MB (with -B). Please let me make the statement that a default of 67 MiB for data size is really too much, The default (if you don't specify a compression level) is 16 MiB as documented. If you want a high compression level without altering the default --dictionary-size and --data-size, you can use plzip -m273 -n4 -c < 76-MiB-file > au.lz In *my* opinion, the defaults should satisfy the occasional "i wanna compress something" lady (Diaz is south of Texas, is it?), instead of those "i want to compress my 10 GiB scientific database file" specialists. Maybe the occasional "i wanna compress something" lady would be better served by using plain lzip, which compresses equally well files of any size. About malloc, my plan is to stick to standard usage. I do not plan to play with it in any way. Best regards, Antonio.
Re: [tarlz] testsuite failure on Alpine (edge)
jd...@ube.sent.com wrote: so it seems that we are on the right track with the unsupported 'T' separator. will just patch out the section as before and put tarlz to work around the office. Thanks again for the feedback. I'll modify the test to skip the dates that the system's 'touch' can't manage because that seems to mean that the system itself (mktime) is unable to manage them. thanks for the lzip suite of utilities: they saved us from a ton of recovery work over the years. You are welcome. :-) Best regards, Antonio.
Re: [tarlz] testsuite failure on Alpine (edge)
hello J Dean, Thank you very much for reporting this. J Dean wrote: testing --create...touch: invalid date '2022-01-05T12:22:13' [...] could this be a busybox touch quirk? only by patching the --create --mtime test section out does the build complete with all other tests passing. This seems to be the case; the touch command you are using probably does not understand the 'T' separator in '2022-01-05T12:22:13' and requires a space instead. You may try to work around this limitation of touch by changing line 749 of testsuite/check.sh to something like the following: touch -d 2022-01-05T12:22:13 bar >/dev/null 2>&1 || touch -d '2022-01-05 12:22:13' bar || framework_failure would like to package this for Alpine (lzip/lzlib/plzip are already in the official repos). is there a workaround, perhaps an alternative test? or can the test be patched out as above and the resultant binary be used in production, despite the check failure? If the workaround above does not work, you may remove the test and use the binary until I release a version that skips the test in this case. This is not a bug in tarlz, but a limitation of this particular touch. Best regards, Antonio.
[Lzip-bug] Lzlib 1.15-pre1 released
Lzlib 1.15-pre1 is ready for testing here http://download.savannah.gnu.org/releases/lzip/lzlib/lzlib-1.15-pre1.tar.lz http://download.savannah.gnu.org/releases/lzip/lzlib/lzlib-1.15-pre1.tar.gz The sha256sums are: b801f5fca2623078853a967793447fd81cb7dfb687ab7aed79ae8c4cac532795 lzlib-1.15-pre1.tar.lz f58f261dad22dc347554fc1cf643b4b3a7d0b06b4ed8a0691d7329e4ebe003c0 lzlib-1.15-pre1.tar.gz Please, test it and report any bugs you find. Lzlib is a data compression library providing in-memory LZMA compression and decompression functions, including integrity checking of the decompressed data. The compressed data format used by the library is the lzip format. Lzlib is written in C. Lzlib is free software distributed under a 2-clause BSD license. The homepage is at http://www.nongnu.org/lzip/lzlib.html Changes in this version: * The targets 'lib' and 'bin' have been added to Makefile.in. 'lib' is the new default and builds just the library. 'bin' builds both the library and minilzip. * minilzip is no longer built by default. * 'install-bin' installs minilzip and its man page again. * The use of the target 'bin' has been documented in INSTALL. Regards, Antonio Diaz, lzlib author and maintainer. -- If you are using gzip, bzip2, or xz, please consider the long-term advantages of switching to lzip: http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance http://www.nongnu.org/lzip/safety_of_the_lzip_format.html
Re: [lzlib] Questions regarding minilzip
Hi NRK, NRK wrote: I'm a bit a confused about the minilzip program included in lzlib, the manpage says it's a test program, but also says that it's compatible with lzip 1.4. "compatible" in the sense that it is interoperable (it can decompress files produced by lzip, and vice versa), not in the sense of offering identical functionality. All the tools in the lzip family are compatible. See for example http://www.nongnu.org/lzip/plzip.html a) What exactly is "mini" about it? Nothing really. I just adapted the name (minigzip) from the test program of zlib http://zlib.net/zlib_faq.html#faq06 b) Is it intended to be used as a functional lzip(1) replacement or should it not be installed as such and only used for testing? It is not intended to be installed because lzip has more features (like --list). But I test minilzip as thoroughly as I test lzip. AFAIK, minilzip is 100% reliable and you can use it as your main compressor if so you wish. See http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance "Three related but independent compressor implementations, lzip, clzip, and minilzip/lzlib, are developed concurrently. Every stable release of any of them is tested to check that it produces identical output to the other two. This guarantees that all three implement the same algorithm, and makes it unlikely that any of them may contain serious undiscovered errors." c) What's the "recommended" way to build the library but not build minilzip? I can currently do `make liblz.so.1.14` but that seems a bit odd. Interesting question. There is not a recommended way. I Think I have not implemented a way to build the library without building minilzip because I never build lzlib without also testing it (and minilzip is required for 'make check'). Would you be willing to add something like `make lib` and/or `make static-lib` that builds the library only? Sure. I guess it will need just 'make lib', which corresponds with 'make install-lib', because the kind of lib is selected with the configure options --disable-static and --enable-shared. I can release a new version with 'make lib' maybe tomorrow. Thanks for writing and maintaining lzip. You are welcome. :-) Best regards, Antonio.
Re: Statement about the maintenance of lzip
Hi Steffen, Steffen Nurpmeso wrote: How about malloc hookability? https://developers.redhat.com/articles/2021/08/25/securing-malloc-glibc-why-malloc-hooks-had-go "The key misfeature of the debugging hooks, though, was their presence as unprotected function pointers that were guaranteed to be executed at specific events. This made the hooks an easy exploit primitive in practically every program that ran on Linux distributions. A trivial search for __malloc_hook "house of" turns up a long list of exploit methods that use the hooks as either an intermediate step or the final goal for the exploit." I prefer to use valgrind. Someday someone will discover that preloading malloc can also be exploited. git (at least an automated conversion to Savannah / xy)? Consider lzip as a work of literature. I just won't reveal my writing habits to the world. I think the BSDs refrain because of GPL; at least it is v2 not v3, but still; zstd instead (i think: "then, later") changed to BSD-style OR GPLv2 saying "You may select, at your option, one of the above-listed licenses". There is a fully functional permissively licensed implementation of the lzip data compressor: http://www.nongnu.org/lzip/pdlzip.html Pdlzip is free software. The (de)compression code is in the public domain. The rest is under a 2-clause BSD license. Lzlib is free software distributed under a 2-clause BSD license. While CRC-32 is ok, i guess people (including me) doubt its viability for long-term archiving, especially when compared with other algorithms. It is not so terrible as years ago, since most people surely have lots of copies, and the filesystems use checksumming. But as a standalone archive, CRC-32 fails badly, for example smhash says "insecure, 8590x collisions, distrib, PerlinNoise": The tests performed by smhasher are 100% unrelated to error detection in a decompressor context. CRC32 is probably optimal to detect errors in lzip members. See http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance "Lzip, like gzip and bzip2, uses a CRC32 to check the integrity of the decompressed data because it provides optimal accuracy in the detection of errors up to a compressed size of about 16 GiB, a size larger than that of most files. In the case of lzip, the additional detection capability of the decompressor reduces the probability of undetected errors several million times more, resulting in a combined integrity checking optimally accurate for any member size produced by lzip." See also http://www.nongnu.org/lzip/safety_of_the_lzip_format.html#lzma_crc '4.1 Interaction between LZMA compression and CRC32' and '7 Conclusions': "After 14 years of testing, the MTBF of lzip can only be estimated because not even one false negative has ever been observed. If one were to continuously decompress corrupt lzip files of about one megabyte in size (10 decompressions per second), each of them containing the kind of corruption most difficult to detect (one random bit flip), then a false negative would be expected to happen every 694 million years." Best regards, Antonio.
Statement about the maintenance of lzip
In public discussions about the backdoor recently discovered in xz-utils, I have noticed that some people are considering a possible switch to lzip but are worried that the same social engineering techniques that made possible the insertion of the backdoor in xz might also be used against lzip. I would like to explain why I think this is not the case. I don't write free software for fun (even if I enjoy writing it) nor for competing with other fellow developers. I wrote lzip because there was a need to replace the lzma-alone format. I think lzip is a good format and a good software. I am very pleased that the NASA, the IANA Time Zone Database, and many others have found lzip useful. I believe in human progress and I consider lzip as one of my contributions to the common good. I am therefore very careful about the maintenance of lzip. I have been a GNU maintainer for more than 20 years and I plan to maintain lzip for many years more. But if something bad happens to me, my daughter is ready to continue maintaining it. She has been helping me for years, and is familiar with the code of my programs. In fact she has borrowed some of my code for her own programs. She has also translated the pages of some of my programs. Moreover, when the day comes that the GNU project needs to find adequate replacement maintainers for my GNU packages, it will be easy for any of the new maintainers to also maintain lzip because lzip shares a good amount of code with my GNU packages and maintaining it takes little work. So, if you are considering a possible switch to lzip, rest assured that lzip has a low probability of suffering maintenance problems in the foreseeable future. Finally I would like to show my support for Lasse Collin, who seems to be the main victim of this incident. Best regards, Antonio.
[Lzip-bug] Lzip 1.24.1 released
I am pleased to announce the release of lzip 1.24.1. Lzip is a lossless data compressor with a user interface similar to the one of gzip or bzip2. Lzip uses a simplified form of the 'Lempel-Ziv-Markov chain-Algorithm' (LZMA) stream format to maximize interoperability. The maximum dictionary size is 512 MiB so that any lzip file can be decompressed on 32-bit machines. Lzip provides accurate and robust 3-factor integrity checking. Lzip can compress about as fast as gzip (lzip -0) or compress most files more than bzip2 (lzip -9). Decompression speed is intermediate between gzip and bzip2. Lzip is better than gzip and bzip2 from a data recovery perspective. Lzip has been designed, written, and tested with great care to replace gzip and bzip2 as the standard general-purpose compressed format for Unix-like systems. The homepage is at http://www.nongnu.org/lzip/lzip.html The sources can be downloaded from http://download.savannah.gnu.org/releases/lzip/ The sha256sums are: 281564cfab331f76c2e8a112604f2339bfff2ecad4d330edc5d4b87f1904de53 lzip-1.24.1.tar.lz 30c9cb6a0605f479c496c376eb629a48b0a1696d167e3c1e090c5defa481b162 lzip-1.24.1.tar.gz This release is also GPG signed. You can download the signature by appending '.sig' to the URL. If the 'gpg --verify' command fails because you don't have the required public key, then run this command to import it: gpg --recv-keys 8FE99503132D7742 Key fingerprint = 1D41 C14B 272A 2219 A739 FA4F 8FE9 9503 132D 7742 Changes in version 1.24.1: * No changes have been made to the code. Just a compilation failure on MinGW has been fixed. (Caused by different mkdir signature in POSIX and Windows). Please send bug reports and suggestions to lzip-bug@nongnu.org Regards, Antonio Diaz, lzip author and maintainer. -- If you are using gzip, bzip2, or xz, please consider the long-term advantages of switching to lzip: http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance http://www.nongnu.org/lzip/safety_of_the_lzip_format.html
Re: mingw fails to build with latest update
Sean Link wrote: I tested both rc1 and rc2 for 1.24. Both failed. Of course both failed. ;-) The problem is that nobody tested them before I released 1.24 stable, and now I have to release 1.24.1 just to fix a compilation failure on mingw. Best regards, Antonio.
Re: mingw fails to build with latest update
Weijia Wang wrote: I can confirm that the patch fixed the regression on MinGW, both on my machine and on CI: https://github.com/NixOS/nixpkgs/pull/292137/checks?check_run_id=22158903421 Thank you very much. I'll release lzip-1.24.1 ASAP. Best regards, Antonio.
Re: mingw fails to build with latest update
Hi Sean, Sean Link wrote: See https://github.com/NixOS/nixpkgs/pull/292137 Thanks. As reported at https://github.com/NixOS/nixpkgs/pull/292137#issuecomment-1970071254 , the problem is already (hopefully) fixed upstream. It is a pity that nobody cares to test the release candidates (I released at least one RC for each tool in the lzip family that includes the mkdir change, 2 for lzip[1]). I'll release lzip-1.24.1 with the following patch applied as soon as I receive confirmation that it fixes the compilation problen on mingw: --- lzip-1.24/main.cc 2024-01-26 00:08:47.0 +0100 +++ lzip-1.24.1/src/main.cc 2024-02-06 16:04:00.0 +0100 @@ -42,8 +42,10 @@ #if defined __MSVCRT__ || defined __OS2__ || defined __DJGPP__ #include #if defined __MSVCRT__ +#include #define fchmod(x,y) 0 #define fchown(x,y,z) 0 +#define mkdir(name,mode) _mkdir(name) #define strtoull std::strtoul #define SIGHUP SIGTERM #define S_ISSOCK(x) 0 [1] http://download.savannah.gnu.org/releases/lzip/lzip-1.24-rc1.tar.lz http://download.savannah.gnu.org/releases/lzip/lzip-1.24-rc2.tar.lz Best regards, Antonio.
Re: [Lzip-bug] Please, help me to compile lzip extract support into 7zip or
wrotycz wrote: You should make prepare standalone lzip codec that could be compiled to .dll/.so and published, so people can put it in their 7zip/Codedecs directory and use. [...] I can campile it for windows but I can't make it on my own, though I tried. Thank you for your offer to help. I have downloaded 7z_xorfilter_v0.7z and will try to replace the xor codec in it with the decompression code of lzip. But I can't compile the codec because I don't own any Microsoft software. I don't even remember who compiled my patch for 7zip. Trying to integrate it into 7zip code is not going to work. You had tried, some oher people had tried, and you all had been brushed off. I guess, its because Pavlov sees it as a competitor to his child (and yes xz is also 'his', as he actively participated in it). Nothing you can do. If Igor Pavlov sees lzip as a competitor to xz and 7zip, then it is a pity because it is not. Lzip is designed for long-term archiving, while xz and 7zip are designed for features. IIRC xz can be extended to up to 2^63 filters. Best regards, Antonio.
Re: Replace http:// with https://
Francesco Turco wrote: I suggest to replace all instances of "http://; with "https://; in the source code. As far as I can see, all instances of "http://; only refer to the domains nongnu.org and gnu.org (both of which support HTTPS), so a simple global search-and-replace with sed would do the job. The problem is that many browsers on 32-bit systems do not support a recent enough version of HTTPS and can't be updated. For example, when I try to access https://lists.gnu.org/mailman/listinfo/lzip-bug I get Secure Connection Failed An error occurred during a connection to lists.gnu.org. Cannot communicate securely with peer: no common encryption algorithm(s). (Error code: ssl_error_no_cypher_overlap) Best regards, Antonio.
Re: Spelling issue
Hello Francesco, Francesco Turco wrote: I found two instances of the name "Markov" which are incorrectly capitalized in the documentation. Thank you very much for reporting this. I have fixed it in the packages affected (lzip, clzip, and lzlib). The change will appear in the next version of each package. Best regards, Antonio.
Re: mingw patch for mkdir()
wrotycz wrote: I don't know the stand of mingw of other compiler developers, what is and what is not standard/non standard, deprecated or not, but I, personally, would use mkdir, just because it does not compile on MSVC. I mean, last time I tested it compiled but did not work - it produced corrupetd bitstream somewhere on the way. I don't understand. You would use mkdir because it does not work? You'll have to add `#include ' along with `' otherwise you get: `implicit declaration of function '_mkdir''. Thank you again. I have added '#include ' to all tools. The change will appear in the next version of each tool. BTW, your mail program mangles the plain-text in your emails. Best regards, Antonio.
Re: mingw patch for mkdir()
wrotycz wrote: In (c)lzip definition of mkdir takes 2 arguments while in windows it takes 1, which causes compilation error. Thank you for reporting this. I have fixed it mostly as you suggest, but prepending an underscore as suggested by gnulib[1]. I have inserted the following at line 47 of main.cc and at corresponding lines in the rest of tools: #define mkdir(name,mode) _mkdir(name) [1] http://www.gnu.org/software/gnulib/manual/html_node/mkdir.html Best regards, Antonio.
[Lzip-bug] Lzip 1.24 released
I am pleased to announce the release of lzip 1.24. Lzip is a lossless data compressor with a user interface similar to the one of gzip or bzip2. Lzip uses a simplified form of the 'Lempel-Ziv-Markov chain-Algorithm' (LZMA) stream format to maximize interoperability. The maximum dictionary size is 512 MiB so that any lzip file can be decompressed on 32-bit machines. Lzip provides accurate and robust 3-factor integrity checking. Lzip can compress about as fast as gzip (lzip -0) or compress most files more than bzip2 (lzip -9). Decompression speed is intermediate between gzip and bzip2. Lzip is better than gzip and bzip2 from a data recovery perspective. Lzip has been designed, written, and tested with great care to replace gzip and bzip2 as the standard general-purpose compressed format for Unix-like systems. The homepage is at http://www.nongnu.org/lzip/lzip.html The sources can be downloaded from http://download.savannah.gnu.org/releases/lzip/ The sha256sums are: 345049e3747d9a588e16d4a2aff870085e200d23170f1b9f61e6ef761d675a55 lzip-1.24.tar.lz d42659229b10e066eeb6e81eb673cdd893b672e512d26719c2d95975556ca56c lzip-1.24.tar.gz This release is also GPG signed. You can download the signature by appending '.sig' to the URL. If the 'gpg --verify' command fails because you don't have the required public key, then run this command to import it: gpg --recv-keys 8FE99503132D7742 Key fingerprint = 1D41 C14B 272A 2219 A739 FA4F 8FE9 9503 132D 7742 Changes in version 1.24: * The option '--empty-error', which forces exit status 2 if any empty member is found, has been added. * The option '--marking-error', which forces exit status 2 if the first LZMA byte is non-zero in any member, has been added. * File diagnostics have been reformatted as 'PROGRAM: FILE: MESSAGE'. * Diagnostics caused by invalid arguments to command-line options now show the argument and the name of the option. * The option '-o, --output' now preserves dates, permissions, and ownership of the file when (de)compressing exactly one file. * The option '-o, --output' now creates missing intermediate directories when writing to a file. * The variable MAKEINFO has been added to configure and Makefile.in. Please send bug reports and suggestions to lzip-bug@nongnu.org Regards, Antonio Diaz, lzip author and maintainer. -- If you are using gzip, bzip2, or xz, please consider the long-term advantages of switching to lzip: http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance http://www.nongnu.org/lzip/safety_of_the_lzip_format.html
[Lzip-bug] Clzip 1.14 released
I am pleased to announce the release of clzip 1.14. Clzip is a C language version of lzip, compatible with lzip 1.4 or newer. As clzip is written in C, it may be easier to integrate in applications like package managers, embedded devices, or systems lacking a C++ compiler. Lzip is a lossless data compressor with a user interface similar to the one of gzip or bzip2. Lzip uses a simplified form of the 'Lempel-Ziv-Markov chain-Algorithm' (LZMA) stream format to maximize interoperability. The maximum dictionary size is 512 MiB so that any lzip file can be decompressed on 32-bit machines. Lzip provides accurate and robust 3-factor integrity checking. Lzip can compress about as fast as gzip (lzip -0) or compress most files more than bzip2 (lzip -9). Decompression speed is intermediate between gzip and bzip2. Lzip is better than gzip and bzip2 from a data recovery perspective. Lzip has been designed, written, and tested with great care to replace gzip and bzip2 as the standard general-purpose compressed format for Unix-like systems. The homepage is at http://www.nongnu.org/lzip/clzip.html The sources can be downloaded from http://download.savannah.gnu.org/releases/lzip/clzip/ The sha256sums are: 78b40e42a0bcada2629eb255b5fc39a23bf793a2b20bb3c5e23c2b0fc2ed5c30 clzip-1.14.tar.lz f63fe1245b832fe07ff679e9b9b8687e9379ab613a26bfb0aca3754c8b162d73 clzip-1.14.tar.gz This release is also GPG signed. You can download the signature by appending '.sig' to the URL. If the 'gpg --verify' command fails because you don't have the required public key, then run this command to import it: gpg --recv-keys 8FE99503132D7742 Key fingerprint = 1D41 C14B 272A 2219 A739 FA4F 8FE9 9503 132D 7742 Changes in version 1.14: * The option '--empty-error', which forces exit status 2 if any empty member is found, has been added. * The option '--marking-error', which forces exit status 2 if the first LZMA byte is non-zero in any member, has been added. * File diagnostics have been reformatted as 'PROGRAM: FILE: MESSAGE'. * Diagnostics caused by invalid arguments to command-line options now show the argument and the name of the option. * The option '-o, --output' now preserves dates, permissions, and ownership of the file when (de)compressing exactly one file. * The option '-o, --output' now creates missing intermediate directories when writing to a file. * The variable MAKEINFO has been added to configure and Makefile.in. * It has been documented in INSTALL that when choosing a C standard, the POSIX features need to be enabled explicitly: ./configure CFLAGS+='--std=c99 -D_XOPEN_SOURCE=500' Please send bug reports and suggestions to lzip-bug@nongnu.org Regards, Antonio Diaz, clzip author and maintainer. -- If you are using gzip, bzip2, or xz, please consider the long-term advantages of switching to lzip: http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance http://www.nongnu.org/lzip/safety_of_the_lzip_format.html
[Lzip-bug] Lunzip 1.14 released
I am pleased to announce the release of lunzip 1.14. Lunzip is a decompressor for the lzip format written in C. Its small size makes it well suited for embedded devices or software installers that need to decompress files but don't need compression capabilities. Lunzip is compatible with lzip 1.4 or newer and is able to decompress any lzip file using as little memory as 50 kB. Lzip is a lossless data compressor with a user interface similar to the one of gzip or bzip2. Lzip uses a simplified form of the 'Lempel-Ziv-Markov chain-Algorithm' (LZMA) stream format to maximize interoperability. The maximum dictionary size is 512 MiB so that any lzip file can be decompressed on 32-bit machines. Lzip provides accurate and robust 3-factor integrity checking. Lzip can compress about as fast as gzip (lzip -0) or compress most files more than bzip2 (lzip -9). Decompression speed is intermediate between gzip and bzip2. Lzip is better than gzip and bzip2 from a data recovery perspective. Lzip has been designed, written, and tested with great care to replace gzip and bzip2 as the standard general-purpose compressed format for Unix-like systems. The homepage is at http://www.nongnu.org/lzip/lunzip.html The sources can be downloaded from http://download.savannah.gnu.org/releases/lzip/lunzip/ The sha256sums are: 0e52fb05016df3ac4585bc60441965a98f03c8bf5b49249e3c9a71f773871eff lunzip-1.14.tar.lz 70a30ca88c538b074a04a6d5fa12a57f8e89febcb9145d322e9525f3694e4cb0 lunzip-1.14.tar.gz This release is also GPG signed. You can download the signature by appending '.sig' to the URL. If the 'gpg --verify' command fails because you don't have the required public key, then run this command to import it: gpg --recv-keys 8FE99503132D7742 Key fingerprint = 1D41 C14B 272A 2219 A739 FA4F 8FE9 9503 132D 7742 Changes in version 1.14: * The option '--empty-error', which forces exit status 2 if any empty member is found, has been added. * The option '--marking-error', which forces exit status 2 if the first LZMA byte is non-zero in any member, has been added. * File diagnostics have been reformatted as 'PROGRAM: FILE: MESSAGE'. * Diagnostics caused by invalid arguments to command-line options now show the argument and the name of the option. * The option '-o, --output' now preserves dates, permissions, and ownership of the file when decompressing exactly one file. * The option '-o, --output' now creates missing intermediate directories when writing to a file. * The variable MAKEINFO has been added to configure and Makefile.in. * It has been documented in INSTALL that when choosing a C standard, the POSIX features need to be enabled explicitly: ./configure CFLAGS+='--std=c99 -D_XOPEN_SOURCE=500' Please send bug reports and suggestions to lzip-bug@nongnu.org Regards, Antonio Diaz, lunzip author and maintainer. -- If you are using gzip, bzip2, or xz, please consider the long-term advantages of switching to lzip: http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance http://www.nongnu.org/lzip/safety_of_the_lzip_format.html
[Lzip-bug] Pdlzip 1.13 released
I am pleased to announce the release of pdlzip 1.13. Pdlzip is a permissively licensed implementation of the lzip data compressor, intended for those who can't distribute (or even use) GPL licensed Free Software. The name of pdlzip comes from 'public domain lzip'. Pdlzip is written in C and is compatible with lzip 1.4 or newer. Lzip is a lossless data compressor with a user interface similar to the one of gzip or bzip2. Lzip uses a simplified form of the 'Lempel-Ziv-Markov chain-Algorithm' (LZMA) stream format to maximize interoperability. The maximum dictionary size is 512 MiB so that any lzip file can be decompressed on 32-bit machines. Lzip provides accurate and robust 3-factor integrity checking. Lzip can compress about as fast as gzip (lzip -0) or compress most files more than bzip2 (lzip -9). Decompression speed is intermediate between gzip and bzip2. Lzip is better than gzip and bzip2 from a data recovery perspective. Lzip has been designed, written, and tested with great care to replace gzip and bzip2 as the standard general-purpose compressed format for Unix-like systems. Pdlzip is also able to decompress legacy lzma-alone (.lzma) files. Pdlzip includes public domain (de)compression code from the LZMA SDK (Software Development Kit) written by Igor Pavlov. The homepage is at http://www.nongnu.org/lzip/pdlzip.html The sources can be downloaded from http://download.savannah.gnu.org/releases/lzip/pdlzip/ The sha256sums are: fe0f2d78df2aa43bea07bf7546ab9fa4bc67c8e7beb584de850d60f16fceadf0 pdlzip-1.13.tar.lz 5cf17a68dc358124329621f6edf960fdd41db44c80e323d3470e2c0ec383227d pdlzip-1.13.tar.gz This release is also GPG signed. You can download the signature by appending '.sig' to the URL. If the 'gpg --verify' command fails because you don't have the required public key, then run this command to import it: gpg --recv-keys 8FE99503132D7742 Key fingerprint = 1D41 C14B 272A 2219 A739 FA4F 8FE9 9503 132D 7742 Changes in version 1.13: * File diagnostics have been reformatted as 'PROGRAM: FILE: MESSAGE'. * Diagnostics caused by invalid arguments to command-line options now show the argument and the name of the option. * The option '-o, --output' now preserves dates, permissions, and ownership of the file when (de)compressing exactly one file. * The variable MAKEINFO has been added to configure and Makefile.in. * It has been documented in INSTALL that when choosing a C standard, the POSIX features need to be enabled explicitly: ./configure CFLAGS+='--std=c99 -D_XOPEN_SOURCE=500' Please send bug reports and suggestions to lzip-bug@nongnu.org Regards, Antonio Diaz, pdlzip co-author and maintainer. -- If you are using gzip, bzip2, or xz, please consider the long-term advantages of switching to lzip: http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance http://www.nongnu.org/lzip/safety_of_the_lzip_format.html
[Lzip-bug] Plzip 1.11 released
I am pleased to announce the release of plzip 1.11. Plzip is a massively parallel (multi-threaded) implementation of lzip, compatible with lzip 1.4 or newer. Plzip uses the compression library lzlib. Lzip is a lossless data compressor with a user interface similar to the one of gzip or bzip2. Lzip uses a simplified form of the 'Lempel-Ziv-Markov chain-Algorithm' (LZMA) stream format to maximize interoperability. The maximum dictionary size is 512 MiB so that any lzip file can be decompressed on 32-bit machines. Lzip provides accurate and robust 3-factor integrity checking. Lzip can compress about as fast as gzip (lzip -0) or compress most files more than bzip2 (lzip -9). Decompression speed is intermediate between gzip and bzip2. Lzip is better than gzip and bzip2 from a data recovery perspective. Lzip has been designed, written, and tested with great care to replace gzip and bzip2 as the standard general-purpose compressed format for Unix-like systems. Plzip can compress/decompress large files on multiprocessor machines much faster than lzip, at the cost of a slightly reduced compression ratio (0.4 to 2 percent larger compressed files). Note that the number of usable threads is limited by file size; on files larger than a few GB plzip can use hundreds of processors, but on files of only a few MB plzip is no faster than lzip. The homepage is at http://www.nongnu.org/lzip/plzip.html The sources can be downloaded from http://download.savannah.gnu.org/releases/lzip/plzip/ The sha256sums are: 51f48d33df659bb3e1e7e418275e922ad752615a5bc984139da08f1e6d7d10fd plzip-1.11.tar.lz d8e3cbe45c9222383339130e1bcc6e5e884d776b63f188896e6df67bc1d5626b plzip-1.11.tar.gz This release is also GPG signed. You can download the signature by appending '.sig' to the URL. If the 'gpg --verify' command fails because you don't have the required public key, then run this command to import it: gpg --recv-keys 8FE99503132D7742 Key fingerprint = 1D41 C14B 272A 2219 A739 FA4F 8FE9 9503 132D 7742 Changes in version 1.11: * File diagnostics have been reformatted as 'PROGRAM: FILE: MESSAGE'. * Diagnostics caused by invalid arguments to command-line options now show the argument and the name of the option. * The option '-o, --output' now preserves dates, permissions, and ownership of the file when (de)compressing exactly one file. * The option '-o, --output' now creates missing intermediate directories when writing to a file. * The variable MAKEINFO has been added to configure and Makefile.in. Please send bug reports and suggestions to lzip-bug@nongnu.org Regards, Antonio Diaz, plzip co-author and maintainer. -- If you are using gzip, bzip2, or xz, please consider the long-term advantages of switching to lzip: http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance http://www.nongnu.org/lzip/safety_of_the_lzip_format.html
[Lzip-bug] Lzlib 1.14 released
I am pleased to announce the release of lzlib 1.14. Lzlib is a compression library providing in-memory LZMA compression and decompression functions, including integrity checking of the decompressed data. The compressed data format used by lzlib is the lzip format. Lzlib is written in C. Lzlib is free software distributed under a 2-clause BSD license. Compression/decompression is done by repeatedly calling a couple of read/write functions until all the data have been processed by the library. This interface is safer and less error prone than the traditional zlib interface. The homepage is at http://www.nongnu.org/lzip/lzlib.html The sources can be downloaded from http://download.savannah.gnu.org/releases/lzip/lzlib/ The sha256sums are: e362ecccd82d4dd297df6a51b952c65d2172f9bf41a5c4590d3604d83aa519d3 lzlib-1.14.tar.lz 5acac8714ed4f306020bae660dddce706e5f8a795863679037da9fe6bf4dcf6f lzlib-1.14.tar.gz This release is also GPG signed. You can download the signature by appending '.sig' to the URL. If the 'gpg --verify' command fails because you don't have the required public key, then run this command to import it: gpg --recv-keys 8FE99503132D7742 Key fingerprint = 1D41 C14B 272A 2219 A739 FA4F 8FE9 9503 132D 7742 Changes in version 1.14: * In minilzip, file diagnostics have been reformatted as 'PROGRAM: FILE: MESSAGE'. * In minilzip, diagnostics caused by invalid arguments to command-line options now show the argument and the name of the option. * The option '-o, --output' of minilzip now preserves dates, permissions, and ownership of the file when (de)compressing exactly one file. * It has been documented in the manual that it is the responsibility of the program using lzlib to include before 'lzlib.h' some header that declares the type 'uint8_t'. (Reported by Michal Górny). * The variable MAKEINFO has been added to configure and Makefile.in. * It has been documented in INSTALL that when choosing a C standard, the POSIX features need to be enabled explicitly: ./configure CFLAGS+='--std=c99 -D_XOPEN_SOURCE=500' Please send bug reports and suggestions to lzip-bug@nongnu.org Regards, Antonio Diaz, lzlib author and maintainer. -- If you are using gzip, bzip2, or xz, please consider the long-term advantages of switching to lzip: http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance http://www.nongnu.org/lzip/safety_of_the_lzip_format.html
[Lzip-bug] Lziprecover 1.24 released
I am pleased to announce the release of lziprecover 1.24. Lziprecover is a data recovery tool and decompressor for files in the lzip compressed data format (.lz). Lziprecover is able to repair slightly damaged files (up to one single-byte error per member), produce a correct file by merging the good parts of two or more damaged copies, reproduce a missing (zeroed) sector using a reference file, extract data from damaged files, decompress files, and test integrity of files. A nice feature of the lzip format is that a corrupt byte is easier to repair the nearer it is from the beginning of the file. Therefore, with the help of lziprecover, losing an entire archive just because of a corrupt byte near the beginning is a thing of the past. The homepage is at http://www.nongnu.org/lzip/lziprecover.html The sources can be downloaded from http://download.savannah.gnu.org/releases/lzip/lziprecover/ The sha256sums are: 92d09e7e9d7c1f5a144fe631e5c20eb7e82ff6eb43d4f8ab6b260c926303d89e lziprecover-1.24.tar.lz 1d699cfaefe92eb2624a3652580bcafe0bbb98fe7818c25e6de823bcdd0d458f lziprecover-1.24.tar.gz This release is also GPG signed. You can download the signature by appending '.sig' to the URL. If the 'gpg --verify' command fails because you don't have the required public key, then run this command to import it: gpg --recv-keys 8FE99503132D7742 Key fingerprint = 1D41 C14B 272A 2219 A739 FA4F 8FE9 9503 132D 7742 Changes in version 1.24: * The option '--empty-error', which forces exit status 2 if any empty member is found, has been added. * The option '--marking-error', which forces exit status 2 if the first LZMA byte is non-zero in any member, has been added. * The option '--clear-marking', which sets to zero the first LZMA byte of each member, has been added. * The keyword 'empty' is now recognized in the argument of '--dump', '--remove', and '--strip'. * The option '--repair' has been renamed to '--byte-repair'. * The option '--debug-repair' has been renamed to '--debug-byte-repair'. * File diagnostics have been reformatted as 'PROGRAM: FILE: MESSAGE'. * Diagnostics caused by invalid arguments to command-line options now show the argument and the name of the option. * The option '-o, --output' now preserves dates, permissions, and ownership of the file when decompressing exactly one file. * The option '-o, --output' now creates missing intermediate directories when writing to a file. * The option '--no-verify' of unzcrash has been renamed to '--no-check'. * The variable MAKEINFO has been added to configure and Makefile.in. * The makefile target 'install-as-lzip' has been removed because '--reproduce' needs a lzip compressor (not just a decompressor) named 'lzip' by default. Please send bug reports and suggestions to lzip-bug@nongnu.org Regards, Antonio Diaz, lziprecover author and maintainer. -- If you are using gzip, bzip2, or xz, please consider the long-term advantages of switching to lzip: http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance http://www.nongnu.org/lzip/safety_of_the_lzip_format.html
[Lzip-bug] Lzip 1.24-rc2 released
Lzip 1.24-rc2 is ready for testing here http://download.savannah.gnu.org/releases/lzip/lzip-1.24-rc2.tar.lz http://download.savannah.gnu.org/releases/lzip/lzip-1.24-rc2.tar.gz The sha256sums are: 0f997ce64c2cdd90d62072b10f17a069fea8e6762afea11a9ae2fc6914e48dd8 lzip-1.24-rc2.tar.lz d2de91f29eb978c4d5a33bed0b08c1349b0e84abeae6569f2984c2a9229c296c lzip-1.24-rc2.tar.gz Please, test it and report any bugs you find. Lzip is a lossless data compressor with a user interface similar to the one of gzip or bzip2. Lzip uses a simplified form of the 'Lempel-Ziv-Markov chain-Algorithm' (LZMA) stream format to maximize interoperability. The maximum dictionary size is 512 MiB so that any lzip file can be decompressed on 32-bit machines. Lzip provides accurate and robust 3-factor integrity checking. Lzip can compress about as fast as gzip (lzip -0) or compress most files more than bzip2 (lzip -9). Decompression speed is intermediate between gzip and bzip2. Lzip is better than gzip and bzip2 from a data recovery perspective. Lzip has been designed, written, and tested with great care to replace gzip and bzip2 as the standard general-purpose compressed format for Unix-like systems. The homepage is at http://www.nongnu.org/lzip/lzip.html Changes in this version: * The option '-o, --output' now preserves dates, permissions, and ownership of the file when (de)compressing exactly one file. Regards, Antonio Diaz, lzip author and maintainer. -- If you are using gzip, bzip2, or xz, please consider the long-term advantages of switching to lzip: http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance http://www.nongnu.org/lzip/safety_of_the_lzip_format.html
[Lzip-bug] Xlunzip 0.8 released
I am pleased to announce the release of xlunzip 0.8. Xlunzip is a test tool for the lzip decompression code of my lzip patch for linux. Xlunzip is similar to lunzip, but it uses the lzip_decompress linux module as a backend. Xlunzip tests the module for stream, buffer-to-buffer, and mixed decompression modes, including in-place decompression (using the same buffer for input and output). You can use xlunzip to check that the module produces correct results when decompressing single member files, multimember files, or the concatenation of two or more compressed files. Xlunzip can be used with unzcrash to test the robustness of the module to the decompression of corrupted data. The distributed index feature of the lzip format allows xlunzip to decompress concatenated files in place. This can't be guaranteed to work with formats like gzip or bzip2 because they can't detect whether a high compression ratio in the first members of the multimember data is being masked by a low compression ratio in the last members. The xlunzip tarball contains a copy of the lzip_decompress module and can be compiled and tested without downloading or applying the patch to the kernel. The homepage is at http://www.nongnu.org/lzip/xlunzip.html The sources can be downloaded from http://download.savannah.gnu.org/releases/lzip/xlunzip/ The sha256sum is: 38923014b922a4c491b285c5f9ba1d92bb5769effa35efe29ef625245990ca01 xlunzip-0.8.tar.lz The lzip patch for linux can be found at http://download.savannah.gnu.org/releases/lzip/kernel/ Changes in version 0.8: * File diagnostics have been reformatted as 'PROGRAM: FILE: MESSAGE'. * In case of error in a numerical argument to a command line option, lunzip now shows the name of the option and the range of valid values. * Diagnostics caused by invalid arguments to command-line options now show the argument and the name of the option. * The option '-o, --output' now preserves dates, permissions, and ownership of the file, when decompressing exactly one file. * The variable MAKEINFO has been added to configure and Makefile.in. * It has been documented in INSTALL that when choosing a C standard, the POSIX features need to be enabled explicitly: ./configure CFLAGS+='--std=c99 -D_XOPEN_SOURCE=500' Please send bug reports and suggestions to lzip-bug@nongnu.org Regards, Antonio Diaz, xlunzip author and maintainer. -- If you are using gzip, bzip2, or xz, please consider the long-term advantages of switching to lzip: http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance http://www.nongnu.org/lzip/safety_of_the_lzip_format.html
Re: stale link on https://www.nongnu.org/lzip/lzip.html
Hello Roland, Roland wrote: I just noted that the link http://www.cs.indiana.edu/~kinzler/z/ at the bottom of the section https://www.nongnu.org/lzip/lzip.html#links does not lead to a web page. Thank you very much for reporting this. It seems that 'z' has moved to https://kinzler.com/me/z/ . I'll update the link ASAP. Best regards, Antonio.
[Lzip-bug] Tarlz 0.25 released
I am pleased to announce the release of tarlz 0.25. Tarlz is a massively parallel (multi-threaded) combined implementation of the tar archiver and the lzip compressor. Tarlz uses the compression library lzlib. Tarlz creates tar archives using a simplified and safer variant of the POSIX pax format compressed in lzip format, keeping the alignment between tar members and lzip members. The resulting multimember tar.lz archive is backward compatible with standard tar tools like GNU tar, which treat it like any other tar.lz archive. Tarlz can append files to the end of such compressed archives. Keeping the alignment between tar members and lzip members has two advantages. It adds an indexed lzip layer on top of the tar archive, making it possible to decode the archive safely in parallel. It also minimizes the amount of data lost in case of corruption. Compressing a tar archive with plzip may even double the amount of files lost for each lzip member damaged because it does not keep the members aligned. The homepage is at http://www.nongnu.org/lzip/tarlz.html An online manual for tarlz can be found at http://www.nongnu.org/lzip/manual/tarlz_manual.html The sources can be downloaded from http://download.savannah.gnu.org/releases/lzip/tarlz/ The sha256sum is: 7d0bbe9c3a137bb93a10be56988fcf7362e4dbc65490639edc4255b704105fce tarlz-0.25.tar.lz Changes in version 0.25: * The new option '--ignore-metadata', which makes '-d, --diff' ignore differences in file permissions, owner and group IDs, and modification time, has been added. * '#include ' for major, minor, and makedev on BSD systems. * File diagnostics of '-z' have been reformatted as 'PROGRAM: FILE: MESSAGE'. * The option '-o, --output' now creates missing intermediate directories when compressing to a file. * The variable MAKEINFO has been added to configure and Makefile.in. Please send bug reports and suggestions to lzip-bug@nongnu.org Regards, Antonio Diaz, tarlz author and maintainer. -- If you are using gzip, bzip2, or xz, please consider the long-term advantages of switching to lzip: http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance http://www.nongnu.org/lzip/safety_of_the_lzip_format.html
Request for review of Media Type application/lzip
Dear all, I'm trying to register the media type 'application/lzip' at IANA. You can see the details in the current version of the Internet Draft for lzip at https://datatracker.ietf.org/doc/draft-diaz-lzip/ As lzip is not backed/endorsed/authored by any large internet corporation, the people at the IETF want to be sure that lzip is worth the trouble before asking IANA to register the type, and have suggested me to provide some kind of independent review. I agree with them. Any media type registered in the standards tree should be well tested, free of problems, and better at least in some aspect than the similar types already registered. The rationale is that the more evidence exists of community understanding and positive reactions to the lzip draft (or at least reactions that it will not cause harm), the easier the process of AD sponsorship gets for the Area Director at the IETF. Therefore, if you have read the draft, or deployed lzip somewhere, or know of a successful deployment of lzip, please write to the media-ty...@ietf.org[1] mailing list to add your review or tell your experience. [1] If possible, answer in the thread 'Request for review of Media Type application/lzip' https://mailarchive.ietf.org/arch/msg/media-types/uv9sVTbxsMKnhxRUrNrLP6utUL0/ OTOH, if you find any error in the draft, please report it to this list so that I can fix it. Thanks. Antonio.
[Lzip-bug] Lzd 1.4 released
I am pleased to announce the release of lzd 1.4. Lzd is a simplified decompressor for the lzip format with an educational purpose. Studying its source code is a good first step to understand how lzip works. Lzd is written in C++. The source code of lzd is used in the lzip manual as a reference decompressor in the description of the lzip file format. Reading the lzip manual will help you understand the source code. Lzd is compliant with the lzip specification[1]; it checks the 3 integrity factors. [1] https://datatracker.ietf.org/doc/draft-diaz-lzip Comments about the internet draft above are solicited and should be addressed to the lzip's mailing list at lzip-bug@nongnu.org and/or the author. Thanks. The lzd homepage is at http://www.nongnu.org/lzip/lzd.html The sources can be downloaded from http://download.savannah.gnu.org/releases/lzip/lzd/ The sha256sums are: 041f12826bd9c613fdf034e02ea2c3d846caa99cba1b071ea320900d9ae998ba lzd-1.4.tar.lz 927f918bbea37bcf6b2ad14804db1d2e4578c0ffb02f900f201ea755d44b1b8a lzd-1.4.tar.gz Changes in version 1.4: * The constants 'header_size' and 'trailer_size' have been defined, instead of using 6 and 20 directly. (Lzd is supposed to be educational, and using numbers is bad practice). Regards, Antonio Diaz, lzd author and maintainer. -- If you are using gzip, bzip2, or xz, please consider the long-term advantages of switching to lzip: http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance http://www.nongnu.org/lzip/safety_of_the_lzip_format.html
[Lzip-bug] Lzip 1.24-rc1 released
Lzip 1.24-rc1 is ready for testing here http://download.savannah.gnu.org/releases/lzip/lzip-1.24-rc1.tar.lz http://download.savannah.gnu.org/releases/lzip/lzip-1.24-rc1.tar.gz The sha256sums are: d3f24664ebb93a7e1e90f7daabd5cdd5655d36e81b8ac8ac5d439ab7184125e4 lzip-1.24-rc1.tar.lz dceff3b29719dbe489c036d7247932154d09c5b683c0dff2aa74f749b4eac0c1 lzip-1.24-rc1.tar.gz Please, test it and report any bugs you find. Lzip is a lossless data compressor with a user interface similar to the one of gzip or bzip2. Lzip uses a simplified form of the 'Lempel-Ziv-Markov chain-Algorithm' (LZMA) stream format to maximize interoperability. The maximum dictionary size is 512 MiB so that any lzip file can be decompressed on 32-bit machines. Lzip provides accurate and robust 3-factor integrity checking. Lzip can compress about as fast as gzip (lzip -0) or compress most files more than bzip2 (lzip -9). Decompression speed is intermediate between gzip and bzip2. Lzip is better than gzip and bzip2 from a data recovery perspective. Lzip has been designed, written, and tested with great care to replace gzip and bzip2 as the standard general-purpose compressed format for Unix-like systems. The homepage is at http://www.nongnu.org/lzip/lzip.html Changes in this version: * The option '--empty-error', which forces exit status 2 if any empty member is found, has been added. * The option '--marking-error', which forces exit status 2 if the first LZMA byte is non-zero in any member, has been added. * File diagnostics have been reformatted as 'PROGRAM: FILE: MESSAGE'. * Diagnostics caused by invalid arguments to command-line options now show the argument and the name of the option. * The option '-o, --output' now creates missing intermediate directories when writing to a file. * The variable MAKEINFO has been added to configure and Makefile.in. Regards, Antonio Diaz, lzip author and maintainer. -- If you are using gzip, bzip2, or xz, please consider the long-term advantages of switching to lzip: http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance http://www.nongnu.org/lzip/safety_of_the_lzip_format.html
[Lzip-bug] Clzip 1.14-rc1 released
Clzip 1.14-rc1 is ready for testing here http://download.savannah.gnu.org/releases/lzip/clzip/clzip-1.14-rc1.tar.lz http://download.savannah.gnu.org/releases/lzip/clzip/clzip-1.14-rc1.tar.gz The sha256sums are: c3911d9a4d505bc5bf9136033b19295752e08615c2873312a4c11e175c9fe20f clzip-1.14-rc1.tar.lz fc9865933e0b8d576e8be33e6c7e2fb6a6d2244c314fb6bf6f52e6dd128e3ef0 clzip-1.14-rc1.tar.gz Please, test it and report any bugs you find. Clzip is a C language version of lzip, compatible with lzip 1.4 or newer. As clzip is written in C, it may be easier to integrate in applications like package managers, embedded devices, or systems lacking a C++ compiler. Lzip is a lossless data compressor with a user interface similar to the one of gzip or bzip2. Lzip uses a simplified form of the 'Lempel-Ziv-Markov chain-Algorithm' (LZMA) stream format to maximize interoperability. The maximum dictionary size is 512 MiB so that any lzip file can be decompressed on 32-bit machines. Lzip provides accurate and robust 3-factor integrity checking. Lzip can compress about as fast as gzip (lzip -0) or compress most files more than bzip2 (lzip -9). Decompression speed is intermediate between gzip and bzip2. Lzip is better than gzip and bzip2 from a data recovery perspective. Lzip has been designed, written, and tested with great care to replace gzip and bzip2 as the standard general-purpose compressed format for Unix-like systems. The homepage is at http://www.nongnu.org/lzip/clzip.html Changes in this version: * The option '--empty-error', which forces exit status 2 if any empty member is found, has been added. * The option '--marking-error', which forces exit status 2 if the first LZMA byte is non-zero in any member, has been added. * File diagnostics have been reformatted as 'PROGRAM: FILE: MESSAGE'. * Diagnostics caused by invalid arguments to command-line options now show the argument and the name of the option. * The option '-o, --output' now creates missing intermediate directories when writing to a file. * The variable MAKEINFO has been added to configure and Makefile.in. * It has been documented in INSTALL that when choosing a C standard, the POSIX features need to be enabled explicitly: ./configure CFLAGS+='--std=c99 -D_XOPEN_SOURCE=500' Regards, Antonio Diaz, clzip author and maintainer. -- If you are using gzip, bzip2, or xz, please consider the long-term advantages of switching to lzip: http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance http://www.nongnu.org/lzip/safety_of_the_lzip_format.html
[Lzip-bug] Plzip 1.11-rc1 released
Plzip 1.11-rc1 is ready for testing here http://download.savannah.gnu.org/releases/lzip/plzip/plzip-1.11-rc1.tar.lz http://download.savannah.gnu.org/releases/lzip/plzip/plzip-1.11-rc1.tar.gz The sha256sums are: 6eb398c07c3623b7f879361d7a3ec0b8e69309bc1d1a29b8fb306d1cc0aaad1d plzip-1.11-rc1.tar.lz 554b9aee3f694a19eb7dd04b06d67ac28258a4aff17cc2953b65ca2673b67ed6 plzip-1.11-rc1.tar.gz Please, test it and report any bugs you find. Plzip is a massively parallel (multi-threaded) implementation of lzip, fully compatible with lzip 1.4 or newer. Plzip uses the compression library lzlib. Lzip is a lossless data compressor with a user interface similar to the one of gzip or bzip2. Lzip uses a simplified form of the 'Lempel-Ziv-Markov chain-Algorithm' (LZMA) stream format to maximize interoperability. The maximum dictionary size is 512 MiB so that any lzip file can be decompressed on 32-bit machines. Lzip provides accurate and robust 3-factor integrity checking. Lzip can compress about as fast as gzip (lzip -0) or compress most files more than bzip2 (lzip -9). Decompression speed is intermediate between gzip and bzip2. Lzip is better than gzip and bzip2 from a data recovery perspective. Lzip has been designed, written, and tested with great care to replace gzip and bzip2 as the standard general-purpose compressed format for Unix-like systems. Plzip can compress/decompress large files on multiprocessor machines much faster than lzip, at the cost of a slightly reduced compression ratio (0.4 to 2 percent larger compressed files). Note that the number of usable threads is limited by file size; on files larger than a few GB plzip can use hundreds of processors, but on files of only a few MB plzip is no faster than lzip. The homepage is at http://www.nongnu.org/lzip/plzip.html Changes in this version: * File diagnostics have been reformatted as 'PROGRAM: FILE: MESSAGE'. * Diagnostics caused by invalid arguments to command-line options now show the argument and the name of the option. * The option '-o, --output' now creates missing intermediate directories when writing to a file. * The variable MAKEINFO has been added to configure and Makefile.in. Regards, Antonio Diaz, plzip co-author and maintainer. -- If you are using gzip, bzip2, or xz, please consider the long-term advantages of switching to lzip: http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance http://www.nongnu.org/lzip/safety_of_the_lzip_format.html
[Lzip-bug] Lzlib 1.14-rc1 released
Lzlib 1.14-rc1 is ready for testing here http://download.savannah.gnu.org/releases/lzip/lzlib/lzlib-1.14-rc1.tar.lz http://download.savannah.gnu.org/releases/lzip/lzlib/lzlib-1.14-rc1.tar.gz The sha256sums are: 291fe6ed1830632908c1c66f3a85ecb7ad4ea457b7908762dd1bfefd7bad68c6 lzlib-1.14-rc1.tar.lz 08f03e354021be2113e3588471d3915db61220153406d8c9b2936a8e9f4b674d lzlib-1.14-rc1.tar.gz Please, test it and report any bugs you find. Lzlib is a data compression library providing in-memory LZMA compression and decompression functions, including integrity checking of the decompressed data. The compressed data format used by the library is the lzip format. Lzlib is written in C. The homepage is at http://www.nongnu.org/lzip/lzlib.html Changes in this version: * In minilzip, file diagnostics have been reformatted as 'PROGRAM: FILE: MESSAGE'. * In minilzip, diagnostics caused by invalid arguments to command-line options now show the argument and the name of the option. * It has been documented in the manual that it is the responsibility of the program using lzlib to include before 'lzlib.h' some header that declares the type 'uint8_t'. (Reported by Michal Górny). * The variable MAKEINFO has been added to configure and Makefile.in. * It has been documented in INSTALL that when choosing a C standard, the POSIX features need to be enabled explicitly: ./configure CFLAGS+='--std=c99 -D_XOPEN_SOURCE=500' Regards, Antonio Diaz, lzlib author and maintainer. -- If you are using gzip, bzip2, or xz, please consider the long-term advantages of switching to lzip: http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance http://www.nongnu.org/lzip/safety_of_the_lzip_format.html
[Lzip-bug] Lziprecover 1.24-rc1 released
Lziprecover 1.24-rc1 is ready for testing here http://download.savannah.gnu.org/releases/lzip/lziprecover/lziprecover-1.24-rc1.tar.lz http://download.savannah.gnu.org/releases/lzip/lziprecover/lziprecover-1.24-rc1.tar.gz The sha256sums are: 4f633adad14babba394d3c0ddf171eef6545323a6e156bd8a69a728aefed4e64 lziprecover-1.24-rc1.tar.lz 0d473034aafe9e24ee91ba69eba9c83e7dafd5a7c6986b8e2b3b4587628080b7 lziprecover-1.24-rc1.tar.gz Please, test it and report any bugs you find. Lziprecover is a data recovery tool and decompressor for files in the lzip compressed data format (.lz). Lziprecover is able to repair slightly damaged files (up to one single-byte error per member), produce a correct file by merging the good parts of two or more damaged copies, reproduce a missing (zeroed) sector using a reference file, extract data from damaged files, decompress files, and test integrity of files. Lziprecover can remove the damaged members from multimember files, for example multimember tar.lz archives. Lziprecover provides random access to the data in multimember files; it only decompresses the members containing the desired data. Lziprecover facilitates the management of metadata stored as trailing data in lzip files. A nice feature of the lzip format is that a corrupt byte is easier to repair the nearer it is from the beginning of the file. Therefore, with the help of lziprecover, losing an entire archive just because of a corrupt byte near the beginning is a thing of the past. If the cause of file corruption is a damaged medium, the combination GNU ddrescue[1] + lziprecover is the recommended option for recovering data from damaged lzip files. The homepage is at http://www.nongnu.org/lzip/lziprecover.html Changes in this version: * File diagnostics have been reformatted as 'PROGRAM: FILE: MESSAGE'. * The option '-o, --output' now creates missing intermediate directories when writing to a file. * The variable MAKEINFO has been added to configure and Makefile.in. * The makefile target 'install-as-lzip' has been removed because --reproduce needs a lzip compressor (not just a decompressor) named 'lzip' by default. [1] http://www.gnu.org/software/ddrescue/ddrescue.html Regards, Antonio Diaz, lziprecover author and maintainer. -- If you are using gzip, bzip2, or xz, please consider the long-term advantages of switching to lzip: http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance and http://www.nongnu.org/lzip/safety_of_the_lzip_format.html Thanks.
[Lzip-bug] Lunzip 1.14-rc1 released
Lunzip 1.14-rc1 is ready for testing here http://download.savannah.gnu.org/releases/lzip/lunzip/lunzip-1.14-rc1.tar.lz http://download.savannah.gnu.org/releases/lzip/lunzip/lunzip-1.14-rc1.tar.gz The sha256sums are: 85c14e7b4e1294b4610b71d2201e3b86a87ba223e1a85f42c4a2928eb94dcb9f lunzip-1.14-rc1.tar.lz 1e738b8d440ae20a237fd4d8f90e85385a2499c44cc8bae31474deeb817ae64e lunzip-1.14-rc1.tar.gz Please, test it and report any bugs you find. Lunzip is a decompressor for the lzip format written in C. Its small size makes it well suited for embedded devices or software installers that need to decompress files but don't need compression capabilities. Lunzip is fully compatible with lzip 1.4 or newer and is able to decompress any lzip file using as little memory as 50 kB. The homepage is at http://www.nongnu.org/lzip/lunzip.html Changes in this version: * The option '--empty-error', which forces exit status 2 if any empty member is found, has been added. * The option '--marking-error', which forces exit status 2 if the first LZMA byte is non-zero in any member, has been added. * File diagnostics have been reformatted as 'PROGRAM: FILE: MESSAGE'. * Diagnostics caused by invalid arguments to command-line options now show the argument and the name of the option. * The option '-o, --output' now creates missing intermediate directories when writing to a file. * The variable MAKEINFO has been added to configure and Makefile.in. * It has been documented in INSTALL that when choosing a C standard, the POSIX features need to be enabled explicitly: ./configure CFLAGS+='--std=c99 -D_XOPEN_SOURCE=500' Regards, Antonio Diaz, lunzip author and maintainer. -- If you are using gzip, bzip2, or xz, please consider the long-term advantages of switching to lzip: http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance http://www.nongnu.org/lzip/safety_of_the_lzip_format.html
[Lzip-bug] Pdlzip 1.13-rc1 released
Pdlzip 1.13-rc1 is ready for testing here http://download.savannah.gnu.org/releases/lzip/pdlzip/pdlzip-1.13-rc1.tar.lz http://download.savannah.gnu.org/releases/lzip/pdlzip/pdlzip-1.13-rc1.tar.gz The sha256sums are: ec8b14c901336d5e966d915f87d8c55cc11d3d4b89baf984718c01937aee2f08 pdlzip-1.13-rc1.tar.lz 371edbcf82cdb3a00b8649bec066aba70eda8fcb8e08fbf5b3d5a0ce1f6ae99b pdlzip-1.13-rc1.tar.gz Please, test it and report any bugs you find. Pdlzip is a permissively licensed implementation of the lzip data compressor, intended for those who can't distribute (or even use) GPL licensed Free Software. The name of pdlzip comes from 'public domain lzip'. Pdlzip is written in C and is (hope)fully compatible with lzip 1.4 or newer. Lzip is a lossless data compressor with a user interface similar to the one of gzip or bzip2. Lzip uses a simplified form of the 'Lempel-Ziv-Markov chain-Algorithm' (LZMA) stream format to maximize interoperability. The maximum dictionary size is 512 MiB so that any lzip file can be decompressed on 32-bit machines. Lzip provides accurate and robust 3-factor integrity checking. Lzip can compress about as fast as gzip (lzip -0) or compress most files more than bzip2 (lzip -9). Decompression speed is intermediate between gzip and bzip2. Lzip is better than gzip and bzip2 from a data recovery perspective. Lzip has been designed, written, and tested with great care to replace gzip and bzip2 as the standard general-purpose compressed format for Unix-like systems. Pdlzip is also able to decompress legacy lzma-alone (.lzma) files. Lzma-alone is a very bad format; it is essentially a raw LZMA stream. If you keep any lzma-alone files, it is advisable to recompress them to lzip format. Lziprecover can convert some lzma-alone files to lzip format without recompressing. Pdlzip includes public domain compression/decompression code from the LZMA SDK (Software Development Kit) written by Igor Pavlov. The homepage is at http://www.nongnu.org/lzip/pdlzip.html Changes in this version: * File diagnostics have been reformatted as 'PROGRAM: FILE: MESSAGE'. * Diagnostics caused by invalid arguments to command-line options now show the argument and the name of the option. * The variable MAKEINFO has been added to configure and Makefile.in. * It has been documented in INSTALL that when choosing a C standard, the POSIX features need to be enabled explicitly: ./configure CFLAGS+='--std=c99 -D_XOPEN_SOURCE=500' Regards, Antonio Diaz, pdlzip co-author and maintainer. -- If you are using gzip, bzip2, or xz, please consider the long-term advantages of switching to lzip: http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance http://www.nongnu.org/lzip/safety_of_the_lzip_format.html
Re: 0.24 build failure on macos
Hi Rui, Rui Chen wrote: I am trying to upgrade tarlz to 0.24 for homebrew. But need some build patch as below for major,minor,makedev on OSX builds Thank you very much for reporting this. makedev, major, and minor are annoying because the POSIX pax format requires them, but POSIX leaves them unspecified. GNU/Linux systems define them in sys/sysmacros.h[1], while BSD systems define them in sys/types.h[2]. [1] http://manpages.ubuntu.com/manpages/focal/en/man3/minor.3.html [2] http://man.openbsd.org/minor.3 I have added an #else clause to create.cc, decode.cc, and decode_lz.cc that I hope fixes the problem: create.cc: #if !defined __FreeBSD__ && !defined __OpenBSD__ && !defined __NetBSD__ && \ !defined __DragonFly__ && !defined __APPLE__ && !defined __OS2__ #include // for major, minor #else #include // for major, minor #endif decode.cc, decode_lz.cc: #if !defined __FreeBSD__ && !defined __OpenBSD__ && !defined __NetBSD__ && \ !defined __DragonFly__ && !defined __APPLE__ && !defined __OS2__ #include // for major, minor, makedev #else #include // for major, minor, makedev #endif Best regards, Antonio.
Re: LZSS is not LZ77
wrotycz wrote: So, in my humble opinion, to do be precise and do justice to creators of the scheme, namely Storer and Szymanski, in Readme it would be good to state about LZSS rather than LZ77 as they have as much in common as Stevenson's locomotive and TGV. I have never seen neither the papers from Lempel and Ziv nor the paper from Storer and Szymanski, and most probably Igor Pavlov (the inventor of LZMA) has not seen them either. One could say that LZMA is based on the LZ77 concept of deduplicating strings of text, rather than in the specifics of LZ77 (or LZSS) itself. In fact, all one needs to know to develop a LZ-style compression algorithm is the following idea: https://en.wikipedia.org/wiki/LZ77 "LZ77 algorithms achieve compression by replacing repeated occurrences of data with references to a single copy of that data existing earlier in the uncompressed data stream. A match is encoded by a pair of numbers called a length-distance pair, which is equivalent to the statement "each of the next length characters is equal to the characters exactly distance characters behind it in the uncompressed stream"." IMO the pseudocode at https://en.wikipedia.org/wiki/LZ77#Pseudocode describes well the fast variant of LZMA used by lzip. You may compare it with the description of lzip at https://datatracker.ietf.org/doc/html/draft-diaz-lzip-08#section-3.5: "The fast encoder in the [lzip] reference implementation shows how this can be done in almost the simplest way possible; issuing the longest match found, or a literal byte if no match is found, and repeating until all the data have been compressed." Moreover, https://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Storer%E2%80%93Szymanski says: "The main difference between LZ77 and LZSS is that in LZ77 the dictionary reference could actually be longer than the string it was replacing." which is an important feature of LZMA. So I tend to think that LZMA has more in common with LZ77 than with LZSS. 3. CSC578B: Topics in Software Applications: "Data Compression" https://heat.csc.uvic.ca/coview/course/2020051/CSC578B Data Compression (Summer 2020) - Lecture 10 - Lempel-Ziv Schemes All I can see in the link above is the message: "We are sorry, but your web browser must have JavaScript enabled in order for this application to operate correctly." Antonio.
[Lzip-bug] Tarlz 0.24 released
I am pleased to announce the release of tarlz 0.24. Tarlz is a massively parallel (multi-threaded) combined implementation of the tar archiver and the lzip compressor. Tarlz uses the compression library lzlib. Tarlz creates tar archives using a simplified and safer variant of the POSIX pax format compressed in lzip format, keeping the alignment between tar members and lzip members. The resulting multimember tar.lz archive is fully backward compatible with standard tar tools like GNU tar, which treat it like any other tar.lz archive. Tarlz can append files to the end of such compressed archives. Keeping the alignment between tar members and lzip members has two advantages. It adds an indexed lzip layer on top of the tar archive, making it possible to decode the archive safely in parallel. It also minimizes the amount of data lost in case of corruption. Compressing a tar archive with plzip may even double the amount of files lost for each lzip member damaged because it does not keep the members aligned. Tarlz can create tar archives with five levels of compression granularity: per file (--no-solid), per block (--bsolid, default), per directory (--dsolid), appendable solid (--asolid), and solid (--solid). It can also create uncompressed tar archives. Of course, compressing each file (or each directory) individually can't achieve a compression ratio as high as compressing solidly the whole tar archive, but it has the following advantages: * The resulting multimember tar.lz archive can be decompressed in parallel, multiplying the decompression speed. * New members can be appended to the archive (by removing the end-of-archive member), and unwanted members can be deleted from the archive. Just like an uncompressed tar archive. * It is a safe POSIX-style backup format. In case of corruption, tarlz can extract all the undamaged members from the tar.lz archive, skipping over the damaged members, just like the standard (uncompressed) tar. Moreover, the option '--keep-damaged' can be used to recover as much data as possible from each damaged member, and lziprecover can be used to recover some of the damaged members. * A multimember tar.lz archive is usually smaller than the corresponding solidly compressed tar.gz archive, except when individually compressing files smaller than about 32 KiB. Note that the POSIX pax format has a serious flaw. The metadata stored in pax extended records are not protected by any kind of check sequence. Because of this, tarlz protects the extended records with a Cyclic Redundancy Check (CRC) in a way compatible with standard tar tools. The homepage is at http://www.nongnu.org/lzip/tarlz.html An online manual for tarlz can be found at http://www.nongnu.org/lzip/manual/tarlz_manual.html The sources can be downloaded from http://download.savannah.gnu.org/releases/lzip/tarlz/ The sha256sum is: 49838effe95acb29d548b7ef2ddbb4b63face40536df0d9a80a62900c7170576 tarlz-0.24.tar.lz Changes in version 0.24: * The option '-C, --directory' is now position-dependent also for diff and extract. (Reported by Devon Sean McCullough). * Option '--uncompressed' can now be omitted if it can be deduced from the archive name. (An uncompressed archive name lacks a '.lz' or '.tlz' extension). * The diagnostic shown when a file being archived or an archive being compressed fails to read, now shows the cause of the failure; end-of-file or read error. * '-z, --compress' now exits with error status 2 if any input archive is an empty file. * A failure in the '--diff' test of the testsuite on OS/2 has been fixed. (Reported by Elbert Pol). Please send bug reports and suggestions to lzip-bug@nongnu.org Regards, Antonio Diaz, tarlz author and maintainer. -- If you care about data safety and long-term archiving, please consider using lzip. See http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance and http://www.nongnu.org/lzip/safety_of_the_lzip_format.html Thanks.
Re: tarlz --diff broken
Devon Sean McCullough wrote: The reason why I implemented --diff as I did is that the documentation for GNU tar seems to imply that --directory is order sensitive only for --create: LOL, now to test GNU tar for the same confusion, and if present, report that bug as well - as if I didn't have enough to do already! As soon as I have the feature implemented in tarlz, I plan to report that the documentation for -C in GNU tar is incomplete. (Unless somebody else reports it first). It seems that making -C order sensitive for anything except create and append is not so easy to do efficiently. The code for this in GNU tar is so intricate that I have needed to run it to check for sure for what operations -C is order sensitive. It will take me some time to figure out how to make it efficient without the complications of GNU tar. P.S. I use # (time tarlz ...) \ | (pv --average-rate --bytes --eta --size $SIZE) \ | (cd /exfat-sd-card && split ...) to make backups on pricey but handy SD Cards preformatted with EXFAT, universally readable but lacking large file support, after # (time tarlz --uncompressed ...) | wc -c for a quick $SIZE guesstimate. Any thoughts on --multi-volume or split support? Lacking large file support is a problem nowadays because files larger than 2 GiB are not uncommon. I plan to implement some sort of multivolume support in tarlz, but the way tarlz could split a tar file (putting at least one tar member in each lzip member) is not suitable for such file systems if large files are to be archived. Have you tried to create the multivolume archive using lzip? It has the inconvenient that it does not keep the corresponcence between tar members and lzip members. But it has the advantage that it produces valid complete lzip files of a guaranteed maximum size. The command could be something like this: (time tarlz --uncompressed ...) \ | (pv --average-rate --bytes --eta --size $SIZE) \ | (lzip -S 2GB -o /exfat-sd-card/volume_name -) http://www.nongnu.org/lzip/manual/lzip_manual.html#Invoking-lzip -S bytes --volume-size=bytes When compressing, and '-c' has not been also specified, split the compressed output into several volume files with names 'original_name1.lz', 'original_name2.lz', etc, and set the volume size limit to bytes. Input files are kept unchanged. Each volume is a complete, maybe multimember, lzip file. A small volume size may degrade compression ratio, so use it only when needed. Valid values range from 100 kB to 4 EiB. See also examples 10, 11, and 12 about how to create and restore multivolume compressed tar archives at http://www.nongnu.org/lzip/manual/lzip_manual.html#Examples Best regards, Antonio.
Re: tarlz --diff broken
Dear Devon, Devon Sean McCullough wrote: tarlz --create and tarlz --diff fail to work compatibly. Thank you very much for reporting this. This is not a bug, in the sense that I implemented --diff on purpose as it is now. But I agree with you that it would be more useful to process option --directory in --diff as in --create. I'll fix the behavior of --diff and release a new version as soon as possible. The reason why I implemented --diff as I did is that the documentation for GNU tar seems to imply that --directory is order sensitive only for --create: http://www.gnu.org/software/tar/manual/html_node/Option-Summary.html '--directory=dir' '-C dir' When this option is specified, tar will change its current directory to dir before performing any operations. When this option is used during archive creation, it is order sensitive. Best regards, Antonio.
Re: patch to lzip 1.23
Hello Klaus, Thank you for the patch. Klaus Holst Jacobsen wrote: I have a patch which will enable the usage of a custom compiler. You may already use a custom compiler with a command like this: ./configure CXX=your_compiler See the output of "./configure --help" and the following message: https://lists.nongnu.org/archive/html/lzip-bug/2020-03/msg5.html It will also enable the use of mingw compiler to compile for windows (and recognise the .exe extension for the output) I don't use Windows, but as far as I know, lzip compiles fine there without special handling of the .exe extension. Best regards, Antonio.
Re: File ends unexpectedly at pos XXX when compressing
Scott Mcdermott wrote: You are right, I did not think it were possible, but there were read errors on the NVME device in the kernel log, which I was able to reproduce with 'dd' skip to the LBA and issue reads. Interestingly, "tar | pbzip2" finished the whole thing without error, although repeated attempts with "tarlz" always produced errors. This might be because tarlz reads and compresses files in parallel (multithreaded) when creating a compressed archive, while tar reads files sequentially (one at a time). You can make tarlz read files sequentially by passing option -n0 in the command line: -n N --threads=N Set the number of (de)compression threads, overriding the system's default. Valid values range from 0 to "as many as your system can support". A value of 0 disables threads entirely. I have just added to my list of pending tasks to improve the diagnostic above to specify whether the cause is EOF or some error. Excellent, strerror() would be useful to print in latter case... I have already implemented the improved diagnostic. It will appear in the next version of tarlz. Best regards, Antonio.
Re: File ends unexpectedly at pos XXX when compressing
Hello Scott, Scott Mcdermott wrote: Hello, I was using tarlz 0.22 / liblz 1.13 like this: tarlz -cpf file.tlz dir Note that option '-p, --preserve-permissions' (don't subtract the umask on extraction) does not have effect on archive creation. to compress a 37GB directory 'dir' containing approximately 12,000 files. The output grew to 15GB and then I received this report: File 'dir/foo' ends unexpectedly at pos 106496 with exit code 1. The file it errored on was an .exo cache file from the YouTube app (it was an sdcard backup of a phone), with size 1265209. It worked fine when I gave it just this file. The file was not sparse when checked with 'ls -s'. Do you mean the command 'tarlz -cpf file.tlz dir/foo' produced an archive with a 'dir/foo' member of the right size? Or did you use some other command? I looked at source code and this appears to be from minilzip.c in do_decompress(), so I am puzzled why this routine is used with '-c' option. The diagnostic above does not come from minilzip.c, but from line 476 in function 'cworker' in create_lz.cc. It happens when the file being read and archived ends before its expected size. The reading is performad by the function 'readblock', which reads in a loop until the file ends or until something fails. So it is probable that the file 'dir/foo' has indeed some issue. I am also curious if there are any known bugs which might cause this error message to occur. The files were not in use at time of 'tarlz', as this was an old backup tree of a long-dead phone sdcard (stored on an unrelated ext4 fs). This error message may happen when the file reduces its size while it is being archived, or when the file size does not match the real file contents because of a file system error, for example. I have just added to my list of pending tasks to improve the diagnostic above to specify whether the cause is EOF or some error. Best regards, Antonio.
[Lzip-bug] Lziprecover 1.24-pre1 released
Lziprecover 1.24-pre1 is ready for testing here http://download.savannah.gnu.org/releases/lzip/lziprecover/lziprecover-1.24-pre1.tar.lz http://download.savannah.gnu.org/releases/lzip/lziprecover/lziprecover-1.24-pre1.tar.gz The sha256sums are: 5d3f4dfa3a36b531276d5535caa5b766ca7c2702980c22afa511d6d06237fa3c lziprecover-1.24-pre1.tar.lz d3740f0503a7fb7da83da26400cfe7c4471f66eea4bcdb7764af5b78f4cb106a lziprecover-1.24-pre1.tar.gz Please, test it and report any bugs you find. Lziprecover is a data recovery tool and decompressor for files in the lzip compressed data format (.lz). Lziprecover is able to repair slightly damaged files (up to one single-byte error per member), produce a correct file by merging the good parts of two or more damaged copies, reproduce a missing (zeroed) sector using a reference file, extract data from damaged files, decompress files, and test integrity of files. Lziprecover can remove the damaged members from multimember files, for example multimember tar.lz archives. Lziprecover provides random access to the data in multimember files; it only decompresses the members containing the desired data. Lziprecover facilitates the management of metadata stored as trailing data in lzip files. A nice feature of the lzip format is that a corrupt byte is easier to repair the nearer it is from the beginning of the file. Therefore, with the help of lziprecover, losing an entire archive just because of a corrupt byte near the beginning is a thing of the past. If the cause of file corruption is a damaged medium, the combination GNU ddrescue[1] + lziprecover is the recommended option for recovering data from damaged lzip files. The homepage is at http://www.nongnu.org/lzip/lziprecover.html Changes in this version: * The option '--empty-error', which forces exit status 2 if any empty member is found, has been added. * The option '--marking-error', which forces exit status 2 if the first LZMA byte is non-zero in any member, has been added. * The option '--clear-marking', which sets to zero the first LZMA byte of each member, has been added. * The keyword 'empty' is now recognized in the argument of --dump, --remove, and --strip. * The option '--repair' has been renamed to '--byte-repair'. * The option '--debug-repair' has been renamed to '--debug-byte-repair'. * Diagnostics caused by invalid arguments to command line options now show the argument and the name of the option. * The option '--no-verify' of unzcrash has been renamed to '--no-check'. [1] http://www.gnu.org/software/ddrescue/ddrescue.html Regards, Antonio Diaz, lziprecover author and maintainer. -- If you care about data safety and long-term archiving, please consider using lzip. See http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance and http://www.nongnu.org/lzip/safety_of_the_lzip_format.html Thanks.
Re: Section "2.10.4 The 'Block Check' field" in your paper: Xz format inadequate for long-term archiving
Dear Wolfgang, Thank you for your message. I'll try to clarify things and maybe improve the wording of Section 2.10.4. Wolfgang Liessmann wrote: in section "2.10.4 The 'Block Check' field" you present a formula Inaccuracy = (compressed_size * Pudc + CS_size) / (compressed_size + CS_size) which is not explained (derived in a way so one sees its necessity). That formula calculates inaccuracy by plain counting of cases. Let's see: Inaccuracy is defined in the Glossary (http://www.nongnu.org/lzip/xz_inadequate.html#glossary) as: Inaccuracy: Ratio of error detection mistakes. Is defined as (false_negatives + false_positives) / total_cases. For the single-byte error model (one corrupt byte per test), the inaccuracy for each compressed size and check sequence (CS) size is calculated by the following formula (all sizes in bytes): Inaccuracy = (compressed_size * Pudc + CS_size) / (compressed_size + CS_size) Where 'compressed_size * Pudc' are the false negatives (cases where the check sequence does not detect the corruption), 'CS_size' is the size of the check sequence, in bytes (cases where the corrupt byte belongs to the check sequence, not to the compressed payload), and 'compressed_size + CS_size' is the total size of the file (total number of cases). In particular, the statement It should be noted that SHA-256 provides worse accuracy than CRC64 for all possible block sizes. and the inaccuracy 3e-08 at 1e+09 bytes (= 1 GB) compressed size seems to contradict simple cryptographic standards at the first glance. This may be because "cryptographic standards" do not take into account mismatches caused by the corruption of the hash value itself. If we take Pudc (the probability of a single-byte error being undetected by the check sequence) as 2^-n (See [Koopman], p. 5), where 'n' is the size of the check sequence in bits, then the inaccuracy for the 3 types of check sequences considered can be calculated for a compressed size 'x' as: ( x * 2^-32 + 4 ) / ( x + 4 ) for CRC32 ( x * 2^-64 + 8 ) / ( x + 8 ) for CRC64 ( x * 2^-256 + 32 ) / ( x + 32 ) for SHA-256 The largest block size that a xz file can contain is 2^63 bytes. So the inaccuracies for the largest size are: ( 2^63 * 2^-32 + 4 ) / ( 2^63 + 4 ) ~= 2.3e-10 for CRC32 ( 2^63 * 2^-64 + 8 ) / ( 2^63 + 8 ) ~= 9.2e-19 for CRC64 ( 2^63 * 2^-256 + 32 ) / ( 2^63 + 32 ) ~= 3.5e-18 for SHA-256 When we apply the formula '(false_negatives + false_positives) / total_cases' to a compressed size of 1 GB we get the following inaccuracies: ( 1e9 * 2^-32 + 4 ) / ( 1e9 + 4 ) ~= 4e-9 for CRC32 ( 1e9 * 2^-64 + 8 ) / ( 1e9 + 8 ) ~= 8e-9 for CRC64 ( 1e9 * 2^-256 + 32 ) / ( 1e9 + 32 ) ~= 3.2e-8 for SHA-256 Which matches figure 3 in the article. The rule of thumb (the mathematical approximation) is that the security, measured in bits of security, is half of the hash length (if there is no attack on the algorithm, and there is no known effective attack on SHA-256 as of now), which means that the probability of a collision of two SHA-256 hashes is less than With respect to the check sequence, the inaccuracy described in the article has nothing to do with cryptography, collisions, security, or attacks. It just relates to the size of the check sequence; the larger the check sequence, the more probable that corruption affects it (causing a false positive). Just to illustrate: if you use a check sequence as large as the compressed payload, half of the bit flips occurring in the resulting file will affect the payload and the other half will affect the check sequence (on average). It is immaterial whether the check sequence is a CRC or a cryptographic hash. As a SHA-256 hash is 8 times larger than a 32-bit CRC, it is 8 times more probable that a random bit corruption affects it, making the decompressor believe that the payload is corrupt because it does not match the check sequence. Similarly, it is 4 times more probable that a random bit corruption affects a SHA-256 than a CRC64. Choosing a check sequence larger than needed is just a bad idea. While at the first glance lzip indeed seems to have a better design than xz, and the compression rate is better comparing the maximum compression levels, it is not clear to me why a secure cryptographic hash function should perform that badly. This is explained in [Koopman], p. 33: "there can be safety tradeoffs with the addition of an error-detection scheme. As with almost all fault tolerance mechanisms, there is a tradeoff between availability and integrity. That is, techniques that increase integrity tend to reduce availability and vice versa. Employing error detection by adding a check sequence to a dataword increases integrity, but decreases availability. The decrease in availability happens through false-positive detections. These failures preclude the use of some data that otherwise would not have been rejected had it not been for the addition of
Re: pkg-config file for lzlib
Hello Niclas, Niclas Rosenvik wrote: I like to be able to look up lzlib via pkg-config, I searched the mailing list and found that others want it too. Maybe pkg-config is useful for libraries with non-standard dependencies, but AFAICT pkg-config is 100% useless for lzlib. You may have noticed in my previous answers that I don't like the idea of pkg-config and that I am still waiting for someone to show me a real compilation problem solved by the use of pkg-config: http://lists.nongnu.org/archive/html/lzip-bug/2019-04/msg00015.html "If I have understood correctly how pkg-config works[1], the .pc file must be generated by the makefile at build time, and it contains fixed directory names that prevent the library to be found if it is moved after instalation." http://lists.nongnu.org/archive/html/lzip-bug/2022-06/msg00013.html "Before even start thinking about this I need an example of a failed build that is fixed by the inclusion of such a pc.in file (without requiring any kind of autotools support)." So here are patches that make this work. Thank you. Nice patch. It certainly "makes this work" in the sense that it creates a .pc file that can be used by pkg-config. But I still do not see that it solves any real problem. For example, your patch stores the .pc file in "pkgconfigdir='$(libdir)/pkgconfig'", which makes me wonder why pkg-config can find the .pc file but can't find the library just one directory above it (in libdir). Storing a database of .pc files recording the locations of libraries adds some problems, like finding the database itself, deciding what to do if there is more than one database (in /usr/lib/pkgconfig and /usr/local/lib/pkgconfig for example), or dealing with stale location data (library moved or reinstalled elsewhere). I have several versions of lzlib installed on my machine (in different directories) to test plzip and tarlz, and I can't see how a .pc file could help me find the one I want to use each time. Having to type ./configure lzlib_CFLAGS='-I ' lzlib_LIBS='-L -llz' instead of ./configure CPPFLAGS='-I ' LDFLAGS='-L ' is not an improvement. (In fact it is wrong, because belongs in CPPFLAGS, not in CFLAGS). BTW, lzlib provides macros and functions to check the version of the library both at compile time and run time which are more reliable than the configuration time test possible with pkg-config. (An application may be dynamically linked at run time with a different version of lzlib than the one it was compiled for, and this should not break the application as long as the library used provides the functionality required by the application). The install-pc target is not run by default, I leave it up to Antonio to decide if the pkg-config file should be installed automatically or if install-pc has to be run separately. Installing the .pc file separately would be a bad idea because if one forgets to do it the second time, the old .pc file may become out of sync with the new installed library. I think that adding a .pc file to lzlib would not solve any problems and may create problems where there are currently none. If lzlib is installed in a standard location, the compiler and the linker will find it. If lzlib is installed in a non-standard location (or moved after installation), pkg-config won't find it. Moreover, there are discrepancies[1] among pkg-config implementations. Which makes me think that the concept itself of how pkg-config should work is not (yet?) fully resolved. Maybe it is better to provide .pc files only for those libraries that can take advantage of the services (?) provided by pkg-config. In fact, after reading in the link below that "Current release versions of pkg-config, on the other hand, build a database of all known pkg-config files on the system before attempting to resolve dependencies", I'm moving all the .pc files where pkg-config won't find them. I find it unacceptably wasteful to read hundreds of files just to find out that you didn't need to read them in the first place. [1] See section "compatibility with pkg-config" at http://github.com/pkgconf/pkgconf#readme Best regards, Antonio.
Re: Wishlist: xattr support in tarlz
Daniel Baumann wrote: completely unrelated: the freeze for Debian 12 (bookworm) is coming up.. Thank you for the reminder. I planned to release zutils 1.12 by about today, but I got delayed. I expect to release it in a few days, much sooner than the 5th of February. Best regards, Antonio.
Re: Wishlist: xattr support in tarlz
Hi Daniel, Happy new year! :-) Daniel Baumann wrote: in Debian we've recieved a bug report that tarlz isn't supporting xattr like tar. It would be nice if tarlz could support both xattr and acls like tar does, to make it (more of) a drop-in replacement of tar. I have never planned to implement in tarlz support for the non-standardized can of worms that are xattr and acls. The initial set of patches[1] for GNU tar was pretty complicated and unportable, and it is still causing trouble to tar maintainers[2]. [1] https://lists.gnu.org/archive/html/bug-tar/2012-08/msg00012.html [2] https://lists.gnu.org/archive/html/bug-tar/2022-08/msg00010.html Moreover, I'm now too busy to even think about it. But thanks for reporting it. Best regards, Antonio.
Re: Missing includes for uint8_t type in lzlib.h
Hello Michał, Michał Górny wrote: The lzlib.h header installed by lzlib uses the uint8_t type, yet doesn't include . As a result, programs using it fail to compile unless they happen to include prior to , Thank you very much for reporting this. It is the responsibility of the program using lzlib to include some header that declares the type 'uint8_t'. 'lzlib.h' can be used by C and C++ programs, and therefore it must not impose a choice of header on the program. AFAIK, there are at least 4 headers to choose from: 'stdint.h', 'cstdint', 'inttypes.h', and 'cinttypes'. I'll document in the manual and in 'lzlib.h' itself the need of including one of those headers before including 'lzlib.h'. Best regards, Antonio.
Re: [Lzip-bug] Parameter is not bool (clzip)
Hi Adam. Adam Tuja wrote: As you mentioned C99, out of curiosity I tried to compile it with -std=c99 and the result was, to say the least, surprising. I got bunch of serious warnings and an error. If you choose a C standard, you also need to enable the POSIX stuff explicitly like this: '--std=c99 -D_XOPEN_SOURCE=500' Regards, Antonio.
Re: [Lzip-bug] Parameter is not bool (clzip)
Stefan A. Haubenthal wrote: vc +kick13 -Ivbcc:PosixLib/include -DPROGVERSION="1.13" -c -o main.o main.c if( !open_outstream( true, in_statsp ) ) error 39 in line 702 of "main.c": invalid types for assignment 1 error found! It seems that your report is about clzip. Note that 'in_statsp' is a pointer, and according to the C99 standard: 6.3.1.2 Boolean type 1 When any scalar value is converted to _Bool, the result is 0 if the value compares equal to 0; otherwise, the result is 1. and 6.2.5 Types [...] 21 Arithmetic types and pointer types are collectively called scalar types. Therefore, using 'in_statsp' as a bool argument is valid. Best regards, Antonio.
[Lzip-bug] Lzd 1.3 released
I am pleased to announce the release of lzd 1.3. Lzd is a simplified decompressor for the lzip format with an educational purpose. Studying its source code is a good first step to understand how lzip works. Lzd is written in C++. The source code of lzd is used in the lzip manual as a reference decompressor in the description of the lzip file format. Reading the lzip manual will help you understand the source code. Lzd is compliant with the lzip specification; it verifies the 3 integrity factors. The source code of lzd is also used as a reference in the description of the media type 'application/lzip'. See http://datatracker.ietf.org/doc/draft-diaz-lzip Comments about the internet draft above are solicited and should be addressed to the lzip's mailing list at lzip-bug@nongnu.org and/or the author. Thanks. The lzd homepage is at http://www.nongnu.org/lzip/lzd.html The sources can be downloaded from http://download.savannah.gnu.org/releases/lzip/lzd/ The sha256sums are: d64fe778b4ed5b797855c12a8d1960781565f9f00298dc0fe0fadc02a56faf83 lzd-1.3.tar.lz 5a52f450b9343508815f9e1fb35c14db37602ac5cd9f8e8ff6743903526d5f34 lzd-1.3.tar.gz Changes in version 1.3: * Lzd now discards the first byte of the LZMA stream explicity (instead of shifting 1 extra byte in the initialization of 'code') to make the intent clearer. Regards, Antonio Diaz, lzd author and maintainer. -- If you care about data safety and long-term archiving, please consider using lzip. See http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance and http://www.nongnu.org/lzip/safety_of_the_lzip_format.html Thanks.
[Lzip-bug] Tarlz 0.23 released
I am pleased to announce the release of tarlz 0.23. Tarlz is a massively parallel (multi-threaded) combined implementation of the tar archiver and the lzip compressor. Tarlz uses the compression library lzlib. Tarlz creates tar archives using a simplified and safer variant of the POSIX pax format compressed in lzip format, keeping the alignment between tar members and lzip members. The resulting multimember tar.lz archive is fully backward compatible with standard tar tools like GNU tar, which treat it like any other tar.lz archive. Tarlz can append files to the end of such compressed archives. Keeping the alignment between tar members and lzip members has two advantages. It adds an indexed lzip layer on top of the tar archive, making it possible to decode the archive safely in parallel. It also minimizes the amount of data lost in case of corruption. Compressing a tar archive with plzip may even double the amount of files lost for each lzip member damaged because it does not keep the members aligned. Tarlz can create tar archives with five levels of compression granularity: per file (--no-solid), per block (--bsolid, default), per directory (--dsolid), appendable solid (--asolid), and solid (--solid). It can also create uncompressed tar archives. Of course, compressing each file (or each directory) individually can't achieve a compression ratio as high as compressing solidly the whole tar archive, but it has the following advantages: * The resulting multimember tar.lz archive can be decompressed in parallel, multiplying the decompression speed. * New members can be appended to the archive (by removing the end-of-archive member), and unwanted members can be deleted from the archive. Just like an uncompressed tar archive. * It is a safe POSIX-style backup format. In case of corruption, tarlz can extract all the undamaged members from the tar.lz archive, skipping over the damaged members, just like the standard (uncompressed) tar. Moreover, the option '--keep-damaged' can be used to recover as much data as possible from each damaged member, and lziprecover can be used to recover some of the damaged members. * A multimember tar.lz archive is usually smaller than the corresponding solidly compressed tar.gz archive, except when individually compressing files smaller than about 32 KiB. Note that the POSIX pax format has a serious flaw. The metadata stored in pax extended records are not protected by any kind of check sequence. Because of this, tarlz protects the extended records with a Cyclic Redundancy Check (CRC) in a way compatible with standard tar tools. The homepage is at http://www.nongnu.org/lzip/tarlz.html An online manual for tarlz can be found at http://www.nongnu.org/lzip/manual/tarlz_manual.html The sources can be downloaded from http://download.savannah.gnu.org/releases/lzip/tarlz/ The sha256sum is: 3cefb4f889da25094f593b43a91fd3aaba33a02053a51fb092e9b5e8adb660a3 tarlz-0.23.tar.lz Changes in version 0.23: * Tarlz now can create and decode the extended records 'atime' and 'mtime', allowing times beyond the ustar range (before 1970-01-01 00:00:00 UTC or after 2242-03-16 12:56:31 UTC). * Tarlz now can create and decode the extended records 'uid' and 'gid', allowing user and group IDs beyond the ustar limit of 2_097_151. * The new option '--ignore-overflow', which makes '-d, --diff' ignore differences in mtime caused by overflow on 32-bit systems, has been added. * Tarlz now refuses to read archive data from a terminal or write archive data to a terminal. (Reported by DustDFG). * In the date format of option '--mtime' the time of day 'HH:MM:SS' is now optional and defaults to '00:00:00'. Both space and 'T' are now accepted as separator between date and time. * Diagnostics caused by invalid arguments to command line options now show the argument and the name of the option. * Tarlz now diagnoses separately the failure to create an intermediate directory during extraction. * Failure to extract a member due to environmental problems is no longer fatal in serial extraction. (It was already non-fatal in parallel extraction). * The diagnostics emitted by the parallel decoder should now be identical to the corresponding diagnostics of the serial decoder. * Column alignment has been improved in listings by printing "user/group size" in a field of minimum width 19 with at least 8 characters for size. * The diagnostic shown when the filesystem reports a wrong st_size for a symbolic link has been improved. (Reported by Jason Lenz). * The diagnostic "File is the archive" has been changed to "Archive can't contain itself" following a similar change made by Paul Eggert to GNU tar. * The warning "Removing leading '/' from member names." is now not shown when compressing nor if the member causing it is excluded with '--exclude'. * The texinfo
[Lzip-bug] Tarlz does not need --libdir
DustDFG wrote: Maybe it is a noob question but I want to know an answer. Why does tarlz 'configure' file not use standard command line options like --libdir --includedir --syslibdir? Because tarlz does not need them. (Tarlz does not install libraries). See http://www.gnu.org/prep/standards/html_node/Directory-Variables.html "These variables specify the directory for installing certain specific types of files, if your program has them. Every GNU package should have Info files, so every program needs 'infodir', but not all need 'libdir' or 'lispdir'." Antonio.
Re: Bug with infinite work when has -t without -f
DustDFG wrote: I found that tarlz is starting infinite work when I run command with -t flag but without -f flag. The GNU tar prints an error message in this case If no archive is specified (with -f), both GNU tar and tarlz try to read it from standard input. This is not an infinite loop, just waiting for input. $ tarlz -t tdelibs-trinity-14.0.11.tar.lz ^C $ tar -t tdelibs-trinity-14.0.11.tar.lz ^C $ If would have been easier if you had specified the exact commands used and the error message reported by GNU tar, because as you can see above I can't reproduce the error. Antonio.
Re: Typo in lzlib manual
Hi Firas, Firas Khalil Khana wrote: The word "dinamically" should be "dynamically". Thank you very much for reporting this. I have just fixed it in the development version and will fix the online version soon. Best regards, Antonio.
Re: [Lzip-bug] Missing pkg-config support?
Hello Avinash, Avinash Sonawane wrote: Here's a 10+ lines long liblzma.pc.in: https://git.tukaani.org/?p=xz.git;a=blob;f=src/liblzma/liblzma.pc.in;h=9fa489115a0a23c6162fa7d8bf1f634210071036;hb=HEAD to get inspiration from. Here's bzip2.pc.in (10+ lines): https://gitlab.com/bzip2/bzip2/-/blob/master/bzip2.pc.in Thank you, but those pc.in files are 100% useless without configuration support and not much more useful with it. Note from the bzip2 link above: "This is a patch from Stanislav Bravec that adds an autotools build environment to bzip2. This lets libtool handle the construction of the shared library. The patch also adds a bzip2.pc.in template to build data for the pkg-config tool." Before even start thinking about this I need an example of a failed build that is fixed by the inclusion of such a pc.in file (without requiring any kind of autotools support). Best regards, Antonio.
Re: Fast lzma radix matchfinder
Adam Tuja wrote: The comparison here would be the same as with lzma, that is slightly faster. [1] Bigger advantage, beside compression speed, is revealed in memory consumption for multiple threads - it's halved for single thread but 1/4 for 2 threads and 1/8 for 4 threads [1][2]. Very interesting. Thank you for bringing this to my attention. I expect to look at it in depth when I find the time, but I guess it may be difficult (or impossible) to integrate it meaningfully into plzip because it seems very different from what plzip does. See for example https://github.com/conor42/fast-lzma2#readme "Speed gains depend on the nature of the source data." "The largest caveat is that the match-finder is a block algorithm, and to achieve about the same ratio as 7-Zip requires double the dictionary size, which raises the decompression memory usage." it is not worth the trouble of breaking lzip's reproducibility Don't know what you mean by "reproducibility" Lzip is more than a compressor. It is a set of tools designed around a format tuned for long-term archiving. It is important that the output of lzip does not change frequently between versions because such changes may hinder some kinds of data recovery. See for example http://www.nongnu.org/lzip/manual/lziprecover_manual.html#Reproducing-one-sector We need to think about the consequences of the consequences (sic) of any change to the interface or to the algorithm. but I didn't mean to replace current encoder/s, rather complement them. If it was used it could be different compression levels, like 11-19. Increasing the number of levels also hinders data recovery. Moreover, options like -11 or -19 are not compatible with POSIX or GNU standards. See http://www.nongnu.org/arg-parser/manual/arg_parser_manual.html#Argument-syntax Also, having a level 11 that compresses less than level 9 is confusing to users. So these may also be difficult to integrate meaningfully into lzip. Best regards, Antonio.
Re: Docs still unclear on integrity
Hi Askar, Askar Safin wrote: Hi. I installed lzip 1.23-3 to my debian from debian repo. Is it true that default decompression mode (i. e. lzip -d file.lz) verifies archive? "man lzip" doesn't say anything about this. Thanks for reporting this. I have just extended the description of '-d, --decompress' in the man page to "decompress, test compressed file integrity" and have made a similar change to the info manual. Also, is it true that compressed file truncation is absolutely always detected? (Multiply bit flips are hard to detect with absolute guarantee, but I think it is quite possible to always detect file truncation.) Is file truncation detected in default mode (lzip -d file.lz)? "man lzip" is silent on this File truncation is always detected if it happens inside a member, but it can't be detected if it happens just after the end of a member. This is a well-known limitation[1] of any tool that allows file concatenation (gzip, bzip2, lzip, and even tar): $ echo foo | lzip > foo.lz $ echo bar | lzip > bar.lz $ tar -c foo.lz bar.lz | dd bs=512 count=4 2>/dev/null | tar -t ; echo $? foo.lz bar.lz 0 If you want to be sure that file truncation is always detected, you may either use single-member files, or append some trailing data[2] to the file and then verify it at the receiving side. For example you may append the size of the whole concatenated file and verify it against the compressed size shown by --list: $ lziprecover --strip=tdata foo.lz bar.lz > foobar.lz $ wc -c < foobar.lz >> foobar.lz $ lzip --verbose --list foobar.lz dict memb trail uncompressed compressed saved name 4 KiB 2 3 8 80 -900.00% foobar.lz $ lziprecover --dump=tdata foobar.lz 80 Best regards, Antonio. [1] http://www.nongnu.org/lzip/manual/lziprecover_manual.html#Unzcrash If the decompressor returns with zero status, unzcrash compares the output of the decompressor for the original and corrupt files. If the outputs differ, it means that the decompressor returned a false negative; it failed to recognize the corruption and produced garbage output. The only exception is when a multimember file is truncated just after the last byte of a member, producing a shorter but valid compressed file. [2] http://www.nongnu.org/lzip/manual/lziprecover_manual.html#Trailing-data
Re: Fast lzma radix matchfinder
Hello Adam, Adam Tuja wrote: I stumbled upon this fast lzma radix matchfinder that turned out to be twice as fast, using half of memory single threaded, and even less multi threaded, with only very slighlty worse compression ratio. Thank you. Twice as fast as what? Have you tried it against 'lzip -0'? What is the difference with the matchfinders that have been tried and discarded in lzip 1.11? http://fresh-center.net/linux/lzip/ChangeLog * Matchfinder types HC4 (4 bytes hash-chain) and HT4 (4 bytes hash-table) have been tested and found no better than the current BT4. What do you think of it? Waiting for the comparison with 'lzip -0', my first impression is that it is not worth the trouble of breaking lzip's reproducibility. Best regards, Antonio.
[Lzip-bug] Lzip patch for kmod-29 released
I am pleased to announce the release of the lzip patch for kmod-29. This patch adds lzip decompression support to kmod. The patched kmod requires lzlib to compile: http://www.nongnu.org/lzip/lzlib.html You may also need to pass the following options to configure: ./configure --with-lzlib lzlib_LIBS=-llz lzlib_CFLAGS=-llz The patch can be downloaded from http://download.savannah.gnu.org/releases/lzip/kernel/kmod-29_lzip-0.diff.lz The sha256sum is: 44c27051d0e1bfb60a0c80867c02dbb8ac8bb815cbfda30d782abd7334c7f9e5 kmod-29_lzip-0.diff.lz Please, test it and report any bugs you find. Regards, Antonio Diaz, lzip author and maintainer. -- If you care about data safety and long-term archiving, please consider using lzip. See http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance and http://www.nongnu.org/lzip/safety_of_the_lzip_format.html Thanks.
[Lzip-bug] Lzip patch for linux 5.16.11 released
I am pleased to announce the release of the lzip patch for linux 5.16.11. Since 2016 I maintain a patch adding lzip decompression support to the linux kernel. This includes support for lzip-compressed kernel image, initramfs, initrd, modules, and squashfs filesystems. The patch implements a linux module named lzip_decompress. Decompression time is usually a small fraction of the total boot time. Therefore it is not worth using a faster but weaker algorithm just to save a fraction of a second from a total boot time of, say, 20 seconds. Lzip related components in the kernel = The lzip_decompress module in lib/lzip_decompress.c provides a versatile lzip decompression function able to do buffer to buffer decompression or stream decompression with fill and flush callback functions. The usage of the function is documented in include/linux/lzip.h. For decompressing the kernel image, initramfs, and initrd, there is a wrapper function in lib/decompress_lunzip.c providing the same common interface as the other decompress_*.c files, which is defined in include/linux/decompress/generic.h. For kernel makefiles, two commands are provided in scripts/Makefile.lib for use with $(call if_changed). The kernel image must be compressed with $(call if_changed,klzip) which will append a four-byte trailer containing the size of the uncompressed data, which is needed by the boot code. Other things should be compressed with $(call if_changed,lzip). Testing === Xlunzip is a test tool for the lzip_decompress module. It is similar to lunzip, but it uses the lzip_decompress module as a backend. The xlunzip tarball contains a copy of the lzip_decompress module and can be compiled and tested without downloading or applying the patch to the kernel. The homepage of xlunzip is at http://www.nongnu.org/lzip/xlunzip.html The patch can be downloaded from http://download.savannah.gnu.org/releases/lzip/kernel/linux-5.16.11_lzip-0.diff.lz The sha256sum is: d86f22ff643ef66b4bdcca25abea254e9f0c35998d54fbd068bd2c3dd96f27de linux-5.16.11_lzip-0.diff.lz Please, test it and report any bugs you find. Regards, Antonio Diaz, lzip author and maintainer. -- If you care about data safety and long-term archiving, please consider using lzip. See http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance and http://www.nongnu.org/lzip/safety_of_the_lzip_format.html Thanks.
Re: Time/Speed of de/compression summary
Hi Adam, Antonio Diaz Diaz wrote: Thank you for the patch, but it introduces dependencies on things like clock_t or CLOCKS_PER_SEC which I'm not sure how portable and problem-free they are, As I suspected, they are not problem-free: http://pubs.opengroup.org/onlinepubs/9699919799/functions/clock.html "In programming environments where clock_t is a 32-bit integer type and CLOCKS_PER_SEC is one million, clock() will start failing in less than 36 minutes of processor time for signed clock_t, or 72 minutes for unsigned clock_t." As measuring speed is not the job of lzip, I prefer to leave lzip as it is now. Best regards, Antonio.
Re: Time/Speed of de/compression summary
Hi Adam, Adam Tuja wrote: I discovered that verbose summary is more verbose than I thought but I felt like it lacks something. I added simple time and dotted of de/compression to verbose -vv mode. It does not require additional libraries and adds about 300B to the code. Thank you for the patch, but it introduces dependencies on things like clock_t ot CLOCKS_PER_SEC which I'm not sure how portable and problem-free they are, and just now I have not the time to assess such potential problems for a minor detail like this. I have put it in my list of things to investigate later. Best regards, Antonio.
Re: tarlz: error archiving broken symbolic link
Jason Lenz wrote: The symlink size of zero was indeed the issue as to why tarlz would not work with the Restic fuse filesystem. I worked with someone from the Restic team to update their fuse filesystem implementation to comply with POSIX symlink sizes. After making the updates tarlz worked great. More info in the link below: https://github.com/restic/restic/issues/3667 WOW! Thank you very much. This indeed fixes the problem much better than working around it in tarlz would have. Best regards, Antonio.
Re: CRC slicing-by-8
p.z.l wrote: Have you got any idea why this: c = crc32[(c ^ buffer[i]) & 0xFF] ^ ( c >> 8 ); /// lzip is faster than this: c = (c >> 8) ^ crc32[(c & 0xFF) ^ buffer[i]]; /// sbrumme crc May be because of the different ordering of the operations. You may try to transform one of the lines above into the other step by step, and test the speed of the intermediate stages.
Re: tarlz: error archiving broken symbolic link
Jason Lenz wrote: Output from ls -l on fuse: lrwxrwxrwx 1 lenzj users 0 Nov 11 10:59 broken_symlink_filename -> .. Output from ls -l on ext4: lrwxrwxrwx 1 lenzj users 120 Nov 11 10:59 broken_symlink_filename -> .. So in other words, the fuse filesystem is showing the size of all the symlink files as zero vs. a size of something greater than zero on the ext4 system. Thank you for investigating this. It looks like a bug in fuse which reports 0 in st_size for broken symlinks (at least). This does not meet POSIX's description of 'struct stat': https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_stat.h.html off_t st_size For regular files, the file size in bytes. For symbolic links, the length in bytes of the pathname contained in the symbolic link. and GNU tar says: http://fresh-center.net/linux/tar/gnu/areadlink-with-size.c 55 /* Some buggy file systems report garbage in st_size. Defend 56 against them by ignoring outlandish st_size values in the initial 57 memory allocation. */ and then enters an horribly complicated loop malloc'ing and realloc'ing a buffer until the output of readlink fits in it. I really do not want to do such things in tarlz. GNU tar aims at working everywhere (and supporting all tar formats), but I just want tarlz to work correctly on well-behaved systems. I think I'll just make tarlz emit a diagnostic when it finds a symlink with a st_size of 0. This seems like something the restic development team probably needs to fix on their fuse system implementation, but admittedly I'm not an expert on how fuse systems work. I will reach out to them to see if indeed this is something they can/should fix. Thank you. Fixing their fuse filesystem implementation is the best solution. It would spare other tools from working around the bug, and would even make GNU tar more efficient at reading symlinks (no need to loop). Best regards, Antonio.
Re: CRC slicing-by-8
Hello Adam, Sorry for the late answer. I have been busy. Adam Tuja wrote: Reading discussion about polynomial selection (https://lists.gnu.org/archive/html/lzip-bug/2017-05/msg3.html) I wondered whether better/faster CRC would make a difference in de/compression speed. Thanks for the suggestion, but I have already tried faster ways of calculating the CRC and I do not think that the (small) increase in speed is worth the increase of portability problems it causes. In line 175 of lzip.h you can find a comment describing my current opinion on the matter: // about as fast as it is possible without messing with endianness I also tested lzlib with lzbench (https://github.com/inikep/lzbench) but difference was much smaller. I wonder whether I did something wrong. Probably not. I have also tried, and most changes that make lzip faster don't work as well (or at all) in lzlib. Best regards, Antonio.
Re: tarlz: error archiving broken symbolic link
Hello Jason, Jason Lenz wrote: $ tarlz -cf test.tar.lz /path/to/files ... tarlz: /path/to/files/broken-symlink: Error reading link I guess there is some difference between your system and mine, because I can't reproduce the error (not even by writing the archive to test.tar.lz): $ md dir $ touch dir/a dir/b $ ln -s foo dir/broken-symlink $ ls -go dir total 0 -rw-r--r-- 1 0 Mar 7 16:35 a -rw-r--r-- 1 0 Mar 7 16:35 b lrwxrwxrwx 1 3 Mar 7 16:36 broken-symlink -> foo $ tarlz -c dir | tarlz -tv ; echo $? drwxr-xr-x antonio/users0 2022-03-07 16:36 dir -rw-r--r-- antonio/users0 2022-03-07 16:35 dir/a -rw-r--r-- antonio/users0 2022-03-07 16:35 dir/b lrwxrwxrwx antonio/users0 2022-03-07 16:36 dir/broken-symlink -> foo 0 When replacing tarlz with standard GNU tar version 1.34 and the exact same command line, it would successfully create the archive containing the broken symlink files. Thanks for the hint. I think I may have found the cause of the problem. In line 443 of create.cc tarlz calls readlink passing the size of the buffer and expecting as return value the size of the contents of the symbolic link: len = sz = readlink( filename, (char *)header + linkname_o, linkname_l ); But according to POSIX[1]: "If the number of bytes in the symbolic link is less than bufsize, the contents of the remainder of buf are unspecified." and "Upon successful completion, these functions shall return the count of bytes placed in the buffer." [1] https://pubs.opengroup.org/onlinepubs/9699919799/functions/readlink.html So readlink may be returning more bytes than it should if the link is shorter than the buffer. One possible fix would be to change line 443 to ask just for the bytes required instead of passing the full size of the buffer: len = sz = readlink( filename, (char *)header + linkname_o, st.st_size ); I have tried this change and it works. Please, test the change and report back. If it also works for you, I'll release a corrected version. P.S.: I really appreciate your efforts on lzip and associated utilities. Thanks. :-) Best regards, Antonio.
[Lzip-bug] Lzip 1.23 released
I am pleased to announce the release of lzip 1.23. Lzip is a lossless data compressor with a user interface similar to the one of gzip or bzip2. Lzip uses a simplified form of the 'Lempel-Ziv-Markov chain-Algorithm' (LZMA) stream format and provides a 3 factor integrity checking to maximize interoperability and optimize safety. Lzip can compress about as fast as gzip (lzip -0) or compress most files more than bzip2 (lzip -9). Decompression speed is intermediate between gzip and bzip2. Lzip is better than gzip and bzip2 from a data recovery perspective. Lzip has been designed, written, and tested with great care to replace gzip and bzip2 as the standard general-purpose compressed format for unix-like systems. The homepage is at http://www.nongnu.org/lzip/lzip.html The sources can be downloaded from http://download.savannah.gnu.org/releases/lzip/ The sha256sums are: ea02b9e6fb25d0e1a1d3ef4fdeb6ab26b931f1ece66fa1cee2983c95215e930a lzip-1.23.tar.lz 4792c047ddf15ef29d55ba8e68a1a21e0cb7692d87ecdf7204419864582f280d lzip-1.23.tar.gz This release is also GPG signed. You can download the signature by appending '.sig' to the URL. If the 'gpg --verify' command fails because you don't have the required public key, then run this command to import it: gpg --keyserver keys.gnupg.net --recv-keys 8FE99503132D7742 Key fingerprint = 1D41 C14B 272A 2219 A739 FA4F 8FE9 9503 132D 7742 Changes in version 1.23: * Decompression time has been reduced by 5-12% depending on the file. * In case of error in a numerical argument to a command line option, lzip now shows the name of the option and the range of valid values. * Several descriptions have been improved in manual, '--help', and man page. * The texinfo category of the manual has been changed from 'Data Compression' to 'Compression' to match that of gzip. (Reported by Alfred M. Szmidt). Please send bug reports and suggestions to lzip-bug@nongnu.org Regards, Antonio Diaz, lzip author and maintainer. -- If you care about data safety and long-term archiving, please consider using lzip. See http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance and http://www.nongnu.org/lzip/safety_of_the_lzip_format.html Thanks.
[Lzip-bug] Clzip 1.13 released
I am pleased to announce the release of clzip 1.13. Clzip is a C language version of lzip, fully compatible with lzip 1.4 or newer. As clzip is written in C, it may be easier to integrate in applications like package managers, embedded devices, or systems lacking a C++ compiler. Lzip is a lossless data compressor with a user interface similar to the one of gzip or bzip2. Lzip uses a simplified form of the 'Lempel-Ziv-Markov chain-Algorithm' (LZMA) stream format and provides a 3 factor integrity checking to maximize interoperability and optimize safety. Lzip can compress about as fast as gzip (lzip -0) or compress most files more than bzip2 (lzip -9). Decompression speed is intermediate between gzip and bzip2. Lzip is better than gzip and bzip2 from a data recovery perspective. Lzip has been designed, written, and tested with great care to replace gzip and bzip2 as the standard general-purpose compressed format for unix-like systems. The homepage is at http://www.nongnu.org/lzip/clzip.html The sources can be downloaded from http://download.savannah.gnu.org/releases/lzip/clzip/ The sha256sums are: 5c2ed42a00c6eb7e27276a8e07e0f319fea297ec38321421a7b582d58cd3ca7a clzip-1.13.tar.lz 7ac9fbf5036bf50fb0b6a20e84d2293cb0d24d4044eaf33cbe9760bb9e7fea7a clzip-1.13.tar.gz This release is also GPG signed. You can download the signature by appending '.sig' to the URL. If the 'gpg --verify' command fails because you don't have the required public key, then run this command to import it: gpg --keyserver keys.gnupg.net --recv-keys 8FE99503132D7742 Key fingerprint = 1D41 C14B 272A 2219 A739 FA4F 8FE9 9503 132D 7742 Changes in version 1.13: * Decompression time has been reduced by 5-12% depending on the file. * In case of error in a numerical argument to a command line option, clzip now shows the name of the option and the range of valid values. * Several descriptions have been improved in manual, '--help', and man page. * The texinfo category of the manual has been changed from 'Data Compression' to 'Compression' to match that of gzip. (Reported by Alfred M. Szmidt). Please send bug reports and suggestions to lzip-bug@nongnu.org Regards, Antonio Diaz, clzip author and maintainer. -- If you care about data safety and long-term archiving, please consider using lzip. See http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance and http://www.nongnu.org/lzip/safety_of_the_lzip_format.html Thanks.
[Lzip-bug] Plzip 1.10 released
I am pleased to announce the release of plzip 1.10. Plzip is a massively parallel (multi-threaded) implementation of lzip, fully compatible with lzip 1.4 or newer. Plzip uses the compression library lzlib. Lzip is a lossless data compressor with a user interface similar to the one of gzip or bzip2. Lzip uses a simplified form of the 'Lempel-Ziv-Markov chain-Algorithm' (LZMA) stream format and provides a 3 factor integrity checking to maximize interoperability and optimize safety. Lzip can compress about as fast as gzip (lzip -0) or compress most files more than bzip2 (lzip -9). Decompression speed is intermediate between gzip and bzip2. Lzip is better than gzip and bzip2 from a data recovery perspective. Lzip has been designed, written, and tested with great care to replace gzip and bzip2 as the standard general-purpose compressed format for unix-like systems. Plzip can compress/decompress large files on multiprocessor machines much faster than lzip, at the cost of a slightly reduced compression ratio (0.4 to 2 percent larger compressed files). Note that the number of usable threads is limited by file size; on files larger than a few GB plzip can use hundreds of processors, but on files of only a few MB plzip is no faster than lzip. The homepage is at http://www.nongnu.org/lzip/plzip.html The sources can be downloaded from http://download.savannah.gnu.org/releases/lzip/plzip/ The sha256sums are: 62f16a67be0dabf0da7fd1cb7889fe5bfae3140cea6cafa1c39e7e35a5b3c661 plzip-1.10.tar.lz 43faa58265d8b89ad75bd4ed11d347ef10065a8070748bc1ed0e06f191458098 plzip-1.10.tar.gz This release is also GPG signed. You can download the signature by appending '.sig' to the URL. If the 'gpg --verify' command fails because you don't have the required public key, then run this command to import it: gpg --keyserver keys.gnupg.net --recv-keys 8FE99503132D7742 Key fingerprint = 1D41 C14B 272A 2219 A739 FA4F 8FE9 9503 132D 7742 Changes in version 1.10: * In case of error in a numerical argument to a command line option, plzip now shows the name of the option and the range of valid values. * '--check-lib' now checks that LZ_API_VERSION and LZ_version_string match. * The variable LIBS can now be set from configure. * Several descriptions have been improved in manual, '--help', and man page. * The texinfo category of the manual has been changed from 'Data Compression' to 'Compression' to match that of gzip. (Reported by Alfred M. Szmidt). Please send bug reports and suggestions to lzip-bug@nongnu.org Regards, Antonio Diaz, plzip co-author and maintainer. -- If you care about data safety and long-term archiving, please consider using lzip. See http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance and http://www.nongnu.org/lzip/safety_of_the_lzip_format.html Thanks.
[Lzip-bug] Lzlib 1.13 released
I am pleased to announce the release of lzlib 1.13. Lzlib is a compression library providing in-memory LZMA compression and decompression functions, including integrity checking of the decompressed data. The compressed data format used by lzlib is the lzip format. Lzlib is written in C. Lzlib is free software distributed under a 2-clause BSD license. Compression/decompression is done by repeatedly calling a couple of read/write functions until all the data have been processed by the library. This interface is safer and less error prone than the traditional zlib interface. The homepage is at http://www.nongnu.org/lzip/lzlib.html The sources can be downloaded from http://download.savannah.gnu.org/releases/lzip/lzlib/ The sha256sums are: dde6bd5b32535f17b28c9ac24b66607e0250506ac1432a4112ca3c73f5d662c3 lzlib-1.13.tar.lz a1ab58f3148ba4b2674e938438166042137a9275bed747306641acfddc9ffb80 lzlib-1.13.tar.gz This release is also GPG signed. You can download the signature by appending '.sig' to the URL. If the 'gpg --verify' command fails because you don't have the required public key, then run this command to import it: gpg --keyserver keys.gnupg.net --recv-keys 8FE99503132D7742 Key fingerprint = 1D41 C14B 272A 2219 A739 FA4F 8FE9 9503 132D 7742 Changes in version 1.13: * The variables AR and ARFLAGS can now be set from configure. (Before you needed to run 'make AR='. (Reported by Hoël Bézier). * In case of error in a numerical argument to a command line option, minilzip now shows the name of the option and the range of valid values. * 'minilzip --check-lib' now checks that LZ_API_VERSION and LZ_version_string match. * Several descriptions have been improved in manual, '--help', and man page. * The texinfo category of the manual has been changed from 'Data Compression' to 'Compression' to match that of gzip. (Reported by Alfred M. Szmidt). Please send bug reports and suggestions to lzip-bug@nongnu.org Regards, Antonio Diaz, lzlib author and maintainer. -- If you care about data safety and long-term archiving, please consider using lzip. See http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance and http://www.nongnu.org/lzip/safety_of_the_lzip_format.html Thanks.
Re: ChangeLog encoding; testsuite file/s
Adam Tuja wrote: Last thing, for the record, if someone needed it. CRLF is actually simple and reliable to generate - I use it for years now: Making any modification to a file in a portable way (not only among posix systems) is not reliable. Adding CR to some lines of a file, but not to others, even less so. "It works for me" is not enough. You always can find a system where it doesn't work as expected. It requires `awk` which as `printf` are part of POSIX.1-2008. Depending on any external tool to generate a test file instead of shipping the file in the tarball can't be guaranteed to work. Lzip is not only for posix systems, and the external tool may have bugs. See for example this post about bugs in mawk: http://mm.icann.org/pipermail/tz/2022-January/031164.html Also, shipping the file in the tarball is future-proof. Regards, Antonio.
[Lzip-bug] Lunzip 1.13 released
I am pleased to announce the release of lunzip 1.13. Lunzip is a decompressor for the lzip format written in C. Its small size makes it well suited for embedded devices or software installers that need to decompress files but don't need compression capabilities. Lunzip is fully compatible with lzip 1.4 or newer and is able to decompress any lzip file using as little memory as 50 kB. The lzip file format is designed for data sharing and long-term archiving, taking into account both data integrity and decoder availability. The homepage is at http://www.nongnu.org/lzip/lunzip.html The sources can be downloaded from http://download.savannah.gnu.org/releases/lzip/lunzip/ The sha256sums are: be12c349239b8c8a879c017ba6d28e23dd3f0287bf7cea1288b6e285a0e3ce4c lunzip-1.13.tar.lz 3c7d8320b947d2eb3c6081caf9b6c91b12debecb089ee544407cd14c8e517894 lunzip-1.13.tar.gz This release is also GPG signed. You can download the signature by appending '.sig' to the URL. If the 'gpg --verify' command fails because you don't have the required public key, then run this command to import it: gpg --keyserver keys.gnupg.net --recv-keys 8FE99503132D7742 Key fingerprint = 1D41 C14B 272A 2219 A739 FA4F 8FE9 9503 132D 7742 Changes in version 1.13: * Decompression time has been reduced by 5-12% depending on the file. * In case of error in a numerical argument to a command line option, lunzip now shows the name of the option and the range of valid values. Please send bug reports and suggestions to lzip-bug@nongnu.org Regards, Antonio Diaz, lunzip author and maintainer. -- If you care about data safety and long-term archiving, please consider using lzip. See http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance and http://www.nongnu.org/lzip/safety_of_the_lzip_format.html Thanks.
[Lzip-bug] Lziprecover 1.23 released
I am pleased to announce the release of lziprecover 1.23. Lziprecover is a data recovery tool and decompressor for files in the lzip compressed data format (.lz). Lziprecover is able to repair slightly damaged files (up to one single-byte error per member), produce a correct file by merging the good parts of two or more damaged copies, reproduce a missing (zeroed) sector using a reference file, extract data from damaged files, decompress files, and test integrity of files. A nice feature of the lzip format is that a corrupt byte is easier to repair the nearer it is from the beginning of the file. Therefore, with the help of lziprecover, losing an entire archive just because of a corrupt byte near the beginning is a thing of the past. The homepage is at http://www.nongnu.org/lzip/lziprecover.html The sources can be downloaded from http://download.savannah.gnu.org/releases/lzip/lziprecover/ The sha256sums are: 9a41c9823670541573b160945c23783b644e84eb940c2c803b6b8d11b2c3d208 lziprecover-1.23.tar.lz f29804177f06dd51c3fa52615ae198a76002b6ce8d961c3575546c3740f4b572 lziprecover-1.23.tar.gz This release is also GPG signed. You can download the signature by appending '.sig' to the URL. If the 'gpg --verify' command fails because you don't have the required public key, then run this command to import it: gpg --keyserver keys.gnupg.net --recv-keys 8FE99503132D7742 Key fingerprint = 1D41 C14B 272A 2219 A739 FA4F 8FE9 9503 132D 7742 Changes in version 1.23: * Decompression time has been reduced by 5-12% depending on the file. * In case of error in a numerical argument to a command line option, lziprecover now shows the name of the option and the range of valid values. * Options '--dump' and '--strip' now refuse to write compressed data to a terminal except when dumping trailing data with '--dump=tdata'. * The option '-U, --unzcrash' now requires an argument: '1' to test 1-bit errors, or 'B' to test zeroed blocks. * The memory tester now allocates the dictionary once per member instead of doing it for each test. This makes '-U, --unzcrash' about two times faster on my machine on files with an uncompressed size larger than about 30 MB. * '-W, --debug-decompress' now continues decompressing the members following the damaged member if it has been fully decompressed (just failed with a CRC mismatch). * The tool unzcrash now uses execvp instead of popen to avoid invoking /bin/sh and run faster. It also prints byte or block position in messages. * Several descriptions have been improved in manual, '--help', and man page. * The texinfo category of the manual has been changed from 'Data Compression' to 'Compression' to match that of gzip. (Reported by Alfred M. Szmidt). Please send bug reports and suggestions to lzip-bug@nongnu.org Regards, Antonio Diaz, lziprecover author and maintainer. -- If you care about data safety and long-term archiving, please consider using lzip. See http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance and http://www.nongnu.org/lzip/safety_of_the_lzip_format.html Thanks.
[Lzip-bug] Pdlzip 1.12 released
I am pleased to announce the release of pdlzip 1.12. Pdlzip is a permissively licensed implementation of the lzip data compressor, intended for those who can't distribute (or even use) GPL licensed Free Software. (The name of pdlzip comes from 'public domain lzip'). Pdlzip is written in C and is (hope)fully compatible with lzip 1.4 or newer. Lzip is a lossless data compressor with a user interface similar to the one of gzip or bzip2. Lzip can compress about as fast as gzip (lzip -0) or compress most files more than bzip2 (lzip -9). Decompression speed is intermediate between gzip and bzip2. Lzip is better than gzip and bzip2 from a data recovery perspective. Lzip has been designed, written, and tested with great care to replace gzip and bzip2 as the standard general-purpose compressed format for unix-like systems. Pdlzip is also able to decompress legacy lzma-alone (.lzma) files. Pdlzip includes public domain (de)compression code from the LZMA SDK (Software Development Kit) written by Igor Pavlov. The homepage is at http://www.nongnu.org/lzip/pdlzip.html The sources can be downloaded from http://download.savannah.gnu.org/releases/lzip/pdlzip/ The sha256sums are: c69821cc22f7cdd431c28daecbf63ff27f1e2e83276c4917a2e695053c7ba734 pdlzip-1.12.tar.lz 44ad504bf262d0aa408a1ef7bb276c451b7c12d48b089e9eed76807db8d4fc13 pdlzip-1.12.tar.gz This release is also GPG signed. You can download the signature by appending '.sig' to the URL. If the 'gpg --verify' command fails because you don't have the required public key, then run this command to import it: gpg --keyserver keys.gnupg.net --recv-keys 8FE99503132D7742 Key fingerprint = 1D41 C14B 272A 2219 A739 FA4F 8FE9 9503 132D 7742 Changes in version 1.12: * In case of error in a numerical argument to a command line option, pdlzip now shows the name of the option and the range of valid values. Please send bug reports and suggestions to lzip-bug@nongnu.org Regards, Antonio Diaz, pdlzip co-author and maintainer. -- If you care about data safety and long-term archiving, please consider using lzip. See http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance and http://www.nongnu.org/lzip/safety_of_the_lzip_format.html Thanks.
Re: ChangeLog encoding; testsuite file/s
Adam Tuja wrote: BTW, the development version of gzip generates and compresses a file of 4 GiB of uncompressed size to test the new functionality of --list because it is the smallest file able to test it. Looks like you have answered yourself to this - they can be generated, as in gzip example you gave. Some can be generated, others can't (or it is more complicated to generate them than to include them in the tarball). Size is not the only important feature of a test file. Moreover, lzip's test script already generates a good number of test files on the fly using as building blocks the test files that you want to replace. I will prepare script later, when have more time. I don't know what problem you are trying to solve, but it surely can't be an excessive size of the lzip source tarball because it is in fact pretty small: -rw-r--r-- 1 772666 Dec 30 2018 gzip-1.10.tar.lz -rw-r--r-- 1 83501 Nov 30 20:15 lzip-1.23-rc1.tar.lz Best regards, Antonio.
Re: [lzlib] Renaming main.c to minilzip.c
Hello, p.z.l wrote: Hello. Speaking of releasing new versions I have suggestion regarding lzlib - to rename main.c to minilzip.c as main suggests it is a main part of library. Maybe even put examples separate directory. I had a problem with it when was testing lzlib in lzbench and got lots of multiple definition of main and stuff. Thanks. I gave main.c its name because it contains the 'main' function of minilzip; the only 'main' function existing then. (Lzlib, being a library, does not contain a 'main' function by itself). But now that there are other three files with 'main' functions (bbexample.c, ffexample.c, and lzcheck.c) I agree that it may be more informative to rename it to minilzip.c. About moving the examples (and lzcheck.c) to a separate directory, I would prefer not to do it because it complicates the package and the Makefile. I expect it to be easy to tell these four files from the library source files once main.c is renamed to minilzip.c. Best regards, Antonio.
Re: ChangeLog encoding; testsuite file/s
Hi Adam, Adam Tuja wrote: Hello, In latest lzlib Changlelog there is a name encoded in ISO (iso-8859-15 in this case). If there will be some other name, form other part of the world then it will be encoded differently, therefore displayed differently. You may want to use UTF-8 in this case to unify it (diff attached). Thank you, but I very much prefer the ISO-8859 encoding. The is a testsuite in every package which, in my opinion is unnecessarily so big. I refer here to `test*(.lz)' files. They don't have to be this big to show and test features they represent. Yes, they have to be that big, or even bigger. In lziprecover 1.16 a bug in --split was not detected by 'make check' because the test file was too small. In general a file needs to be larger than the input buffer to test that there are no problems when reaching the end of the buffer. Shipping large test files in a separate package won't work; nobody will download it to run the additional tests. BTW, the development version of gzip generates and compresses a file of 4 GiB of uncompressed size to test the new functionality of --list because it is the smallest file able to test it. Best regards, Antonio.
Re: adjust texinfo category for lzip
Hi Alfred, Alfred M. Szmidt wrote: This changes the dircategory to better match what is generally used for compression tools categories. Thank you very much for reporting this, and just at the right moment; I'm going to release a stable version of lzip this month. I'll apply the change also to all the related projects (plzip, lzlib...). Thanks, Antonio.
[Lzip-bug] Tarlz 0.22 released
I am pleased to announce the release of tarlz 0.22. Tarlz is a massively parallel (multi-threaded) combined implementation of the tar archiver and the lzip compressor. Tarlz uses the compression library lzlib. Tarlz creates tar archives using a simplified and safer variant of the POSIX pax format compressed in lzip format, keeping the alignment between tar members and lzip members. The resulting multimember tar.lz archive is fully backward compatible with standard tar tools like GNU tar, which treat it like any other tar.lz archive. Tarlz can append files to the end of such compressed archives. Keeping the alignment between tar members and lzip members has two advantages. It adds an indexed lzip layer on top of the tar archive, making it possible to decode the archive safely in parallel. It also minimizes the amount of data lost in case of corruption. Compressing a tar archive with plzip may even double the amount of files lost for each lzip member damaged because it does not keep the members aligned. Tarlz can create tar archives with five levels of compression granularity: per file (--no-solid), per block (--bsolid, default), per directory (--dsolid), appendable solid (--asolid), and solid (--solid). It can also create uncompressed tar archives. Of course, compressing each file (or each directory) individually can't achieve a compression ratio as high as compressing solidly the whole tar archive, but it has the following advantages: * The resulting multimember tar.lz archive can be decompressed in parallel, multiplying the decompression speed. * New members can be appended to the archive (by removing the EOF member), and unwanted members can be deleted from the archive. Just like an uncompressed tar archive. * It is a safe POSIX-style backup format. In case of corruption, tarlz can extract all the undamaged members from the tar.lz archive, skipping over the damaged members, just like the standard (uncompressed) tar. Moreover, the option '--keep-damaged' can be used to recover as much data as possible from each damaged member, and lziprecover can be used to recover some of the damaged members. * A multimember tar.lz archive is usually smaller than the corresponding solidly compressed tar.gz archive, except when individually compressing files smaller than about 32 KiB. Note that the POSIX pax format has a serious flaw. The metadata stored in pax extended records are not protected by any kind of check sequence. Because of this, tarlz protects the extended records with a Cyclic Redundancy Check (CRC) in a way compatible with standard tar tools. The homepage is at http://www.nongnu.org/lzip/tarlz.html An online manual for tarlz can be found at http://www.nongnu.org/lzip/manual/tarlz_manual.html The sources can be downloaded from http://download.savannah.gnu.org/releases/lzip/tarlz/ The sha256sum is: fccf7226fa24b55d326cab13f76ea349bec446c5a8df71a46d343099a05091dc tarlz-0.22.tar.lz Changes in version 0.22: * In case of error in a numerical argument to a command line option, tarlz now shows the name of the option and the range of valid values. * '--check-lib' now checks that LZ_API_VERSION and LZ_version_string match. * Tarlz now reports an error and exits with status 1 if '-o, --output' is used with any operation other than '-z, --compress'. * The variable LIBS can now be set from configure. Please send bug reports and suggestions to lzip-bug@nongnu.org Regards, Antonio Diaz, tarlz author and maintainer. -- If you care about data safety and long-term archiving, please consider using lzip. See http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance and http://www.nongnu.org/lzip/safety_of_the_lzip_format.html Thanks.
Re: [PATCH] doc: trivial fix
Alessandro Rubini wrote: I got involved by your request: If you report them, they will get fixed. If you don't, no one will ever know about them and they will remain unfixed for all eternity, if not longer. Never heard such a nice compelling, hint. I couldn't refuse :) It is certainly nice, but it is not mine. I copied it many years ago from glibc (I think): http://www.gnu.org/software/libc/manual/html_node/Reporting-Bugs.html "There are probably bugs in the GNU C Library. There are certainly errors and omissions in this manual. If you report them, they will get fixed. If you don’t, no one will ever know about them and they will remain unfixed for all eternity, if not longer." Best, Antonio.
Re: [PATCH] doc: trivial fix
Hello Alessandro,
Alessandro Rubini wrote:
I just read the lzip documentation and comparison with xz. Impressive
and enlightening. Thank you very much.
You are welcome. Glad to know that you find it interesting. :-)
OTOH, I didn't find the official repo. The cvs instructions on savannah
return an empty tree.
I work offline (and I like it this way), so there is no public repository.
Sorry.
-Example 3: Like example 1 but the created @samp{file.lz} is multimember with
+Example 3: Like example 2 but the created @samp{file.lz} is multimember with
Thank you very much for reporting this. It seems that I forgot to update the
reference when I inserted a new example before the first one.
Best regards,
Antonio.
[Lzip-bug] Lzip 1.23-rc1 released
Lzip 1.23-rc1 is ready for testing here http://download.savannah.gnu.org/releases/lzip/lzip-1.23-rc1.tar.lz http://download.savannah.gnu.org/releases/lzip/lzip-1.23-rc1.tar.gz The sha256sums are: 5543c9c575f333813f5b97004cb1e4488433265d6532ecb14d3e84c0e5add33b lzip-1.23-rc1.tar.lz 50c14f0f3162bb77c05fa3f96faa648a9515882455713c10872327205d84c32a lzip-1.23-rc1.tar.gz Please, test it and report any bugs you find. Lzip is a lossless data compressor with a user interface similar to the one of gzip or bzip2. Lzip uses a simplified form of the LZMA stream format and provides a 3 factor integrity checking to maximize interoperability and optimize safety. Lzip can compress about as fast as gzip (lzip -0) or compress most files more than bzip2 (lzip -9). Decompression speed is intermediate between gzip and bzip2. Lzip is better than gzip and bzip2 from a data recovery perspective. Lzip has been designed, written, and tested with great care to replace gzip and bzip2 as the standard general-purpose compressed format for unix-like systems. The lzip file format is designed for data sharing and long-term archiving, taking into account both data integrity and decoder availability: * The lzip format provides very safe integrity checking and some data recovery means. The program lziprecover can repair bit flip errors (one of the most common forms of data corruption) in lzip files, and provides data recovery capabilities, including error-checked merging of damaged copies of a file. * The lzip format is as simple as possible (but not simpler). The lzip manual provides the source code of a simple decompressor along with a detailed explanation of how it works, so that with the only help of the lzip manual it would be possible for a digital archaeologist to extract the data from a lzip file long after quantum computers eventually render LZMA obsolete. * Additionally the lzip reference implementation is copylefted, which guarantees that it will remain free forever. The homepage is at http://www.nongnu.org/lzip/lzip.html Changes in this version: * Decompression time has been reduced by 5-12% depending on the file. * In case of error in a numerical argument to a command line option, lzip now shows the name of the option and the range of valid values. * Several descriptions have been improved in manual, '--help', and man page. Regards, Antonio Diaz, lzip author and maintainer. -- If you care about data safety and long-term archiving, please consider using lzip. See http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance and http://www.nongnu.org/lzip/safety_of_the_lzip_format.html Thanks.
Re: Long help option
Adam Tuja wrote: there is a thing with the help (-h) that makes it increasingly harder to use - it's its length. [...] This way basic help would look like this: You may achieve almost the same effect with lzip -h | head -n 38 or lzip -h | head -n 42 without forcing novice users to search for the long help option to get the information they need. Regards, Antonio.
[Lzip-bug] Clzip 1.13-rc1 released
Clzip 1.13-rc1 is ready for testing here http://download.savannah.gnu.org/releases/lzip/clzip/clzip-1.13-rc1.tar.lz http://download.savannah.gnu.org/releases/lzip/clzip/clzip-1.13-rc1.tar.gz The sha256sums are: da904c1eb39a69951de1ed2d6a55825a5b0a6a1b05b1dff0732b83932b08863d clzip-1.13-rc1.tar.lz 3de78a7702ed2ca563eb76e342344526f0b630e9a5d3028088935eecbc5ca23f clzip-1.13-rc1.tar.gz Please, test it and report any bugs you find. Clzip is a C language version of lzip, fully compatible with lzip 1.4 or newer. As clzip is written in C, it may be easier to integrate in applications like package managers, embedded devices, or systems lacking a C++ compiler. Lzip is a lossless data compressor with a user interface similar to the one of gzip or bzip2. Lzip uses a simplified form of the 'Lempel-Ziv-Markov chain-Algorithm' (LZMA) stream format and provides a 3 factor integrity checking to maximize interoperability and optimize safety. Lzip can compress about as fast as gzip (lzip -0) or compress most files more than bzip2 (lzip -9). Decompression speed is intermediate between gzip and bzip2. Lzip is better than gzip and bzip2 from a data recovery perspective. Lzip has been designed, written, and tested with great care to replace gzip and bzip2 as the standard general-purpose compressed format for unix-like systems. The homepage is at http://www.nongnu.org/lzip/clzip.html Changes in this version: * Decompression time has been reduced by 5-12% depending on the file. * In case of error in a numerical argument to a command line option, clzip now shows the name of the option and the range of valid values. * Several descriptions have been improved in manual, '--help', and man page. Regards, Antonio Diaz, clzip author and maintainer. -- If you care about data safety and long-term archiving, please consider using lzip. See http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance and http://www.nongnu.org/lzip/safety_of_the_lzip_format.html Thanks.
[Lzip-bug] Plzip 1.10-rc1 released
Plzip 1.10-rc1 is ready for testing here http://download.savannah.gnu.org/releases/lzip/plzip/plzip-1.10-rc1.tar.lz http://download.savannah.gnu.org/releases/lzip/plzip/plzip-1.10-rc1.tar.gz The sha256sums are: 76474d0b5f8dbcbfd55cab0c4dbc335ec3e69fb25ed072b7867f90835833f026 plzip-1.10-rc1.tar.lz ae01862c41614fc4ab2d57bffee791c4d6a3ffe3459e6b3901681903dd275215 plzip-1.10-rc1.tar.gz Please, test it and report any bugs you find. Plzip is a massively parallel (multi-threaded) implementation of lzip, fully compatible with lzip 1.4 or newer. Plzip uses the compression library lzlib. Lzip is a lossless data compressor with a user interface similar to the one of gzip or bzip2. Lzip uses a simplified form of the 'Lempel-Ziv-Markov chain-Algorithm' (LZMA) stream format and provides a 3 factor integrity checking to maximize interoperability and optimize safety. Lzip can compress about as fast as gzip (lzip -0) or compress most files more than bzip2 (lzip -9). Decompression speed is intermediate between gzip and bzip2. Lzip is better than gzip and bzip2 from a data recovery perspective. Lzip has been designed, written, and tested with great care to replace gzip and bzip2 as the standard general-purpose compressed format for unix-like systems. Plzip can compress/decompress large files on multiprocessor machines much faster than lzip, at the cost of a slightly reduced compression ratio (0.4 to 2 percent larger compressed files). Note that the number of usable threads is limited by file size; on files larger than a few GB plzip can use hundreds of processors, but on files of only a few MB plzip is no faster than lzip. The homepage is at http://www.nongnu.org/lzip/plzip.html Changes in this version: * In case of error in a numerical argument to a command line option, plzip now shows the name of the option and the range of valid values. * '--check-lib' now checks that LZ_API_VERSION and LZ_version_string match. * The variable LIBS can now be set from configure. * Several descriptions have been improved in manual, '--help', and man page. Regards, Antonio Diaz, plzip co-author and maintainer. -- If you care about data safety and long-term archiving, please consider using lzip. See http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance and http://www.nongnu.org/lzip/safety_of_the_lzip_format.html Thanks.
[Lzip-bug] Lzlib 1.13-rc1 released
Lzlib 1.13-rc1 is ready for testing here http://download.savannah.gnu.org/releases/lzip/lzlib/lzlib-1.13-rc1.tar.lz http://download.savannah.gnu.org/releases/lzip/lzlib/lzlib-1.13-rc1.tar.gz The sha256sums are: 1ee92da3b837be5685bcd159937a43c5255c3c9de3c21213b79056517432400f lzlib-1.13-rc1.tar.lz b98b2ae2a35e3a8df72dfab60c4d1a80ab0a05cc0070223c9aa59f0f5d69a399 lzlib-1.13-rc1.tar.gz Please, test it and report any bugs you find. Lzlib is a data compression library providing in-memory LZMA compression and decompression functions, including integrity checking of the decompressed data. The compressed data format used by the library is the lzip format. Lzlib is written in C. The lzip file format is designed for data sharing and long-term archiving, taking into account both data integrity and decoder availability: * The lzip format provides very safe integrity checking and some data recovery means. The program lziprecover can repair bit flip errors (one of the most common forms of data corruption) in lzip files, and provides data recovery capabilities, including error-checked merging of damaged copies of a file. * The lzip format is as simple as possible (but not simpler). The lzip manual provides the source code of a simple decompressor along witha detailed explanation of how it works, so that with the only help ofthe lzip manual it would be possible for a digital archaeologist to extract the data from a lzip file long after quantum computers eventually render LZMA obsolete. * Additionally the lzip reference implementation is copylefted, which guarantees that it will remain free forever. The homepage is at http://www.nongnu.org/lzip/lzlib.html Changes in this version: * In case of error in a numerical argument to a command line option, minilzip now shows the name of the option and the range of valid values. * 'minilzip --check-lib' now checks that LZ_API_VERSION and LZ_version_string match. Regards, Antonio Diaz, lzlib author and maintainer. -- If you care about data safety and long-term archiving, please consider using lzip. See http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance and http://www.nongnu.org/lzip/safety_of_the_lzip_format.html Thanks.
[Lzip-bug] Pdlzip 1.12-rc1 released
Pdlzip 1.12-rc1 is ready for testing here http://download.savannah.gnu.org/releases/lzip/pdlzip/pdlzip-1.12-rc1.tar.lz http://download.savannah.gnu.org/releases/lzip/pdlzip/pdlzip-1.12-rc1.tar.gz The sha256sums are: 3a2d34f7179f39b2780d141593872c9a2ab4ed75d19d9d794617eaf3b85e22a8 pdlzip-1.12-rc1.tar.lz 7b9fbfb0623d2669bd20fcc5e3ee2b5d5ba08ecc3ad36eaa3ac2e5e2e3e08b5e pdlzip-1.12-rc1.tar.gz Please, test it and report any bugs you find. Pdlzip is a permissively licensed implementation of the lzip data compressor, intended for those who can't distribute (or even use) GPL licensed Free Software. (The name of pdlzip comes from 'public domain lzip'). Pdlzip is written in C and is (hope)fully compatible with lzip 1.4 or newer. Lzip is a lossless data compressor with a user interface similar to the one of gzip or bzip2. Lzip uses a simplified form of the 'Lempel-Ziv-Markov chain-Algorithm' (LZMA) stream format and provides a 3 factor integrity checking to maximize interoperability and optimize safety. Lzip can compress about as fast as gzip (lzip -0) or compress most files more than bzip2 (lzip -9). Decompression speed is intermediate between gzip and bzip2. Lzip is better than gzip and bzip2 from a data recovery perspective. Lzip has been designed, written, and tested with great care to replace gzip and bzip2 as the standard general-purpose compressed format for unix-like systems. Pdlzip is also able to decompress legacy lzma-alone (.lzma) files. Lzma-alone is a very bad format; it is essentially a raw LZMA stream. If you keep any lzma-alone files, it is advisable to recompress them to lzip format. Lziprecover can convert some lzma-alone files to lzip format without recompressing. Pdlzip includes public domain (de)compression code from the LZMA SDK (Software Development Kit) written by Igor Pavlov. The homepage is at http://www.nongnu.org/lzip/pdlzip.html Changes in this version: * In case of error in a numerical argument to a command line option, pdlzip now shows the name of the option and the range of valid values. Regards, Antonio Diaz, pdlzip co-author and maintainer. -- If you care about data safety and long-term archiving, please consider using lzip. See http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance and http://www.nongnu.org/lzip/safety_of_the_lzip_format.html Thanks.
[Lzip-bug] Lunzip 1.13-rc1 released
Lunzip 1.13-rc1 is ready for testing here http://download.savannah.gnu.org/releases/lzip/lunzip/lunzip-1.13-rc1.tar.lz http://download.savannah.gnu.org/releases/lzip/lunzip/lunzip-1.13-rc1.tar.gz The sha256sums are: f0d4befe9e6f445011b86cf3b734489ee266959bd9faaa326558c864a30c7a28 lunzip-1.13-rc1.tar.lz 37b8471035f09295c4f4072610ec68e13a716f8013760efd6144d3ba3bdf6ad5 lunzip-1.13-rc1.tar.gz Please, test it and report any bugs you find. Lunzip is a decompressor for the lzip format written in C. Its small size makes it well suited for embedded devices or software installers that need to decompress files but don't need compression capabilities. Lunzip is fully compatible with lzip 1.4 or newer and is able to decompress any lzip file using as little memory as 50 kB. The lzip file format is designed for data sharing and long-term archiving, taking into account both data integrity and decoder availability. The homepage is at http://www.nongnu.org/lzip/lunzip.html Changes in this version: * Decompression time has been reduced by 5-12% depending on the file. * In case of error in a numerical argument to a command line option, lunzip now shows the name of the option and the range of valid values. Regards, Antonio Diaz, lunzip author and maintainer. -- If you care about data safety and long-term archiving, please consider using lzip. See http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance and http://www.nongnu.org/lzip/safety_of_the_lzip_format.html Thanks.
[Lzip-bug] Lziprecover 1.23-rc1 released
Lziprecover 1.23-rc1 is ready for testing here http://download.savannah.gnu.org/releases/lzip/lziprecover/lziprecover-1.23-rc1.tar.lz http://download.savannah.gnu.org/releases/lzip/lziprecover/lziprecover-1.23-rc1.tar.gz The sha256sums are: 518247bd3b2a4c3c7f55bc9a29f118fb38ef64ce32934c55a99aa1db0cba825a lziprecover-1.23-rc1.tar.lz ad1406b805025857299efb7410f2937e2acb8bbe1180922be84b7028747bd9cd lziprecover-1.23-rc1.tar.gz Please, test it and report any bugs you find. Lziprecover is a data recovery tool and decompressor for files in the lzip compressed data format (.lz). Lziprecover is able to repair slightly damaged files, produce a correct file by merging the good parts of two or more damaged copies, reproduce a missing (zeroed) sector using a reference file, extract data from damaged files, decompress files, and test integrity of files. Lziprecover can remove the damaged members from multimember files, for example multimember tar.lz archives. Lziprecover provides random access to the data in multimember files; it only decompresses the members containing the desired data. Lziprecover facilitates the management of metadata stored as trailing data in lzip files. A nice feature of the lzip format is that a corrupt byte is easier to repair the nearer it is from the beginning of the file. Therefore, with the help of lziprecover, losing an entire archive just because of a corrupt byte near the beginning is a thing of the past. If the cause of file corruption is damaged media, the combination GNU ddrescue[1] + lziprecover is the best option for recovering data from damaged lzip files. The homepage is at http://www.nongnu.org/lzip/lziprecover.html Changes in this version: * Decompression time has been reduced by 5-12% depending on file. * In case of error in a numerical argument to a command line option, lziprecover now shows the name of the option and the range of valid values. * Options '--dump' and '--strip' now refuse to write compressed data to a terminal except when dumping trailing data with '--dump=tdata'. * The option '-U, --unzcrash' now requires an argument: '1' to test 1-bit errors, or 'B' to test zeroed blocks. * The memory tester now allocates the dictionary once per member instead of doing it for each test. This makes '-U, --unzcrash' about two times faster on my machine on files with an uncompressed size larger than about 30 MB. * '-W, --debug-decompress' now continues decompressing the members following the damaged member if it has been decompressed fully (just failed with a CRC mismatch). * The tool unzcrash now runs faster by using execvp instead of popen to avoid invoking /bin/sh. It also prints byte or block position in messages. * Several descriptions have been improved in manual, '--help', and man page. [1] http://www.gnu.org/software/ddrescue/ddrescue.html Regards, Antonio Diaz, lziprecover author and maintainer. -- If you care about data safety and long-term archiving, please consider using lzip. See http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance and http://www.nongnu.org/lzip/safety_of_the_lzip_format.html Thanks.
Re: Long help option
Hello Adam, Adam Tuja wrote: there is a thing with the help (-h) that makes it increasingly harder to use - it's its length. With every next release it becomes bigger and bigger and does not fit on the screen any more. For the program that that makes it's simplicity an asset it's a little paradox. Simplicity of format should not imply "scarcely documented tool". Have you tried 'lzip -h | less', or 'man lzip' ? I attempted to make it simpler and fit on the screen. I slightly rearranged it with more application related things on the top and version and help, on the bottom. I used different kinds of brackets to denote optional/required/default/miscellaneous arguments*. The --help output is used by help2man to create the man page. What is the effect of your rearrangement in the man page produced? To recoclie both needs - brief, quick help and detailed information I brought long help option for detailed help (-H, --help-long). This is a possibility, but I'm about to release lzip 1.23-rc1 in a few days, so it will have to wait until 1.24. All detailed help messages are available in long help option available. I prepared three versions of main.cc, using different ways to implement this long help option - using if()s, like some do (main.cc-if-printf.cc), array (main.cc-arr.cc) and array with -hh option (main.cc-arr-hh.cc). Everything is in temporal tarball (https://files.fm/f/eknq6kwys). I get the following message from that page: "Your browser does not support Javascript. To use this page turn on JavaScript or use another browser." Could you send the compressed tarball with the versions of main.cc to my private address? It can't be too large. Thanks. Regards, Antonio.
[Lzip-bug] Lzlib 1.13-pre1 released
Lzlib 1.13-pre1 is ready for testing here http://download.savannah.gnu.org/releases/lzip/lzlib/lzlib-1.13-pre1.tar.lz http://download.savannah.gnu.org/releases/lzip/lzlib/lzlib-1.13-pre1.tar.gz The sha256sums are: ffd516b96d942ef20facbf7d90efd0dbb841c078be31cfa7f4e6a086d3d68aad lzlib-1.13-pre1.tar.lz 7d4fb32b72d6e8601969da86aee9692812b66e862d20ba7d2d22a01bbfb92e50 lzlib-1.13-pre1.tar.gz Please, test it and report any bugs you find. Lzlib is a data compression library providing in-memory LZMA compression and decompression functions, including integrity checking of the decompressed data. The compressed data format used by the library is the lzip format. Lzlib is written in C. The lzip file format is designed for data sharing and long-term archiving, taking into account both data integrity and decoder availability: * The lzip format provides very safe integrity checking and some data recovery means. The program lziprecover can repair bit flip errors (one of the most common forms of data corruption) in lzip files, and provides data recovery capabilities, including error-checked merging of damaged copies of a file. * The lzip format is as simple as possible (but not simpler). The lzip manual provides the source code of a simple decompressor along with a detailed explanation of how it works, so that with the only help of the lzip manual it would be possible for a digital archaeologist to extract the data from a lzip file long after quantum computers eventually render LZMA obsolete. * Additionally the lzip reference implementation is copylefted, which guarantees that it will remain free forever. The homepage is at http://www.nongnu.org/lzip/lzlib.html Changes in this version: * The variables AR and ARFLAGS can now be set from configure. (Before you needed to run 'make AR='. (Reported by Hoël Bézier). * Several descriptions have been improved in manual, '--help', and man page of minilzip. Regards, Antonio Diaz, lzlib author and maintainer. -- If you care about data safety and long-term archiving, please consider using lzip. See http://www.nongnu.org/lzip/lzip_benchmark.html http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance and http://www.nongnu.org/lzip/safety_of_the_lzip_format.html Thanks.
Re: [PATCH] Makefile: do not call ar directly
Hi Hoël, Hoël Bézier wrote: I've now improved support for lzip compression in ./play.it and it awaits approval to be merged, thanks to you. Thanks. :-) Testing my contribution, I found out that plzip and tarlz are not considered stable on Gentoo, although they are on Debian. This might be because of a "bug" in lzlib, as reported here: https://bugs.gentoo.org/721916 The bug report targets lzlib-1.11, but I've been able to reproduce it with lzlib-1.12. I see. It is not exactly that the lzlib Makefile calls ar directly, (it uses the AR variable), but that AR can't currently be set from configure. I'll fix it immediately so that AR is treated just as CC and will release a corrected version soon. Thanks for reporting it. Of course, the ebuild used to compile the library on Gentoo also needs to be patched, and if I'm not mistaken, you're the maintainer, so I've attached my patch to the ebuild as well. If you're not, well tell me and I'll open an issue on Gentoo bugtracker myself. :) I think the maintainer of lzlib in Gentoo is Michal Górny. See http://packages.gentoo.org/packages/app-arch/lzlib . In fact I know very little about how Gentoo works. Thanks again, Antonio.
