On 03/27/09 01:14, Brion Vibber wrote: > LZMA is nice and fast to decompress... but *insanely* slower to > compress, and doesn't seem as parallelizable. :(
The xz file format should allow for "easy" parallelization, both when compressing and decompressing; see http://tukaani.org/xz/xz-file-format.txt 3. Block 3.1. Block Header 3.1.1. Block Header Size 3.1.3. Compressed Size 3.1.4. Uncompressed Size 3.1.6. Header Padding 3.3. Block Padding At least in theory, this "length-prefixing" should make it fairly straightforward to write a multi-threaded decompressor with a splitter that can work from a pipe and is input-bound. I reckon the xz structure will eventually prove useful even for distributed compression/decompression. lacos _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
