On 2017-05-10, Gary Gregory wrote: > I think the question is can/should [Compress] use any of the stock code > in java.util.zip in a multi-threaded fashion for performance gains.
We rely on java.util.zip.Deflater for DEFLATE which isn't thread safe by itself. But we could implement the same strategy pigz uses, which is to break up the stream into chunks and work on the chunks in parallel. Combining the output of several streams may become tricky using the Java API. If my first read of the comments in https://github.com/madler/pigz/blob/master/pigz.c is correct then we'd need to manipulate the output of Deflater in order to strip headers and trailers and insert empty stored blocks as well as create headers and trailers of our own for the combined output. In theory we could implement something like pigz on top of the LZ77 support I've added for Snappy and LZ4 (and some additional Hufmann code yet to be written) but it would be slower than zlib - probably a lot - and likely eat up the speed gain provided by parallel processing. Stefan --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
