On 2017-05-10, Matt Sicker wrote:
> Would the scattering and gathering byte channel APIs in java.nio be helpful
> in splitting up a stream into chunks for parallel processing?
Possibly. pigz breaks up the stream into chunks of 128k and using the
scattering part we should be able to do the same. I
Exactly.
-Original Message-
From: Gary Gregory [mailto:garydgreg...@gmail.com]
Sent: Tuesday, May 09, 2017 5:29 PM
To: Commons Developers List
Subject: Re: [COMPRESS] Anyone implemented "pigz"?
I think the question is can/should [Compress] use any of the stock code in
java.u
Would the scattering and gathering byte channel APIs in java.nio be helpful
in splitting up a stream into chunks for parallel processing?
On 10 May 2017 at 02:57, Stefan Bodewig wrote:
> On 2017-05-10, Gary Gregory wrote:
>
> > I think the question is can/should [Compress] use any of the stock c
On 2017-05-10, Gary Gregory wrote:
> I think the question is can/should [Compress] use any of the stock code
> in java.util.zip in a multi-threaded fashion for performance gains.
We rely on java.util.zip.Deflater for DEFLATE which isn't thread safe by
itself.
But we could implement the same stra
Those C libraries are pthread (don't need that in Java as it has its own
thread API) and zlib (pretty standard gz library). With that in mind, this
may be a useful reference: http://www.jcraft.com/jzlib/
On 9 May 2017 at 19:22, sebb wrote:
> AFAICT the implementation is written in C and uses som
I think the question is can/should [Compress] use any of the stock code
in java.util.zip in a multi-threaded fashion for performance gains.
Gary
On Tue, May 9, 2017 at 5:22 PM, sebb wrote:
> AFAICT the implementation is written in C and uses some C libraries.
>
> It would have to be completely
AFAICT the implementation is written in C and uses some C libraries.
It would have to be completely rewritten for Java.
Not a trivial job, though it may be possible to use the algorithm.
On 10 May 2017 at 01:03, Gary Gregory wrote:
> I've not heard of it on the ML yet. Go for it! ;-)
>
> Gary
>
I've not heard of it on the ML yet. Go for it! ;-)
Gary
On Tue, May 9, 2017 at 4:44 PM, Roger Whitcomb
wrote:
> Someone here was doing benchmarks using "pigz" (see here:
> http://zlib.net/pigz/, basically multi-threaded "gzip") and I couldn't
> find any "reasonable" Java implementations. Anyon
Someone here was doing benchmarks using "pigz" (see here:
http://zlib.net/pigz/, basically multi-threaded "gzip") and I couldn't find any
"reasonable" Java implementations. Anyone thought about it for Commons
Compress?
Thanks,
Roger Whitcomb
