Will the compression Codec will be inserted in the Netty loops, or before that?
In any case, would it make sense to prototype this on the current code and forward port this to the new network stack later? I assume the code would mostly be similar, especially all the JNI vs. Java considerations and tricks. Am 23.11.2014 23:11 schrieb "Ufuk Celebi" <u...@apache.org>: > > On 23 Nov 2014, at 21:04, Robert Metzger <rmetz...@apache.org> wrote: > > > Ufuk can probably elaborate on the details, but he told me some time ago > > that we can add code to compress outgoing network buffers using snappy or > > other fast compression algorithms. In particular I/O intensive > applications > > would benefit from such a change. > > I agree and think that compressing shuffle data is generally a good idea > (given codecs like Snappy et al.) > > I've opened a corresponding issue ( > https://issues.apache.org/jira/browse/FLINK-1275). The network layer is > going through a set of big changes at the moment. I think it makes sense to > wait for them to be finished/merged, before working on this. > > @Viktor: If you want to work on this, feel free to comment on the JIRA > issue and we can discuss at which points you would have to extend the > system. > > – Ufuk
