On 23 Nov 2014, at 21:04, Robert Metzger <rmetz...@apache.org> wrote:
> Ufuk can probably elaborate on the details, but he told me some time ago > that we can add code to compress outgoing network buffers using snappy or > other fast compression algorithms. In particular I/O intensive applications > would benefit from such a change. I agree and think that compressing shuffle data is generally a good idea (given codecs like Snappy et al.) I've opened a corresponding issue (https://issues.apache.org/jira/browse/FLINK-1275). The network layer is going through a set of big changes at the moment. I think it makes sense to wait for them to be finished/merged, before working on this. @Viktor: If you want to work on this, feel free to comment on the JIRA issue and we can discuss at which points you would have to extend the system. – Ufuk
