I would start with a simple compression of network buffers as a blob. At some point, Flink's internal data layout may become columnar, which should also help the blob-style compression, because more similar strings will be within one window...
On Tue, Nov 25, 2014 at 11:26 AM, Viktor Rosenfeld < viktor.rosenf...@tu-berlin.de> wrote: > Hi, > > A codec like Snappy would work on an entire network buffer as one big blob, > right? I was more thinking along the lines of compressing individual tuples > fields by treating them as columns, e.g., using frame-of-reference encoding > and bit backing. Compression on tuple fields should yield much better > results than compressing the entire blob. Given that Flink controls the > serialization process this should be transparent to other layers in the > code. Not sure it is worth the effort though. > > Cheers, > Viktor > > > > -- > View this message in context: > http://apache-flink-incubator-mailing-list-archive.1008284.n3.nabble.com/Compression-of-network-traffic-tp2568p2607.html > Sent from the Apache Flink (Incubator) Mailing List archive. mailing list > archive at Nabble.com. >
