Always depends on the data - we need to measure that. From what I have heard in Hadoop, anywhere between none and a factor of 3-4 may happen. Especially if the records contain repetitive strings... Am 24.11.2014 10:24 schrieb "Sebastian Schelter" <s...@apache.org>:
> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > What reduction in network traffic can we expect from using compression? > > - -s > > > > On 11/24/2014 10:20 AM, Stephan Ewen wrote: > > Will the compression Codec will be inserted in the Netty loops, or > > before that? > > > > In any case, would it make sense to prototype this on the current > > code and forward port this to the new network stack later? I assume > > the code would mostly be similar, especially all the JNI vs. Java > > considerations and tricks. Am 23.11.2014 23:11 schrieb "Ufuk > > Celebi" <u...@apache.org>: > > > >> > >> On 23 Nov 2014, at 21:04, Robert Metzger <rmetz...@apache.org> > >> wrote: > >> > >>> Ufuk can probably elaborate on the details, but he told me some > >>> time ago that we can add code to compress outgoing network > >>> buffers using snappy or other fast compression algorithms. In > >>> particular I/O intensive > >> applications > >>> would benefit from such a change. > >> > >> I agree and think that compressing shuffle data is generally a > >> good idea (given codecs like Snappy et al.) > >> > >> I've opened a corresponding issue ( > >> https://issues.apache.org/jira/browse/FLINK-1275). The network > >> layer is going through a set of big changes at the moment. I > >> think it makes sense to wait for them to be finished/merged, > >> before working on this. > >> > >> @Viktor: If you want to work on this, feel free to comment on the > >> JIRA issue and we can discuss at which points you would have to > >> extend the system. > >> > >> – Ufuk > > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1 > > iQIcBAEBAgAGBQJUcvjyAAoJEFWNjCUTnssiCYgQAK5QzsYPW9Q+C8BUJBxfeudg > 61zFPW3cfydfD4L6UXghwz/j7addMm5zdY/xJsFgmN4NAXrtacDfr6+KPyB9glZ0 > E8E201q8NN6Czek/TtWEG0SSLj+KXqRAwJBHfBCVaCV/lD7wEAjef2sSFpVL5RK9 > 0RyS5UeCBW/RxTp/fGXWfy2d9G0SSTIy78/u8Y9nxflD6KnZVGZYZA7jrP/h8Wis > ry1ZWLPHKpEbaC+MNbYX1mMJaf9lH0liJ/HopCIUzrxfiwskub17NkfVC/0Uwwc3 > r1+/pHFYNfO3FLIkXxWYRLgNyGW6LCE1zmM/BGUo1/C9/rtuiRl24io+G+4V/ac6 > tZ9jxjwHQkWGFscoep2p7E8pGVezV1QqYv3prWXaZ2+TUKKzlbd6QCL8i2qmymeb > ABy88cpN1U4c/p6GDrva1+llyBG1PK07awAfFmK3OA7VK3MLW1pyHuJiLjOZaFP1 > r2K9T5/d2YZa7nPFaJs8bLXsV/WMzFJDAOaILcaCDymxF1xnNu7iwAyuJhg/i6ht > KaOiM45JnqC++9MQ2D83zyt/+Yc/OT+06hVN7hdRaLFD+OqG7vNv7LZC+cfAWmFE > KyJb9kXNVDxhJXAWyOyKN1nzBtDfaugghGBzvX0u3QP/2uGX1Gca2oqCJvNesOM4 > dBlCMt88/gkYjWXadjnK > =0q6A > -----END PGP SIGNATURE----- >
