https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/Config.java#L212-L217
topology.tuple.serializer
- Bobby
On Friday, September 18, 2015 12:54 PM, Onur Yalazı
<[email protected]> wrote:
Hello Bobby,
If it's possible to enable a TupleSerializer while submitting
topologies, without touching Storm's internals it would be really wise
and a fast way to implement compression. Actually I have no idea if it's
possible this way.
Other than that, If I got the Netty's gist right, to enable Zlib
compression one needs only 2 lines of code for each pipeline to enable
it. It's first class citizen of netty. Looking into
https://github.com/netty/netty/tree/master/codec/src/main/java/io/netty/handler/codec/compression
a good chunk of compression algorithms are also available.
From
https://github.com/netty/netty/blob/ed4a89082bb29b9e7d869c5d25d6b9ea8fc9d25b/example/src/main/java/io/netty/example/factorial/FactorialClientInitializer.java:
// Enable stream compression (you can remove these two if
unnecessary)
pipeline.addLast(ZlibCodecFactory.newZlibEncoder(ZlibWrapper.GZIP));
pipeline.addLast(ZlibCodecFactory.newZlibDecoder(ZlibWrapper.GZIP));
On 09/18/2015 08:32 PM, Bobby Evans wrote:
> Compression was just not something that we really though about all that much.
> The fastest route is probably to replace the tuple serializer with one that
> can handle compression. We did something similar for encryption.
> https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/security/serialization/BlowfishTupleSerializer.java
> But compression is generic enough it might be nice to make it a part of the
> Real TupleSerializer.
> https://github.com/EsotericSoftware/kryo#compression-and-encryption
>
> I would also suggest that you look at Snappy and LZO or LZ4 for your
> compression. As they tend to be much faster and still get good compression
> ratios. - Bobby
>
>
> On Friday, September 18, 2015 11:51 AM, Onur Yalazı
><[email protected]> wrote:
>
>
> Hello,
>
> I'm very new to storm world and the list, so Hello from Turkey.
>
> Because of a recent incident we had to increase our openstack network
> bandwidth soft limits from 1gb/s to 2gb/s.
>
> And of course even though the problem resides in our tuples' size and
> topology size, I thought if storm's netty was using ZlibEncoding.
> Looking into backtype.storm.messaging.netty.StormServerPipelineFactory
> and client counterpart, pipline seems to not have any compression handlers.
>
> Is it an intentional decision to not include compression in the
> pipeline? I know it would need more processing power and reduce topology
> performance, but I would like to know it it was considered before, and
> if not raise the issue.
>
> Thank you.
>
>
>
