[
https://issues.apache.org/jira/browse/FLUME-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15885300#comment-15885300
]
Paul Lin commented on FLUME-2370:
---------------------------------
If the deflate compression is enabled, any uncompressed data should not be sent
to the Avro source, or you will get the decompression error.
{code:title=FlumeSdkExample.java|borderStyle=solid}
//If you're using flume sdk, use this to initiate RpcClient
Properties props = new Properties();
props.put("hosts", "h1");
props.put("hosts.h1", socket);
props.put("compression-type", "deflate");
props.put("compression-level". "6");
RpcClient client = RpcClientFactory.getInstance(props);
{code}
> 'deflate' doesn't work in Flume avro sinc/source
> ------------------------------------------------
>
> Key: FLUME-2370
> URL: https://issues.apache.org/jira/browse/FLUME-2370
> Project: Flume
> Issue Type: Documentation
> Components: Docs
> Affects Versions: v1.4.0
> Reporter: Cesar Delgado
> Attachments: patch-2370.diff
>
>
> I was trying to setup a basic 2-tier Flume setup using the Avro source/sink
> to communicate between tiers. When I setup the system I used compression-type
> on the 1st tier's sink side like documented here
> (https://flume.apache.org/FlumeUserGuide.html#avro-sink)
> agent.sinks.snk1.compression-type = deflate
> And the same on the source side:
> agent.sources.src1.compression-type = deflate
> But I was getting this error:
> 08 Apr 2014 00:15:07,606 WARN pool-7-thread-1
> (org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.exceptionCaught:201)
> - Unexpected exception from downstream.
> org.jboss.netty.handler.codec.compression.CompressionException: decompression
> failure (-3): unknown compression method
> at
> org.jboss.netty.handler.codec.compression.ZlibUtil.exception(ZlibUtil.java:31)
> at org.jboss.netty.handler.codec.compression.ZlibUtil.fail(ZlibUtil.java:27)
> at
> org.jboss.netty.handler.codec.compression.ZlibDecoder.decode(ZlibDecoder.java:153)
> at
> org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:67)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> After some digging I found if I changed 'deflate' for 'gzip' it works.
> agent.sinks.snk1.compression-type = gzip
> agent.sources.src1.compression-type = gzip
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
