Will McQueen created FLUME-1200:
-----------------------------------
Summary: HDFSEventSink causes *.snappy file to be created in HDFS
even when snappy isn't used (due to missing lib)
Key: FLUME-1200
URL: https://issues.apache.org/jira/browse/FLUME-1200
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Affects Versions: v1.2.0
Environment: RHEL 6.2 64-bit
Reporter: Will McQueen
Fix For: v1.2.0
If I use HDFSEventSink and specify the codec to be snappy, then the sink writes
data to HDFS with the ".snappy" extension... but the content of those HDFS
files is not in snappy format when the snappy libs aren't found. The log files
mention this:
2012-05-11 19:38:49,868 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2012-05-11 19:38:49,868 WARN snappy.LoadSnappy: Snappy native library not
loaded
...and I think it should be an error rather than a warning... the sink
shouldn't write data at all to HDFS if it's not in the format expected by the
config file (ie, not compressed with snappy). The config file I used is:
agent.channels = c1
agent.sources = r1
agent.sinks = k1
#
agent.channels.c1.type = MEMORY
#
agent.sources.r1.channels = c1
agent.sources.r1.type = SEQ
#
agent.sinks.k1.channel = c1
agent.sinks.k1.type = LOGGER
#
agent.sinks.k1.channel = c1
agent.sinks.k1.type = HDFS
agent.sinks.k1.hdfs.path = hdfs://<host>:<port>:<path>
agent.sinks.k1.hdfs.fileType = DataStream
agent.sinks.k1.hdfs.codeC = SnappyCodec
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
