Hey Eran, You need to make sure the Flume JVM gets passed -Djava.library.path=XXX with the correct path to where your native snappy libraries are located.
You can set this by adding the option directly to the flume-ng runner script. - Patrick On Fri, Aug 3, 2012 at 7:33 AM, Eran Kutner <[email protected]> wrote: > Hi, > I'm trying to use the snappy codec but keep getting "native snappy library > not available" errors. > I'm using CDH4 but replaced the flume 1.1 JARs that are included with that > distribution with flume 1.2 JARs. > I tried anything I can think of, including symlinking the hadoop native > library under flume-ng/lib/ dirctory both nothing helps. > Any idea how to resolve this? > > This is the error: > 2012-08-03 10:23:30,598 WARN util.NativeCodeLoader: Unable to load > native-hadoop library for your platform... using builtin-java classes where > applicable > 2012-08-03 10:23:35,670 WARN hdfs.HDFSEventSink: HDFS IO error > java.io.IOException: java.lang.RuntimeException: native snappy library not > available > at > org.apache.flume.sink.hdfs.BucketWriter.doOpen(BucketWriter.java:202) > at > org.apache.flume.sink.hdfs.BucketWriter.access$000(BucketWriter.java:48) > at > org.apache.flume.sink.hdfs.BucketWriter$1.run(BucketWriter.java:155) > at > org.apache.flume.sink.hdfs.BucketWriter$1.run(BucketWriter.java:152) > at > org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.java:125) > at > org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:152) > at > org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:307) > at > org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:717) > at > org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:714) > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.lang.RuntimeException: native snappy library not available > at > org.apache.hadoop.io.compress.SnappyCodec.createCompressor(SnappyCodec.java:135) > at > org.apache.hadoop.io.compress.SnappyCodec.createOutputStream(SnappyCodec.java:84) > at > org.apache.flume.sink.hdfs.HDFSCompressedDataStream.open(HDFSCompressedDataStream.java:70) > at > org.apache.flume.sink.hdfs.BucketWriter.doOpen(BucketWriter.java:195) > ... 13 more > > And my sink configuration: > flume05.sinks.hdfsSink.type = hdfs > #flume05.sinks.hdfsSink.type = logger > flume05.sinks.hdfsSink.channel = memoryChannel > flume05.sinks.hdfsSink.hdfs.path=hdfs://hadoop2-m1:8020/test-events/%Y-%m-%d > flume05.sinks.hdfsSink.hdfs.filePrefix=raw-events.avro > flume05.sinks.hdfsSink.hdfs.rollInterval=60 > flume05.sinks.hdfsSink.hdfs.rollCount=0 > flume05.sinks.hdfsSink.hdfs.rollSize=0 > flume05.sinks.hdfsSink.hdfs.fileType=CompressedStream > flume05.sinks.hdfsSink.hdfs.codeC=snappy > flume05.sinks.hdfsSink.hdfs.writeFormat=Text > flume05.sinks.hdfsSink.hdfs.batchSize=1000 > flume05.sinks.hdfsSink.serializer = avro_event > > Thanks. > > -eran >
