Hi all,
I want upload those collected data into amazon s3 cluster. But I encounter the
below when I set
agent.sinks.k1.hdfs.path = s3://bigdata/test <s3://bigdata/test>
14 Nov 2015 16:26:01,518 INFO [hdfs-k1-call-runner-0]
(org.apache.flume.sink.hdfs.AbstractHDFSWriter.reflectGetNumCurrentReplicas:188)
- FileSystem's output stream doesn't support getNumCurrentReplicas;
--HDFS-826 not available;
fsOut=com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream;
err=java.lang.NoSuchMethodException:
com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream.getNumCurrentReplicas()
14 Nov 2015 16:26:01,518 INFO [hdfs-k1-call-runner-0]
(org.apache.flume.sink.hdfs.AbstractHDFSWriter.reflectGetNumCurrentReplicas:188)
- FileSystem's output stream doesn't support getNumCurrentReplicas;
--HDFS-826 not available;
fsOut=com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream;
err=java.lang.NoSuchMethodException:
com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream.getNumCurrentReplicas()
14 Nov 2015 16:26:01,518 WARN [SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.flume.sink.hdfs.BucketWriter.getRefIsClosed:183) - isFileClosed is
not available in the version of HDFS being used. Flume will not attempt to
close files if the close fails on the first attempt
java.lang.NoSuchMethodException:
com.amazon.ws.emr.hadoop.fs.EmrFileSystem.isFileClosed(org.apache.hadoop.fs.Path)
at java.lang.Class.getMethod(Class.java:1665)
at
org.apache.flume.sink.hdfs.BucketWriter.getRefIsClosed(BucketWriter.java:180)
at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:268)
at org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:514)
at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:418)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
I have found the jira issue about it:
https://issues.apache.org/jira/browse/FLUME-2427
<https://issues.apache.org/jira/browse/FLUME-2427>
But i don’t find a solution to fix the WARN issue and i can’t find the
com.amazon.ws.emr.hadoop.fs.EmrFileSystem API.
Please help me fix it.
Thanks
