Have you applied this patch https://issues.apache.org/jira/browse/FLUME-2172 ?
On Tue, Jan 21, 2014 at 5:29 PM, Himanshu Patidar < [email protected]> wrote: > I am getting the same error even after building flume with guava-11.0.2 > > Thanks, > Himanshu > > > ------------------------------ > Date: Tue, 21 Jan 2014 17:18:06 +0800 > Subject: Re: HDFS Sink Error > From: [email protected] > To: [email protected] > > > you can try this patch https://issues.apache.org/jira/browse/FLUME-2172 > and build flume with guava-11.0.2 the same version as hadoop2.0.x used, > currently flume use guava-10.0.1, so just change the version > > <dependency> > <groupId>com.google.guava</groupId> > <artifactId>guava</artifactId> > <version>10.0.1</version> > </dependency> > > > On Tue, Jan 21, 2014 at 3:35 PM, Himanshu Patidar < > [email protected]> wrote: > > Hi, > > I am trying to feed a single node HDFS cluster. > > But I getting this error : > > 2014-01-21 08:29:44,426 (SinkRunner-PollingRunner-DefaultSinkProcessor) > [INFO - > org.apache.flume.sink.hdfs.HDFSDataStream.configure(HDFSDataStream.java:56)] > Serializer = TEXT, UseRawLocalFileSystem = false > 2014-01-21 08:29:44,474 (SinkRunner-PollingRunner-DefaultSinkProcessor) > [INFO - > org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:219)] > Creating hdfs://xyz.16.137.81:54545/flume/FlumeData.1390289384427.tmp > 2014-01-21 08:29:44,878 (SinkRunner-PollingRunner-DefaultSinkProcessor) > [ERROR - > org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:422)] > process failed > java.lang.UnsupportedOperationException: This is supposed to be overridden > by subclasses. > at > com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180) > at > org.apache.hadoop.hdfs.protocol.proto.HdfsProtos$FsPermissionProto.getSerializedSize(HdfsProtos.java:5407) > at > com.google.protobuf.CodedOutputStream.computeMessageSizeNoTag(CodedOutputStream.java:749) > at > com.google.protobuf.CodedOutputStream.computeMessageSize(CodedOutputStream.java:530) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$CreateRequestProto.getSerializedSize(ClientNamenodeProtocolProtos.java:2371) > at > com.google.protobuf.AbstractMessageLite.toByteString(AbstractMessageLite.java:49) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.constructRpcRequest(ProtobufRpcEngine.java:149) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:193) > at $Proxy11.create(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83) > at $Proxy11.create(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:192) > at > org.apache.hadoop.hdfs.DFSOutputStream.<init>(DFSOutputStream.java:1298) > at > org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1317) > at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1242) > at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1199) > at > org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:273) > at > org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:262) > at > org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:79) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:851) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:832) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:731) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:720) > at > org.apache.flume.sink.hdfs.HDFSDataStream.open(HDFSDataStream.java:80) > at > org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:227) > at > org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:220) > at > org.apache.flume.sink.hdfs.BucketWriter$8$1.run(BucketWriter.java:557) > at > org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.java:160) > at > org.apache.flume.sink.hdfs.BucketWriter.access$1000(BucketWriter.java:56) > at > org.apache.flume.sink.hdfs.BucketWriter$8.call(BucketWriter.java:554) > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > > > My conf file :- > > Naming the components in this Agent > ############################### > httpagent.sources = http-source > httpagent.sinks = local-file-sink > httpagent.channels = ch3 > > # Define / Configure Source > ############################### > httpagent.sources.http-source.type = > org.apache.flume.source.http.HTTPSource > > httpagent.sources.http-source.channels = ch3 > httpagent.sources.http-source.port = 44444 > > > # Local File Sink > ############################### > httpagent.sinks.local-file-sink.type = hdfs > httpagent.sinks.local-file-sink.channel = ch3 > httpagent.sinks.local-file-sink.hdfs.path = > hdfs://xyz.16.137.81:54545/flume > httpagent.sinks.local-file-sink.hdfs.fileType = DataStream > #httpagent.sinks.local-file-sink.hdfs.filePrefix = events- > httpagent.sinks.local-file-sink.hdfs.round = true > httpagent.sinks.local-file-sink.hdfs.roundValue = 1 > httpagent.sinks.local-file-sink.hdfs.roundUnit = minute > > # Channels > ############################### > httpagent.channels.ch3.type = memory > httpagent.channels.ch3.capacity = 1000 > httpagent.channels.ch3.transactionCapacity = 100 > > > Apart from that I have these jars for hadoop/hdfs in the /lib folder :- > > commons-codec-1.4.jar > > commons-configuration-1.6.jar > > commons-httpclient-3.1.jar > > hadoop-annotations-2.0.0-cdh4.2.0.jar > > hadoop-auth-2.0.0-cdh4.2.0.jar > > hadoop-client-2.0.0-mr1-cdh4.2.0.jar > > hadoop-common-2.0.0-cdh4.2.0.jar > > hadoop-core-2.0.0-mr1-cdh4.2.0.jar > hadoop-hdfs-2.0.0-cdh4.2.0.jar > protobuf-java-2.5.0.jar > > I believe this error is coming from protobuf-java-2.5.0.jar file. > > Any suggestions will be of great help!!! > > > > Thanks, > Himanshu > > > > > -- > have a good day! > chenshang'an > > -- have a good day! chenshang'an
