bcc: flume-user@incubator.apache.org This issue involves a version of Flume distributed with CDH, so moving to cdh-u...@cloudera.org which is the proper support channel.
Thanks, Mike On Friday, May 11, 2012 at 1:02 PM, lexs...@gmail.com wrote: > Hello, > > I am new to Flume. I am trying to use hdfs as a sink. My first > problem is that flume hangs during initialization. > Same problem as: > > https://issues.apache.org/jira/browse/FLUME-1077 > > and > > https://issues.apache.org/jira/browse/FLUME-1093 > > My version of Hadoop is Cloudera's "cdh3u4", and Flume version is > "flume-ng-1.1.0-cdh3u4" > I noticed that Hadoop and Flume have different versions of Guava. > > If I remove Flume's Guava jar and use Hadoop's: > > <code> > bin/flume-ng agent --conf ./conf/ -f ~/tail2hdfs.conf -n agent1 > --classpath > ~/hadoop-0.20.2-cdh3u4/hadoop-core-0.20.2-cdh3u4.jar:~/hadoop-0.20.2-cdh3u4/lib/guava-r09-jarjar.jar > </code> > > I get this error: > > <code> > Exception in thread "main" java.lang.NoClassDefFoundError: > com/google/common/util/concurrent/ThreadFactoryBuilder > at > org.apache.flume.lifecycle.LifecycleSupervisor.<init>(LifecycleSupervisor.java:54) > at org.apache.flume.node.FlumeNode.<init>(FlumeNode.java:42) > at org.apache.flume.node.Application.run(Application.java:135) > at org.apache.flume.node.Application.main(Application.java:70) > Caused by: java.lang.ClassNotFoundException: > com.google.common.util.concurrent.ThreadFactoryBuilder > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > ... 4 more > </code> > > If I put back Flume's Guava jar: > > <code> > bin/flume-ng agent --conf ./conf/ -f ~/tail2hdfs.conf -n agent1 > --classpath ~/hadoop-0.20.2-cdh3u4/hadoop-core-0.20.2-cdh3u4.jar > </code> > > I get this error: > > <code> > Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" > java.lang.NoClassDefFoundError: > org/apache/hadoop/thirdparty/guava/common/collect/LinkedListMultimap > at org.apache.hadoop.hdfs.SocketCache.<init>(SocketCache.java:48) > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:249) > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:217) > at > org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89) > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1563) > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67) > at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1597) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1579) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:183) > at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:57) > at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:49) > at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:95) > at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:117) > at org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:361) > at org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:357) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:680) > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.thirdparty.guava.common.collect.LinkedListMultimap > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > ... 21 more > </code> > > Has anybody been able to successfully run Flume with a HDFS sink? > What's the magic combination? > > -- > Alex Soto > lexs...@gmail.com (mailto:lexs...@gmail.com) > > - > Amicus Plato, sed magis amica veritas. > >
