Still get the same error: [root@idh251-0 hadoop-consumer]# ./run-class.sh kafka.etl.impl.SimpleKafkaETLJob test/test.properties :./../../core/target/scala_2.8.0/kafka-*.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/*.jar:./../../contrib/hadoop-consumer/target/scala_2.8.0/*.jar:./../../contrib/hadoop-consumer/lib/piggybank.jar:./../../project/boot/scala-2.8.0/lib/scala-library.jar Exception in thread "main" java.lang.NoClassDefFoundError: kafka/etl/impl/SimpleKafkaETLJob Caused by: java.lang.ClassNotFoundException: kafka.etl.impl.SimpleKafkaETLJob at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) Could not find the main class: kafka.etl.impl.SimpleKafkaETLJob. Program will exit. [root@idh251-0 hadoop-consumer]# hadoop fs -ls /tmp/kafka/lib Warning: $HADOOP_HOME is deprecated.
Here is what is in the HDFS /tmp/kafka/lib folder after running the copy-jars.sh: Found 4 items -rw-r--r-- 3 root hadoop 4502 2013-11-08 09:04 /tmp/kafka/lib/metrics-annotation-3.0.0-c0c8be71.jar -rw-r--r-- 3 root hadoop 80316 2013-11-08 09:04 /tmp/kafka/lib/metrics-core-3.0.0-c0c8be71.jar -rw-r--r-- 3 root hadoop 307858 2013-11-08 09:04 /tmp/kafka/lib/piggybank.jar -rw-r--r-- 3 root hadoop 98961 2013-11-08 09:04 /tmp/kafka/lib/zkclient-20120522.jar On Fri, Nov 8, 2013 at 9:06 AM, Abhi Basu <9000r...@gmail.com> wrote: > Ok, sorry, missed a step, ran the copy_jars.sh and now retrying .... > > > On Fri, Nov 8, 2013 at 8:55 AM, Abhi Basu <9000r...@gmail.com> wrote: > >> Hi Neha: >> >> I was following the directions outlined here - >> https://github.com/apache/kafka/tree/0.8/contrib/hadoop-consumer. It >> does not mention anything about registering jars. Can you please provide >> more details? >> >> Thanks, >> >> Abhi >> >> >> On Fri, Nov 8, 2013 at 8:48 AM, Neha Narkhede <neha.narkh...@gmail.com>wrote: >> >>> ClassNotFound means the Hadoop job is not able to find the related jar. >>> Have you made sure the related jars are registered in the distributed >>> cache? >>> >>> >>> On Fri, Nov 8, 2013 at 8:40 AM, Abhi Basu <9000r...@gmail.com> wrote: >>> >>> > Can anyone help me with this issue? I feel like I am very close and am >>> > probably making some silly config error. >>> > >>> > Kafka team, please provide more detailed notes on how to make this >>> > component work. >>> > >>> > Thanks. >>> > >>> > >>> > On Fri, Nov 8, 2013 at 5:23 AM, Abhi Basu <9000r...@gmail.com> wrote: >>> > >>> > > Simplekafkaetljob class, as mentioned in the post. >>> > > >>> > > Thanks >>> > > >>> > > Abhi >>> > > >>> > > From Samsung Galaxy S4 >>> > > On Nov 7, 2013 8:34 PM, "Jun Rao" <jun...@gmail.com> wrote: >>> > > >>> > >> Which class is not found? >>> > >> >>> > >> Thanks, >>> > >> >>> > >> Jun >>> > >> >>> > >> >>> > >> On Thu, Nov 7, 2013 at 11:56 AM, Abhi Basu <9000r...@gmail.com> >>> wrote: >>> > >> >>> > >> > Let me describe my environment. Working on two nodes currently: >>> > >> > 1.Single-node hadoop cluster (will refer as Node1) >>> > >> > 2.Single node Kafka cluster (will refer as Node2) >>> > >> > >>> > >> > Node 2 has 1 broker started with a topic (iot.test.stream) and one >>> > >> command >>> > >> > line producer and one command line consumer to test the kafka >>> install. >>> > >> > Producer can send messages and the Consumer is receiving it. >>> > >> > >>> > >> > Node 1 (hadoop cluster) has kafka hadoop consumer code built. Have >>> > >> edited >>> > >> > the /kafka-0.8/contrib/hadoop-consumer/test/test.properties file >>> with >>> > >> the >>> > >> > following: >>> > >> > >>> > >> > kafka.etl.topic=iot.test.stream >>> > >> > hdfs.default.classpath.dir=/tmp/kafka/lib >>> > >> > hadoop.job.ugi=kafka,hadoop >>> > >> > kafka.server.uri=tcp://idh251-kafka:9095 >>> > >> > input=/tmp/kafka/data >>> > >> > output=/tmp/kafka/output >>> > >> > kafka.request.limit=-1 >>> > >> > ........... >>> > >> > >>> > >> > I have copied the copy-jars.sh to /tmp/kafka/lib (on HDFS) >>> > >> > >>> > >> > Next I run the following on Node 1: >>> > >> > ./run-class.sh kafka.etl.impl.SimpleKafkaETLJob >>> test/test.properties >>> > >> from >>> > >> > the /kafka-0.8/contrib/hadoop-consumer folder and get a >>> > >> > classnotfoundexception for kafka.etl.impl.SimpleKafkaETLJob class. >>> > >> > >>> > >> > What am I missing? I was thinking that running the sh file would >>> allow >>> > >> me >>> > >> > to retrieve messages with the same topic name to HDFS from Node 2 >>> to >>> > >> Node >>> > >> > 1. I just want to do an end to end test to see that messages >>> coming >>> > into >>> > >> > Kafka are being stored in HDFS with the minimal amount of code >>> change >>> > >> > required. >>> > >> > >>> > >> > Thanks, >>> > >> > >>> > >> > Abhi >>> > >> > >>> > >> > -- >>> > >> > Abhi Basu >>> > >> > >>> > >> >>> > > >>> > >>> > >>> > -- >>> > Abhi Basu >>> > >>> >> >> >> >> -- >> Abhi Basu >> > > > > -- > Abhi Basu > -- Abhi Basu
