Copy-jars.sh did not copy the hadoop consumer and kafka jars. I have copied them to HDFS manually, but still getting the same error.
Looks like I have all the reqd jars now: [root@idh251-0 test]# hadoop fs -ls /tmp/kafka/lib Warning: $HADOOP_HOME is deprecated. Found 7 items -rw-r--r-- 3 root hadoop 29614 2013-11-08 10:11 /tmp/kafka/lib/hadoop-consumer-0.8-SNAPSHOT.jar -rw-r--r-- 3 root hadoop 2559966 2013-11-08 10:05 /tmp/kafka/lib/kafka_2.8.0-0.8-SNAPSHOT.jar -rw-r--r-- 3 root hadoop 4502 2013-11-08 10:03 /tmp/kafka/lib/metrics-annotation-3.0.0-c0c8be71.jar -rw-r--r-- 3 root hadoop 80316 2013-11-08 10:03 /tmp/kafka/lib/metrics-core-3.0.0-c0c8be71.jar -rw-r--r-- 3 root hadoop 307858 2013-11-08 10:03 /tmp/kafka/lib/piggybank.jar -rw-r--r-- 3 root hadoop 8807888 2013-11-08 10:07 /tmp/kafka/lib/scala-library.jar -rw-r--r-- 3 root hadoop 98961 2013-11-08 10:03 /tmp/kafka/lib/zkclient-20120522.jar On Fri, Nov 8, 2013 at 9:17 AM, Abhi Basu <9000r...@gmail.com> wrote: > Still get the same error: > [root@idh251-0 hadoop-consumer]# ./run-class.sh > kafka.etl.impl.SimpleKafkaETLJob test/test.properties > > :./../../core/target/scala_2.8.0/kafka-*.jar:./../../contrib/hadoop-consumer/lib_managed/scala_2.8.0/compile/*.jar:./../../contrib/hadoop-consumer/target/scala_2.8.0/*.jar:./../../contrib/hadoop-consumer/lib/piggybank.jar:./../../project/boot/scala-2.8.0/lib/scala-library.jar > Exception in thread "main" java.lang.NoClassDefFoundError: > kafka/etl/impl/SimpleKafkaETLJob > Caused by: java.lang.ClassNotFoundException: > kafka.etl.impl.SimpleKafkaETLJob > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > Could not find the main class: kafka.etl.impl.SimpleKafkaETLJob. Program > will exit. > [root@idh251-0 hadoop-consumer]# hadoop fs -ls /tmp/kafka/lib > Warning: $HADOOP_HOME is deprecated. > > Here is what is in the HDFS /tmp/kafka/lib folder after running the > copy-jars.sh: > > Found 4 items > -rw-r--r-- 3 root hadoop 4502 2013-11-08 09:04 > /tmp/kafka/lib/metrics-annotation-3.0.0-c0c8be71.jar > -rw-r--r-- 3 root hadoop 80316 2013-11-08 09:04 > /tmp/kafka/lib/metrics-core-3.0.0-c0c8be71.jar > -rw-r--r-- 3 root hadoop 307858 2013-11-08 09:04 > /tmp/kafka/lib/piggybank.jar > -rw-r--r-- 3 root hadoop 98961 2013-11-08 09:04 > /tmp/kafka/lib/zkclient-20120522.jar > > > On Fri, Nov 8, 2013 at 9:06 AM, Abhi Basu <9000r...@gmail.com> wrote: > >> Ok, sorry, missed a step, ran the copy_jars.sh and now retrying .... >> >> >> On Fri, Nov 8, 2013 at 8:55 AM, Abhi Basu <9000r...@gmail.com> wrote: >> >>> Hi Neha: >>> >>> I was following the directions outlined here - >>> https://github.com/apache/kafka/tree/0.8/contrib/hadoop-consumer. It >>> does not mention anything about registering jars. Can you please provide >>> more details? >>> >>> Thanks, >>> >>> Abhi >>> >>> >>> On Fri, Nov 8, 2013 at 8:48 AM, Neha Narkhede >>> <neha.narkh...@gmail.com>wrote: >>> >>>> ClassNotFound means the Hadoop job is not able to find the related jar. >>>> Have you made sure the related jars are registered in the distributed >>>> cache? >>>> >>>> >>>> On Fri, Nov 8, 2013 at 8:40 AM, Abhi Basu <9000r...@gmail.com> wrote: >>>> >>>> > Can anyone help me with this issue? I feel like I am very close and am >>>> > probably making some silly config error. >>>> > >>>> > Kafka team, please provide more detailed notes on how to make this >>>> > component work. >>>> > >>>> > Thanks. >>>> > >>>> > >>>> > On Fri, Nov 8, 2013 at 5:23 AM, Abhi Basu <9000r...@gmail.com> wrote: >>>> > >>>> > > Simplekafkaetljob class, as mentioned in the post. >>>> > > >>>> > > Thanks >>>> > > >>>> > > Abhi >>>> > > >>>> > > From Samsung Galaxy S4 >>>> > > On Nov 7, 2013 8:34 PM, "Jun Rao" <jun...@gmail.com> wrote: >>>> > > >>>> > >> Which class is not found? >>>> > >> >>>> > >> Thanks, >>>> > >> >>>> > >> Jun >>>> > >> >>>> > >> >>>> > >> On Thu, Nov 7, 2013 at 11:56 AM, Abhi Basu <9000r...@gmail.com> >>>> wrote: >>>> > >> >>>> > >> > Let me describe my environment. Working on two nodes currently: >>>> > >> > 1.Single-node hadoop cluster (will refer as Node1) >>>> > >> > 2.Single node Kafka cluster (will refer as Node2) >>>> > >> > >>>> > >> > Node 2 has 1 broker started with a topic (iot.test.stream) and >>>> one >>>> > >> command >>>> > >> > line producer and one command line consumer to test the kafka >>>> install. >>>> > >> > Producer can send messages and the Consumer is receiving it. >>>> > >> > >>>> > >> > Node 1 (hadoop cluster) has kafka hadoop consumer code built. >>>> Have >>>> > >> edited >>>> > >> > the /kafka-0.8/contrib/hadoop-consumer/test/test.properties file >>>> with >>>> > >> the >>>> > >> > following: >>>> > >> > >>>> > >> > kafka.etl.topic=iot.test.stream >>>> > >> > hdfs.default.classpath.dir=/tmp/kafka/lib >>>> > >> > hadoop.job.ugi=kafka,hadoop >>>> > >> > kafka.server.uri=tcp://idh251-kafka:9095 >>>> > >> > input=/tmp/kafka/data >>>> > >> > output=/tmp/kafka/output >>>> > >> > kafka.request.limit=-1 >>>> > >> > ........... >>>> > >> > >>>> > >> > I have copied the copy-jars.sh to /tmp/kafka/lib (on HDFS) >>>> > >> > >>>> > >> > Next I run the following on Node 1: >>>> > >> > ./run-class.sh kafka.etl.impl.SimpleKafkaETLJob >>>> test/test.properties >>>> > >> from >>>> > >> > the /kafka-0.8/contrib/hadoop-consumer folder and get a >>>> > >> > classnotfoundexception for kafka.etl.impl.SimpleKafkaETLJob >>>> class. >>>> > >> > >>>> > >> > What am I missing? I was thinking that running the sh file would >>>> allow >>>> > >> me >>>> > >> > to retrieve messages with the same topic name to HDFS from Node >>>> 2 to >>>> > >> Node >>>> > >> > 1. I just want to do an end to end test to see that messages >>>> coming >>>> > into >>>> > >> > Kafka are being stored in HDFS with the minimal amount of code >>>> change >>>> > >> > required. >>>> > >> > >>>> > >> > Thanks, >>>> > >> > >>>> > >> > Abhi >>>> > >> > >>>> > >> > -- >>>> > >> > Abhi Basu >>>> > >> > >>>> > >> >>>> > > >>>> > >>>> > >>>> > -- >>>> > Abhi Basu >>>> > >>>> >>> >>> >>> >>> -- >>> Abhi Basu >>> >> >> >> >> -- >> Abhi Basu >> > > > > -- > Abhi Basu > -- Abhi Basu
