Hi Neha: I was following the directions outlined here - https://github.com/apache/kafka/tree/0.8/contrib/hadoop-consumer. It does not mention anything about registering jars. Can you please provide more details?
Thanks, Abhi On Fri, Nov 8, 2013 at 8:48 AM, Neha Narkhede <neha.narkh...@gmail.com>wrote: > ClassNotFound means the Hadoop job is not able to find the related jar. > Have you made sure the related jars are registered in the distributed > cache? > > > On Fri, Nov 8, 2013 at 8:40 AM, Abhi Basu <9000r...@gmail.com> wrote: > > > Can anyone help me with this issue? I feel like I am very close and am > > probably making some silly config error. > > > > Kafka team, please provide more detailed notes on how to make this > > component work. > > > > Thanks. > > > > > > On Fri, Nov 8, 2013 at 5:23 AM, Abhi Basu <9000r...@gmail.com> wrote: > > > > > Simplekafkaetljob class, as mentioned in the post. > > > > > > Thanks > > > > > > Abhi > > > > > > From Samsung Galaxy S4 > > > On Nov 7, 2013 8:34 PM, "Jun Rao" <jun...@gmail.com> wrote: > > > > > >> Which class is not found? > > >> > > >> Thanks, > > >> > > >> Jun > > >> > > >> > > >> On Thu, Nov 7, 2013 at 11:56 AM, Abhi Basu <9000r...@gmail.com> > wrote: > > >> > > >> > Let me describe my environment. Working on two nodes currently: > > >> > 1.Single-node hadoop cluster (will refer as Node1) > > >> > 2.Single node Kafka cluster (will refer as Node2) > > >> > > > >> > Node 2 has 1 broker started with a topic (iot.test.stream) and one > > >> command > > >> > line producer and one command line consumer to test the kafka > install. > > >> > Producer can send messages and the Consumer is receiving it. > > >> > > > >> > Node 1 (hadoop cluster) has kafka hadoop consumer code built. Have > > >> edited > > >> > the /kafka-0.8/contrib/hadoop-consumer/test/test.properties file > with > > >> the > > >> > following: > > >> > > > >> > kafka.etl.topic=iot.test.stream > > >> > hdfs.default.classpath.dir=/tmp/kafka/lib > > >> > hadoop.job.ugi=kafka,hadoop > > >> > kafka.server.uri=tcp://idh251-kafka:9095 > > >> > input=/tmp/kafka/data > > >> > output=/tmp/kafka/output > > >> > kafka.request.limit=-1 > > >> > ........... > > >> > > > >> > I have copied the copy-jars.sh to /tmp/kafka/lib (on HDFS) > > >> > > > >> > Next I run the following on Node 1: > > >> > ./run-class.sh kafka.etl.impl.SimpleKafkaETLJob test/test.properties > > >> from > > >> > the /kafka-0.8/contrib/hadoop-consumer folder and get a > > >> > classnotfoundexception for kafka.etl.impl.SimpleKafkaETLJob class. > > >> > > > >> > What am I missing? I was thinking that running the sh file would > allow > > >> me > > >> > to retrieve messages with the same topic name to HDFS from Node 2 to > > >> Node > > >> > 1. I just want to do an end to end test to see that messages coming > > into > > >> > Kafka are being stored in HDFS with the minimal amount of code > change > > >> > required. > > >> > > > >> > Thanks, > > >> > > > >> > Abhi > > >> > > > >> > -- > > >> > Abhi Basu > > >> > > > >> > > > > > > > > > -- > > Abhi Basu > > > -- Abhi Basu
