Ok, sorry, missed a step, ran the copy_jars.sh and now retrying ....
On Fri, Nov 8, 2013 at 8:55 AM, Abhi Basu <9000r...@gmail.com> wrote: > Hi Neha: > > I was following the directions outlined here - > https://github.com/apache/kafka/tree/0.8/contrib/hadoop-consumer. It does > not mention anything about registering jars. Can you please provide more > details? > > Thanks, > > Abhi > > > On Fri, Nov 8, 2013 at 8:48 AM, Neha Narkhede <neha.narkh...@gmail.com>wrote: > >> ClassNotFound means the Hadoop job is not able to find the related jar. >> Have you made sure the related jars are registered in the distributed >> cache? >> >> >> On Fri, Nov 8, 2013 at 8:40 AM, Abhi Basu <9000r...@gmail.com> wrote: >> >> > Can anyone help me with this issue? I feel like I am very close and am >> > probably making some silly config error. >> > >> > Kafka team, please provide more detailed notes on how to make this >> > component work. >> > >> > Thanks. >> > >> > >> > On Fri, Nov 8, 2013 at 5:23 AM, Abhi Basu <9000r...@gmail.com> wrote: >> > >> > > Simplekafkaetljob class, as mentioned in the post. >> > > >> > > Thanks >> > > >> > > Abhi >> > > >> > > From Samsung Galaxy S4 >> > > On Nov 7, 2013 8:34 PM, "Jun Rao" <jun...@gmail.com> wrote: >> > > >> > >> Which class is not found? >> > >> >> > >> Thanks, >> > >> >> > >> Jun >> > >> >> > >> >> > >> On Thu, Nov 7, 2013 at 11:56 AM, Abhi Basu <9000r...@gmail.com> >> wrote: >> > >> >> > >> > Let me describe my environment. Working on two nodes currently: >> > >> > 1.Single-node hadoop cluster (will refer as Node1) >> > >> > 2.Single node Kafka cluster (will refer as Node2) >> > >> > >> > >> > Node 2 has 1 broker started with a topic (iot.test.stream) and one >> > >> command >> > >> > line producer and one command line consumer to test the kafka >> install. >> > >> > Producer can send messages and the Consumer is receiving it. >> > >> > >> > >> > Node 1 (hadoop cluster) has kafka hadoop consumer code built. Have >> > >> edited >> > >> > the /kafka-0.8/contrib/hadoop-consumer/test/test.properties file >> with >> > >> the >> > >> > following: >> > >> > >> > >> > kafka.etl.topic=iot.test.stream >> > >> > hdfs.default.classpath.dir=/tmp/kafka/lib >> > >> > hadoop.job.ugi=kafka,hadoop >> > >> > kafka.server.uri=tcp://idh251-kafka:9095 >> > >> > input=/tmp/kafka/data >> > >> > output=/tmp/kafka/output >> > >> > kafka.request.limit=-1 >> > >> > ........... >> > >> > >> > >> > I have copied the copy-jars.sh to /tmp/kafka/lib (on HDFS) >> > >> > >> > >> > Next I run the following on Node 1: >> > >> > ./run-class.sh kafka.etl.impl.SimpleKafkaETLJob >> test/test.properties >> > >> from >> > >> > the /kafka-0.8/contrib/hadoop-consumer folder and get a >> > >> > classnotfoundexception for kafka.etl.impl.SimpleKafkaETLJob class. >> > >> > >> > >> > What am I missing? I was thinking that running the sh file would >> allow >> > >> me >> > >> > to retrieve messages with the same topic name to HDFS from Node 2 >> to >> > >> Node >> > >> > 1. I just want to do an end to end test to see that messages coming >> > into >> > >> > Kafka are being stored in HDFS with the minimal amount of code >> change >> > >> > required. >> > >> > >> > >> > Thanks, >> > >> > >> > >> > Abhi >> > >> > >> > >> > -- >> > >> > Abhi Basu >> > >> > >> > >> >> > > >> > >> > >> > -- >> > Abhi Basu >> > >> > > > > -- > Abhi Basu > -- Abhi Basu
