Make sure you verify the following: - Scala version : I think the correct version would be 2.10.x - SparkMasterURL: Be sure that you copied the one displayed on the webui's top left corner (running on port 8080)
Thanks Best Regards On Mon, Dec 29, 2014 at 12:26 PM, suhshekar52 <suhsheka...@gmail.com> wrote: > Hello Everyone, > > Thank you for the time and the help :). > > My goal here is to get this program working: > > https://github.com/apache/spark/blob/master/examples/scala-2.10/src/main/java/org/apache/spark/examples/streaming/JavaKafkaWordCount.java > > The only lines I do not have from the example are lines 62-67. pom.xml > <http://apache-spark-user-list.1001560.n3.nabble.com/file/n20879/pom.xml> > > Background: Have ec2 instances running. The standalone spark is running on > top of Cloudera Manager 5.2. > > Pom file is attached and the same for both clusters. > pom.xml > <http://apache-spark-user-list.1001560.n3.nabble.com/file/n20879/pom.xml> > > Here are a few different approaches I have taken and the issues I run into: > > *Standalone Mode* > > 1) Use spark-submit script to run: > > /opt/cloudera/parcels/CDH-5.2.1-1.cdh5.2.1.p0.12/lib/spark/bin/spark-submit > --class SimpleApp --master spark://10.0.1.230:7077 --jars $(echo > /home/ec2-user/sparkApps/SimpleApp/lib/*.jar | tr ' ' ',') > /home/ec2-user/sparkApps/SimpleApp/target/simple-project-1.0.jar > > Interesting...I was getting an error like this: Initial job has not > accepted > any resources; check your cluster UI > > Now, when I run, it prints out the 3 Hello world statements in my code: > KafkaJavaConsumer.txt > < > http://apache-spark-user-list.1001560.n3.nabble.com/file/n20879/KafkaJavaConsumer.txt > > > > and then it seems to try to start the Kafka Stream, but fails: > > 14/12/29 05:58:05 INFO KafkaReceiver: Starting Kafka Consumer Stream with > group: c1 > 14/12/29 05:58:05 INFO ReceiverTracker: Registered receiver for stream 0 > from akka://sparkDriver > 14/12/29 05:58:05 INFO KafkaReceiver: Connecting to Zookeeper: > 10.0.1.232:2181 > 14/12/29 05:58:05 INFO BlockGenerator: Started block pushing thread > 14/12/29 05:58:05 INFO ReceiverSupervisorImpl: Stopping receiver with > message: Error starting receiver 0: java.lang.NoClassDefFoundError: > scala/reflect/ClassManifest > 14/12/29 05:58:05 INFO ReceiverSupervisorImpl: Called receiver onStop > 14/12/29 05:58:05 INFO ReceiverSupervisorImpl: Deregistering receiver 0 > ^C14/12/29 05:58:05 ERROR ReceiverTracker: Deregistered receiver for stream > 0: Error starting receiver 0 - java.lang.NoClassDefFoundError: > scala/reflect/ClassManifest > at kafka.utils.Log4jController$.<init>(Log4jController.scala:29) > at kafka.utils.Log4jController$.<clinit>(Log4jController.scala) > at kafka.utils.Logging$class.$init$(Logging.scala:29) > at > kafka.utils.VerifiableProperties.<init>(VerifiableProperties.scala:26) > at kafka.consumer.ConsumerConfig.<init>(ConsumerConfig.scala:94) > at > > org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:96) > at > > org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:121) > at > > org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:106) > at > > org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:264) > at > > org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:257) > at > > org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121) > at > > org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121) > at > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62) > at org.apache.spark.scheduler.Task.run(Task.scala:54) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:180) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.lang.ClassNotFoundException: scala.reflect.ClassManifest > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:423) > at java.lang.ClassLoader.loadClass(ClassLoader.java:356) > ... 18 more > > 14/12/29 05:58:05 INFO ReceiverSupervisorImpl: Stopped receiver 0 > 14/12/29 05:58:05 INFO BlockGenerator: Stopping BlockGenerator > > I ran into a couple other Class not found errors, and was able to solve > them > by adding dependencies on the pom file, but have not found such a solution > to this error. > > On the Kafka side of things, I am simply typing in messages as soon as I > start the Java app on another console. Is this okay? > > I have not set up an advertised host on the kafka side as I was able to > still receive messages from other consoles by setting up a consumer to > listen to the private ip:port. Is this okay? > > Lastly, is there command, like --from-beginning for a consumer in the java > application to get messages from the beginning? > > Thanks a lot for the help and happy holidays! > > > > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Setting-up-Simple-Kafka-Consumer-via-Spark-Java-app-tp20879.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
