Just looked at the pom file that you are using, why are you having different versions in it?
<dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming-kafka_2.10</artifactId> <version>*1.1.1*</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming_2.10</artifactId> <version>*1.0.2*</version> </dependency> can you make both the versions the same? Thanks Best Regards On Mon, Dec 29, 2014 at 12:44 PM, Suhas Shekar <suhsheka...@gmail.com> wrote: > 1) Could you please clarify on what you mean by checking the Scala version > is correct? In my pom.xml file it is 2.10.4 (which is the same as when I > start spark-shell). > > 2) The spark master URL is definitely correct as I have run other apps > with the same script that use Spark (like a word count with a local file) > > Thank you for the help! > > > > > Suhas Shekar > > University of California, Los Angeles > B.A. Economics, Specialization in Computing 2014 > > On Sun, Dec 28, 2014 at 11:04 PM, Akhil Das <ak...@sigmoidanalytics.com> > wrote: > >> Make sure you verify the following: >> >> - Scala version : I think the correct version would be 2.10.x >> - SparkMasterURL: Be sure that you copied the one displayed on the >> webui's top left corner (running on port 8080) >> >> Thanks >> Best Regards >> >> On Mon, Dec 29, 2014 at 12:26 PM, suhshekar52 <suhsheka...@gmail.com> >> wrote: >> >>> Hello Everyone, >>> >>> Thank you for the time and the help :). >>> >>> My goal here is to get this program working: >>> >>> https://github.com/apache/spark/blob/master/examples/scala-2.10/src/main/java/org/apache/spark/examples/streaming/JavaKafkaWordCount.java >>> >>> The only lines I do not have from the example are lines 62-67. pom.xml >>> <http://apache-spark-user-list.1001560.n3.nabble.com/file/n20879/pom.xml >>> > >>> >>> Background: Have ec2 instances running. The standalone spark is running >>> on >>> top of Cloudera Manager 5.2. >>> >>> Pom file is attached and the same for both clusters. >>> pom.xml >>> <http://apache-spark-user-list.1001560.n3.nabble.com/file/n20879/pom.xml >>> > >>> >>> Here are a few different approaches I have taken and the issues I run >>> into: >>> >>> *Standalone Mode* >>> >>> 1) Use spark-submit script to run: >>> >>> >>> /opt/cloudera/parcels/CDH-5.2.1-1.cdh5.2.1.p0.12/lib/spark/bin/spark-submit >>> --class SimpleApp --master spark://10.0.1.230:7077 --jars $(echo >>> /home/ec2-user/sparkApps/SimpleApp/lib/*.jar | tr ' ' ',') >>> /home/ec2-user/sparkApps/SimpleApp/target/simple-project-1.0.jar >>> >>> Interesting...I was getting an error like this: Initial job has not >>> accepted >>> any resources; check your cluster UI >>> >>> Now, when I run, it prints out the 3 Hello world statements in my code: >>> KafkaJavaConsumer.txt >>> < >>> http://apache-spark-user-list.1001560.n3.nabble.com/file/n20879/KafkaJavaConsumer.txt >>> > >>> >>> and then it seems to try to start the Kafka Stream, but fails: >>> >>> 14/12/29 05:58:05 INFO KafkaReceiver: Starting Kafka Consumer Stream with >>> group: c1 >>> 14/12/29 05:58:05 INFO ReceiverTracker: Registered receiver for stream 0 >>> from akka://sparkDriver >>> 14/12/29 05:58:05 INFO KafkaReceiver: Connecting to Zookeeper: >>> 10.0.1.232:2181 >>> 14/12/29 05:58:05 INFO BlockGenerator: Started block pushing thread >>> 14/12/29 05:58:05 INFO ReceiverSupervisorImpl: Stopping receiver with >>> message: Error starting receiver 0: java.lang.NoClassDefFoundError: >>> scala/reflect/ClassManifest >>> 14/12/29 05:58:05 INFO ReceiverSupervisorImpl: Called receiver onStop >>> 14/12/29 05:58:05 INFO ReceiverSupervisorImpl: Deregistering receiver 0 >>> ^C14/12/29 05:58:05 ERROR ReceiverTracker: Deregistered receiver for >>> stream >>> 0: Error starting receiver 0 - java.lang.NoClassDefFoundError: >>> scala/reflect/ClassManifest >>> at kafka.utils.Log4jController$.<init>(Log4jController.scala:29) >>> at kafka.utils.Log4jController$.<clinit>(Log4jController.scala) >>> at kafka.utils.Logging$class.$init$(Logging.scala:29) >>> at >>> kafka.utils.VerifiableProperties.<init>(VerifiableProperties.scala:26) >>> at kafka.consumer.ConsumerConfig.<init>(ConsumerConfig.scala:94) >>> at >>> >>> org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:96) >>> at >>> >>> org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:121) >>> at >>> >>> org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:106) >>> at >>> >>> org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:264) >>> at >>> >>> org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:257) >>> at >>> >>> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121) >>> at >>> >>> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121) >>> at >>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62) >>> at org.apache.spark.scheduler.Task.run(Task.scala:54) >>> at >>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:180) >>> at >>> >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >>> at >>> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >>> at java.lang.Thread.run(Thread.java:722) >>> Caused by: java.lang.ClassNotFoundException: scala.reflect.ClassManifest >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:423) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:356) >>> ... 18 more >>> >>> 14/12/29 05:58:05 INFO ReceiverSupervisorImpl: Stopped receiver 0 >>> 14/12/29 05:58:05 INFO BlockGenerator: Stopping BlockGenerator >>> >>> I ran into a couple other Class not found errors, and was able to solve >>> them >>> by adding dependencies on the pom file, but have not found such a >>> solution >>> to this error. >>> >>> On the Kafka side of things, I am simply typing in messages as soon as I >>> start the Java app on another console. Is this okay? >>> >>> I have not set up an advertised host on the kafka side as I was able to >>> still receive messages from other consoles by setting up a consumer to >>> listen to the private ip:port. Is this okay? >>> >>> Lastly, is there command, like --from-beginning for a consumer in the >>> java >>> application to get messages from the beginning? >>> >>> Thanks a lot for the help and happy holidays! >>> >>> >>> >>> >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/Setting-up-Simple-Kafka-Consumer-via-Spark-Java-app-tp20879.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >>> >>> >> >
