Actually, i've 2 versions of Kafka (0.9 & 0.10) .. btw, i was able to resolve the issue ..
sbt by default considers src/main/scala as default source location, I'd changed the location to a different one. I changed the build.sbt to point to the required location, that fixed the issue regds, Karan Alang On Mon, Jul 10, 2017 at 11:48 AM, David Newberger <da...@phdata.io> wrote: > Karen, > > It looks like the Kafka version is incorrect. You mention Kafka 0.10 > however the classpath references Kafka 0.9 > > Thanks, > > David > > On July 10, 2017 at 1:44:06 PM, karan alang (karan.al...@gmail.com) wrote: > > Hi All, > > I'm running Spark Streaming - Kafka integration using Spark 2.x & Kafka 10. > & seems to be running into issues. > > I compiled the program using sbt, and the compilation went through fine. > I was able able to import this into Eclipse & run the program from Eclipse. > > However, when i run the program using spark-submit, i'm getting the > following error : > > ---------------------------------- > >> $SPARK_HOME/bin/spark-submit --class >> "structuredStreaming.kafka.StructuredKafkaWordCount1" >> --master local[2] /Users/karanalang/Documents/Te >> chnology/Coursera_spark_scala/structuredStreamingKafka/targe >> t/scala-2.11/StructuredStreamingKafka-assembly-1.0.jar > > > >> *java.lang.ClassNotFoundException: >> structuredStreaming.kafka.StructuredKafkaWordCount1* >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >> at java.lang.Class.forName0(Native Method) >> at java.lang.Class.forName(Class.java:348) >> at org.apache.spark.util.Utils$.classForName(Utils.scala:229) >> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy >> $SparkSubmit$$runMain(SparkSubmit.scala:695) >> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit >> .scala:187) >> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) >> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) >> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >> > ------------------------------ > > I've put the jar in the classpath, but i still get the error -> > > echo $CLASSPATH > > .:/Users/karanalang/Documents/Technology/kafka/kafka_2.11-0. >> 9.0.1/lib/jopt-simple-3.2.jar:/Users/karanalang/Documents/Te >> chnology/kafka/kafka_2.11-0.9.0.1/lib/kafka-clients-0.9.0.1. >> jar:/Users/karanalang/Documents/Technology/kafka/kafka_2.11- >> 0.9.0.1/lib/kafka_2.11-0.9.0.1.jar:/Users/karanalang/Documen >> ts/Technology/kafka/kafka_2.11-0.9.0.1/lib/log4j-1.2.17. >> jar:/Users/karanalang/Documents/Technology/kafka/kafka_2.11- >> 0.9.0.1/lib/metrics-core-2.2.0.jar:/Users/karanalang/ >> Documents/Technology/kafka/kafka_2.11-0.9.0.1/lib/scala- >> library-2.11.7.jar:/Users/karanalang/Documents/ >> Technology/kafka/kafka_2.11-0.9.0.1/lib/slf4j-api-1.7.6.jar: >> /Users/karanalang/Documents/Technology/kafka/kafka_2.11-0. >> 9.0.1/lib/slf4j-log4j12-1.7.6.jar:/Users/karanalang/ >> Documents/Technology/kafka/kafka_2.11-0.9.0.1/lib/snappy- >> java-1.1.1.7.jar:/Users/karanalang/Documents/ >> Technology/kafka/kafka_2.11-0.9.0.1/lib/zkclient-0.7.jar:/ >> Users/karanalang/Documents/Technology/kafka/kafka_2.11-0. >> 9.0.1/lib/zookeeper-3.4.6.jar:/Users/karanalang/Documents/ >> Technology/ApacheSpark-v2.1/spark-2.1.0-bin-hadoop2.7/ >> jars/*.jar:/Users/karanalang/Documents/Technology/kafka/ >> mirrormaker_topic_rename-master/target/mmchangetopic-1. >> 0-SNAPSHOT.jar:/Users/karanalang/Documents/Technology/ >> *Coursera_spark_scala/structuredStreamingKafka/target/scala-2.11/* >> *StructuredStreamingKafka-assembly-1.0.jar* > > > When i look inside the jar - *StructuredStreamingKafka-assembly-1.0.jar, > i don't see the file "*StructuredKafkaWordCount1.class" > > Attaching my build.sbt. > > Any ideas on what i need to do ? > > > > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
