Re: error in running StructuredStreaming-Kafka integration code (Spark 2.x & Kafka 10)
Actually, i've 2 versions of Kafka (0.9 & 0.10) .. btw, i was able to resolve the issue .. sbt by default considers src/main/scala as default source location, I'd changed the location to a different one. I changed the build.sbt to point to the required location, that fixed the issue regds, Karan Alang On Mon, Jul 10, 2017 at 11:48 AM, David Newberger wrote: > Karen, > > It looks like the Kafka version is incorrect. You mention Kafka 0.10 > however the classpath references Kafka 0.9 > > Thanks, > > David > > On July 10, 2017 at 1:44:06 PM, karan alang (karan.al...@gmail.com) wrote: > > Hi All, > > I'm running Spark Streaming - Kafka integration using Spark 2.x & Kafka 10. > & seems to be running into issues. > > I compiled the program using sbt, and the compilation went through fine. > I was able able to import this into Eclipse & run the program from Eclipse. > > However, when i run the program using spark-submit, i'm getting the > following error : > > -- > >> $SPARK_HOME/bin/spark-submit --class >> "structuredStreaming.kafka.StructuredKafkaWordCount1" >> --master local[2] /Users/karanalang/Documents/Te >> chnology/Coursera_spark_scala/structuredStreamingKafka/targe >> t/scala-2.11/StructuredStreamingKafka-assembly-1.0.jar > > > >> *java.lang.ClassNotFoundException: >> structuredStreaming.kafka.StructuredKafkaWordCount1* >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >> at java.lang.Class.forName0(Native Method) >> at java.lang.Class.forName(Class.java:348) >> at org.apache.spark.util.Utils$.classForName(Utils.scala:229) >> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy >> $SparkSubmit$$runMain(SparkSubmit.scala:695) >> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit >> .scala:187) >> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) >> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) >> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >> > -- > > I've put the jar in the classpath, but i still get the error -> > > echo $CLASSPATH > > .:/Users/karanalang/Documents/Technology/kafka/kafka_2.11-0. >> 9.0.1/lib/jopt-simple-3.2.jar:/Users/karanalang/Documents/Te >> chnology/kafka/kafka_2.11-0.9.0.1/lib/kafka-clients-0.9.0.1. >> jar:/Users/karanalang/Documents/Technology/kafka/kafka_2.11- >> 0.9.0.1/lib/kafka_2.11-0.9.0.1.jar:/Users/karanalang/Documen >> ts/Technology/kafka/kafka_2.11-0.9.0.1/lib/log4j-1.2.17. >> jar:/Users/karanalang/Documents/Technology/kafka/kafka_2.11- >> 0.9.0.1/lib/metrics-core-2.2.0.jar:/Users/karanalang/ >> Documents/Technology/kafka/kafka_2.11-0.9.0.1/lib/scala- >> library-2.11.7.jar:/Users/karanalang/Documents/ >> Technology/kafka/kafka_2.11-0.9.0.1/lib/slf4j-api-1.7.6.jar: >> /Users/karanalang/Documents/Technology/kafka/kafka_2.11-0. >> 9.0.1/lib/slf4j-log4j12-1.7.6.jar:/Users/karanalang/ >> Documents/Technology/kafka/kafka_2.11-0.9.0.1/lib/snappy- >> java-1.1.1.7.jar:/Users/karanalang/Documents/ >> Technology/kafka/kafka_2.11-0.9.0.1/lib/zkclient-0.7.jar:/ >> Users/karanalang/Documents/Technology/kafka/kafka_2.11-0. >> 9.0.1/lib/zookeeper-3.4.6.jar:/Users/karanalang/Documents/ >> Technology/ApacheSpark-v2.1/spark-2.1.0-bin-hadoop2.7/ >> jars/*.jar:/Users/karanalang/Documents/Technology/kafka/ >> mirrormaker_topic_rename-master/target/mmchangetopic-1. >> 0-SNAPSHOT.jar:/Users/karanalang/Documents/Technology/ >> *Coursera_spark_scala/structuredStreamingKafka/target/scala-2.11/* >> *StructuredStreamingKafka-assembly-1.0.jar* > > > When i look inside the jar - *StructuredStreamingKafka-assembly-1.0.jar, > i don't see the file "*StructuredKafkaWordCount1.class" > > Attaching my build.sbt. > > Any ideas on what i need to do ? > > > > > > > > > > > > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
Re: error in running StructuredStreaming-Kafka integration code (Spark 2.x & Kafka 10)
Karen, It looks like the Kafka version is incorrect. You mention Kafka 0.10 however the classpath references Kafka 0.9 Thanks, David On July 10, 2017 at 1:44:06 PM, karan alang (karan.al...@gmail.com) wrote: Hi All, I'm running Spark Streaming - Kafka integration using Spark 2.x & Kafka 10. & seems to be running into issues. I compiled the program using sbt, and the compilation went through fine. I was able able to import this into Eclipse & run the program from Eclipse. However, when i run the program using spark-submit, i'm getting the following error : -- > $SPARK_HOME/bin/spark-submit --class > "structuredStreaming.kafka.StructuredKafkaWordCount1" > --master local[2] /Users/karanalang/Documents/Te > chnology/Coursera_spark_scala/structuredStreamingKafka/targe > t/scala-2.11/StructuredStreamingKafka-assembly-1.0.jar > *java.lang.ClassNotFoundException: > structuredStreaming.kafka.StructuredKafkaWordCount1* > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > at org.apache.spark.util.Utils$.classForName(Utils.scala:229) > at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy > $SparkSubmit$$runMain(SparkSubmit.scala:695) > at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > -- I've put the jar in the classpath, but i still get the error -> echo $CLASSPATH .:/Users/karanalang/Documents/Technology/kafka/kafka_2.11-0. > 9.0.1/lib/jopt-simple-3.2.jar:/Users/karanalang/Documents/Te > chnology/kafka/kafka_2.11-0.9.0.1/lib/kafka-clients-0.9.0.1. > jar:/Users/karanalang/Documents/Technology/kafka/kafka_2.11- > 0.9.0.1/lib/kafka_2.11-0.9.0.1.jar:/Users/karanalang/ > Documents/Technology/kafka/kafka_2.11-0.9.0.1/lib/log4j- > 1.2.17.jar:/Users/karanalang/Documents/Technology/kafka/ > kafka_2.11-0.9.0.1/lib/metrics-core-2.2.0.jar:/Users/karanalang/Documents/ > Technology/kafka/kafka_2.11-0.9.0.1/lib/scala-library-2.11. > 7.jar:/Users/karanalang/Documents/Technology/kafka/ > kafka_2.11-0.9.0.1/lib/slf4j-api-1.7.6.jar:/Users/ > karanalang/Documents/Technology/kafka/kafka_2.11-0.9.0.1/ > lib/slf4j-log4j12-1.7.6.jar:/Users/karanalang/Documents/ > Technology/kafka/kafka_2.11-0.9.0.1/lib/snappy-java-1.1.1.7. > jar:/Users/karanalang/Documents/Technology/kafka/kafka_2.11-0.9.0.1/lib/ > zkclient-0.7.jar:/Users/karanalang/Documents/Technolog > y/kafka/kafka_2.11-0.9.0.1/lib/zookeeper-3.4.6.jar:/ > Users/karanalang/Documents/Technology/ApacheSpark-v2.1/spark > -2.1.0-bin-hadoop2.7/jars/*.jar:/Users/karanalang/Document > s/Technology/kafka/mirrormaker_topic_rename-master/target/ > mmchangetopic-1.0-SNAPSHOT.jar:/Users/karanalang/Documents/Technology/ > *Coursera_spark_scala/structuredStreamingKafka/target/scala-2.11/* > *StructuredStreamingKafka-assembly-1.0.jar* When i look inside the jar - *StructuredStreamingKafka-assembly-1.0.jar, i don't see the file "*StructuredKafkaWordCount1.class" Attaching my build.sbt. Any ideas on what i need to do ? - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
error in running StructuredStreaming-Kafka integration code (Spark 2.x & Kafka 10)
Hi All, I'm running Spark Streaming - Kafka integration using Spark 2.x & Kafka 10. & seems to be running into issues. I compiled the program using sbt, and the compilation went through fine. I was able able to import this into Eclipse & run the program from Eclipse. However, when i run the program using spark-submit, i'm getting the following error : -- > $SPARK_HOME/bin/spark-submit --class > "structuredStreaming.kafka.StructuredKafkaWordCount1" > --master local[2] /Users/karanalang/Documents/Te > chnology/Coursera_spark_scala/structuredStreamingKafka/targe > t/scala-2.11/StructuredStreamingKafka-assembly-1.0.jar > *java.lang.ClassNotFoundException: > structuredStreaming.kafka.StructuredKafkaWordCount1* > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > at org.apache.spark.util.Utils$.classForName(Utils.scala:229) > at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy > $SparkSubmit$$runMain(SparkSubmit.scala:695) > at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > -- I've put the jar in the classpath, but i still get the error -> echo $CLASSPATH .:/Users/karanalang/Documents/Technology/kafka/kafka_2.11-0. > 9.0.1/lib/jopt-simple-3.2.jar:/Users/karanalang/Documents/Te > chnology/kafka/kafka_2.11-0.9.0.1/lib/kafka-clients-0.9.0.1. > jar:/Users/karanalang/Documents/Technology/kafka/kafka_2.11- > 0.9.0.1/lib/kafka_2.11-0.9.0.1.jar:/Users/karanalang/ > Documents/Technology/kafka/kafka_2.11-0.9.0.1/lib/log4j- > 1.2.17.jar:/Users/karanalang/Documents/Technology/kafka/ > kafka_2.11-0.9.0.1/lib/metrics-core-2.2.0.jar:/Users/karanalang/Documents/ > Technology/kafka/kafka_2.11-0.9.0.1/lib/scala-library-2.11. > 7.jar:/Users/karanalang/Documents/Technology/kafka/ > kafka_2.11-0.9.0.1/lib/slf4j-api-1.7.6.jar:/Users/ > karanalang/Documents/Technology/kafka/kafka_2.11-0.9.0.1/ > lib/slf4j-log4j12-1.7.6.jar:/Users/karanalang/Documents/ > Technology/kafka/kafka_2.11-0.9.0.1/lib/snappy-java-1.1.1.7. > jar:/Users/karanalang/Documents/Technology/kafka/kafka_2.11-0.9.0.1/lib/ > zkclient-0.7.jar:/Users/karanalang/Documents/Technolog > y/kafka/kafka_2.11-0.9.0.1/lib/zookeeper-3.4.6.jar:/ > Users/karanalang/Documents/Technology/ApacheSpark-v2.1/spark > -2.1.0-bin-hadoop2.7/jars/*.jar:/Users/karanalang/Document > s/Technology/kafka/mirrormaker_topic_rename-master/target/ > mmchangetopic-1.0-SNAPSHOT.jar:/Users/karanalang/Documents/Technology/ > *Coursera_spark_scala/structuredStreamingKafka/target/scala-2.11/* > *StructuredStreamingKafka-assembly-1.0.jar* When i look inside the jar - *StructuredStreamingKafka-assembly-1.0.jar, i don't see the file "*StructuredKafkaWordCount1.class" Attaching my build.sbt. Any ideas on what i need to do ? build.sbt Description: Binary data - To unsubscribe e-mail: user-unsubscr...@spark.apache.org