Hi All, I'm running Spark Streaming - Kafka integration using Spark 2.x & Kafka 10. & seems to be running into issues.
I compiled the program using sbt, and the compilation went through fine. I was able able to import this into Eclipse & run the program from Eclipse. However, when i run the program using spark-submit, i'm getting the following error : ---------------------------------- > $SPARK_HOME/bin/spark-submit --class > "structuredStreaming.kafka.StructuredKafkaWordCount1" > --master local[2] /Users/karanalang/Documents/Te > chnology/Coursera_spark_scala/structuredStreamingKafka/targe > t/scala-2.11/StructuredStreamingKafka-assembly-1.0.jar > *java.lang.ClassNotFoundException: > structuredStreaming.kafka.StructuredKafkaWordCount1* > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > at org.apache.spark.util.Utils$.classForName(Utils.scala:229) > at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy > $SparkSubmit$$runMain(SparkSubmit.scala:695) > at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > ------------------------------ I've put the jar in the classpath, but i still get the error -> echo $CLASSPATH .:/Users/karanalang/Documents/Technology/kafka/kafka_2.11-0. > 9.0.1/lib/jopt-simple-3.2.jar:/Users/karanalang/Documents/Te > chnology/kafka/kafka_2.11-0.9.0.1/lib/kafka-clients-0.9.0.1. > jar:/Users/karanalang/Documents/Technology/kafka/kafka_2.11- > 0.9.0.1/lib/kafka_2.11-0.9.0.1.jar:/Users/karanalang/ > Documents/Technology/kafka/kafka_2.11-0.9.0.1/lib/log4j- > 1.2.17.jar:/Users/karanalang/Documents/Technology/kafka/ > kafka_2.11-0.9.0.1/lib/metrics-core-2.2.0.jar:/Users/karanalang/Documents/ > Technology/kafka/kafka_2.11-0.9.0.1/lib/scala-library-2.11. > 7.jar:/Users/karanalang/Documents/Technology/kafka/ > kafka_2.11-0.9.0.1/lib/slf4j-api-1.7.6.jar:/Users/ > karanalang/Documents/Technology/kafka/kafka_2.11-0.9.0.1/ > lib/slf4j-log4j12-1.7.6.jar:/Users/karanalang/Documents/ > Technology/kafka/kafka_2.11-0.9.0.1/lib/snappy-java-1.1.1.7. > jar:/Users/karanalang/Documents/Technology/kafka/kafka_2.11-0.9.0.1/lib/ > zkclient-0.7.jar:/Users/karanalang/Documents/Technolog > y/kafka/kafka_2.11-0.9.0.1/lib/zookeeper-3.4.6.jar:/ > Users/karanalang/Documents/Technology/ApacheSpark-v2.1/spark > -2.1.0-bin-hadoop2.7/jars/*.jar:/Users/karanalang/Document > s/Technology/kafka/mirrormaker_topic_rename-master/target/ > mmchangetopic-1.0-SNAPSHOT.jar:/Users/karanalang/Documents/Technology/ > *Coursera_spark_scala/structuredStreamingKafka/target/scala-2.11/* > *StructuredStreamingKafka-assembly-1.0.jar* When i look inside the jar - *StructuredStreamingKafka-assembly-1.0.jar, i don't see the file "*StructuredKafkaWordCount1.class" Attaching my build.sbt. Any ideas on what i need to do ?
build.sbt
Description: Binary data
--------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
