Hi Cody, I tried with the build file you provided, but it's not working for me, getting same error: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/streaming/kafka/KafkaUtils$
I am not getting this error while building (sbt package). I am getting this error when i am running my spark-streaming program. Do i need to specify kafka jar path manually with spark-submit --jars flag? My build.sbt: name := "NetworkStreaming" libraryDependencies += "org.apache.hbase" % "hbase" % "0.92.1" libraryDependencies += "org.apache.hadoop" % "hadoop-core" % "1.0.2" libraryDependencies += "org.apache.spark" % "spark-mllib_2.10" % "1.0.0" libraryDependencies ++= Seq( "org.apache.spark" % "spark-streaming_2.10" % "1.5.2", "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.5.2" ) Regards, ~Vinti On Wed, Feb 24, 2016 at 9:33 AM, Cody Koeninger <c...@koeninger.org> wrote: > spark streaming is provided, kafka is not. > > This build file > > https://github.com/koeninger/kafka-exactly-once/blob/master/build.sbt > > includes some hacks for ivy issues that may no longer be strictly > necessary, but try that build and see if it works for you. > > > On Wed, Feb 24, 2016 at 11:14 AM, Vinti Maheshwari <vinti.u...@gmail.com> > wrote: > >> Hello, >> >> I have tried multiple different settings in build.sbt but seems like >> nothing is working. >> Can anyone suggest the right syntax/way to include kafka with spark? >> >> Error >> Exception in thread "main" java.lang.NoClassDefFoundError: >> org/apache/spark/streaming/kafka/KafkaUtils$ >> >> build.sbt >> libraryDependencies += "org.apache.hbase" % "hbase" % "0.92.1" >> libraryDependencies += "org.apache.hadoop" % "hadoop-core" % "1.0.2" >> libraryDependencies += "org.apache.spark" % "spark-mllib_2.10" % "1.0.0" >> libraryDependencies ++= Seq( >> "org.apache.spark" % "spark-streaming_2.10" % "1.5.2", >> "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.5.2", >> "org.apache.spark" %% "spark-streaming" % "1.5.2" % "provided", >> "org.apache.spark" %% "spark-streaming-kafka" % "1.5.2" % "provided" >> ) >> >> >> Thanks, >> Vinti >> >> >
