I am not using sbt assembly currently. I need to check how to use sbt assembly.
Regards, ~Vinti On Wed, Feb 24, 2016 at 11:10 AM, Cody Koeninger <c...@koeninger.org> wrote: > Are you using sbt assembly? That's what will include all of the > non-provided dependencies in a single jar along with your code. Otherwise > you'd have to specify each separate jar in your spark-submit line, which is > a pain. > > On Wed, Feb 24, 2016 at 12:49 PM, Vinti Maheshwari <vinti.u...@gmail.com> > wrote: > >> Hi Cody, >> >> I tried with the build file you provided, but it's not working for me, >> getting same error: >> Exception in thread "main" java.lang.NoClassDefFoundError: >> org/apache/spark/streaming/kafka/KafkaUtils$ >> >> I am not getting this error while building (sbt package). I am getting >> this error when i am running my spark-streaming program. >> Do i need to specify kafka jar path manually with spark-submit --jars >> flag? >> >> My build.sbt: >> >> name := "NetworkStreaming" >> libraryDependencies += "org.apache.hbase" % "hbase" % "0.92.1" >> >> libraryDependencies += "org.apache.hadoop" % "hadoop-core" % "1.0.2" >> >> libraryDependencies += "org.apache.spark" % "spark-mllib_2.10" % "1.0.0" >> >> libraryDependencies ++= Seq( >> "org.apache.spark" % "spark-streaming_2.10" % "1.5.2", >> "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.5.2" >> ) >> >> >> >> Regards, >> ~Vinti >> >> On Wed, Feb 24, 2016 at 9:33 AM, Cody Koeninger <c...@koeninger.org> >> wrote: >> >>> spark streaming is provided, kafka is not. >>> >>> This build file >>> >>> https://github.com/koeninger/kafka-exactly-once/blob/master/build.sbt >>> >>> includes some hacks for ivy issues that may no longer be strictly >>> necessary, but try that build and see if it works for you. >>> >>> >>> On Wed, Feb 24, 2016 at 11:14 AM, Vinti Maheshwari <vinti.u...@gmail.com >>> > wrote: >>> >>>> Hello, >>>> >>>> I have tried multiple different settings in build.sbt but seems like >>>> nothing is working. >>>> Can anyone suggest the right syntax/way to include kafka with spark? >>>> >>>> Error >>>> Exception in thread "main" java.lang.NoClassDefFoundError: >>>> org/apache/spark/streaming/kafka/KafkaUtils$ >>>> >>>> build.sbt >>>> libraryDependencies += "org.apache.hbase" % "hbase" % "0.92.1" >>>> libraryDependencies += "org.apache.hadoop" % "hadoop-core" % "1.0.2" >>>> libraryDependencies += "org.apache.spark" % "spark-mllib_2.10" % "1.0.0" >>>> libraryDependencies ++= Seq( >>>> "org.apache.spark" % "spark-streaming_2.10" % "1.5.2", >>>> "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.5.2", >>>> "org.apache.spark" %% "spark-streaming" % "1.5.2" % "provided", >>>> "org.apache.spark" %% "spark-streaming-kafka" % "1.5.2" % "provided" >>>> ) >>>> >>>> >>>> Thanks, >>>> Vinti >>>> >>>> >>> >> >
