Thanks, i was able to get this working. here is what i added in build.sbt file ----------------------------------------------------------------------------------------------
scalaVersion := "2.11.7" val sparkVers = "2.1.0" // Base Spark-provided dependencies libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % sparkVers % "provided", "org.apache.spark" %% "spark-streaming" % sparkVers % "provided", "org.apache.spark" % "spark-streaming-kafka-0-8_2.11" % sparkVers) ------------------------------------------------------------------------------------------------ However, i'm running into addition issue when compiling the file using sbt .. it gives errors as shown below : *Pls note - I've added the jars in eclipse, and it works .. however when i use sbt to compile, it is failing.* *What needs to be done ? * *I've also tried added the jars in CLASSPATH, but still get the same error. * ------------------------------------------------------------------------------------------------- [info] Compiling 1 Scala source to /Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/target/scala-2.11/classes... [error] /Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:6: object SQLContext is not a member of package org.apache.spark.sql [error] import org.apache.spark.sql.SQLContext [error] ^ [error] /Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:10: object SparkSession is not a member of package org.apache.spark.sql [error] import org.apache.spark.sql.SparkSession [error] ^ [error] /Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:11: object types is not a member of package org.apache.spark.sql [error] import org.apache.spark.sql.types.StructField [error] ^ [error] /Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:12: object types is not a member of package org.apache.spark.sql [error] import org.apache.spark.sql.types.StringType [error] ^ [error] /Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:13: object types is not a member of package org.apache.spark.sql [error] import org.apache.spark.sql.types.StructType [error] ^ [error] /Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:14: object Row is not a member of package org.apache.spark.sql [error] import org.apache.spark.sql.Row [error] ^ On Sat, Jun 17, 2017 at 12:35 AM, Jozef.koval <jozef.ko...@protonmail.ch> wrote: > Hi Karan, > > spark-streaming-kafka is for old spark (version < 1.6.3) > spark-streaming-kafka-0.8 is for current spark (version > 2.0) > > Jozef > > n.b. there is also version for kafka 0.10+ see [this]( > https://spark.apache.org/docs/latest/streaming-kafka-integration.html) > > Sent from [ProtonMail](https://protonmail.ch), encrypted email based in > Switzerland. > > -------- Original Message -------- > Subject: Kafka-Spark Integration - build failing with sbt > Local Time: June 17, 2017 1:50 AM > UTC Time: June 16, 2017 11:50 PM > From: karan.al...@gmail.com > To: users@kafka.apache.org > > I"m trying to compile kafka & Spark Streaming integration code i.e. reading > from Kafka using Spark Streaming, > and the sbt build is failing with error - > > [error] (*:update) sbt.ResolveException: unresolved dependency: > org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found > > Scala version -> 2.10.7 > Spark Version -> 2.1.0 > Kafka version -> 0.9 > sbt version -> 0.13 > > Contents of sbt files is as shown below -> > > 1) > vi spark_kafka_code/project/plugins.sbt > > addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2") > > 2) > vi spark_kafka_code/sparkkafka.sbt > > import AssemblyKeys._ > assemblySettings > > name := "SparkKafka Project" > > version := "1.0" > scalaVersion := "2.11.7" > > val sparkVers = "2.1.0" > > // Base Spark-provided dependencies > libraryDependencies ++= Seq( > "org.apache.spark" %% "spark-core" % sparkVers % "provided", > "org.apache.spark" %% "spark-streaming" % sparkVers % "provided", > "org.apache.spark" %% "spark-streaming-kafka" % sparkVers) > > mergeStrategy in assembly := { > case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard > case m if m.toLowerCase.startsWith("META-INF") => MergeStrategy.discard > case "reference.conf" => MergeStrategy.concat > case m if m.endsWith("UnusedStubClass.class") => MergeStrategy.discard > case _ => MergeStrategy.first > } > > i launch sbt, and then try to create an eclipse project, complete error > is as shown below - > > --------------------- > > sbt > [info] Loading global plugins from /Users/karanalang/.sbt/0.13/plugins > [info] Loading project definition from > /Users/karanalang/Documents/Technology/Coursera_spark_ > scala/spark_kafka_code/project > [info] Set current project to SparkKafka Project (in build > file:/Users/karanalang/Documents/Technology/Coursera_ > spark_scala/spark_kafka_code/) > > eclipse > [info] About to create Eclipse project files for your project(s). > [info] Updating > {file:/Users/karanalang/Documents/Technology/Coursera_ > spark_scala/spark_kafka_code/}spark_kafka_code... > [info] Resolving org.apache.spark#spark-streaming-kafka_2.11;2.1.0 ... > [warn] module not found: > org.apache.spark#spark-streaming-kafka_2.11;2.1.0 > [warn] ==== local: tried > [warn] > /Users/karanalang/.ivy2/local/org.apache.spark/spark- > streaming-kafka_2.11/2.1.0/ivys/ivy.xml > [warn] ==== activator-launcher-local: tried > [warn] > /Users/karanalang/.activator/repository/org.apache.spark/ > spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml > [warn] ==== activator-local: tried > [warn] > /Users/karanalang/Documents/Technology/SCALA/activator- > dist-1.3.10/repository/org.apache.spark/spark-streaming- > kafka_2.11/2.1.0/ivys/ivy.xml > [warn] ==== public: tried > [warn] > https://repo1.maven.org/maven2/org/apache/spark/spark- > streaming-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom > [warn] ==== typesafe-releases: tried > [warn] > http://repo.typesafe.com/typesafe/releases/org/apache/ > spark/spark-streaming-kafka_2.11/2.1.0/spark-streaming- > kafka_2.11-2.1.0.pom > [warn] ==== typesafe-ivy-releasez: tried > [warn] > http://repo.typesafe.com/typesafe/ivy-releases/org. > apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml > [info] Resolving jline#jline;2.12.1 ... > [warn] :::::::::::::::::::::::::::::::::::::::::::::: > [warn] :: UNRESOLVED DEPENDENCIES :: > [warn] :::::::::::::::::::::::::::::::::::::::::::::: > [warn] :: org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found > [warn] :::::::::::::::::::::::::::::::::::::::::::::: > [warn] > [warn] Note: Unresolved dependencies path: > [warn] org.apache.spark:spark-streaming-kafka_2.11:2.1.0 > (/Users/karanalang/Documents/Technology/Coursera_spark_ > scala/spark_kafka_code/sparkkafka.sbt#L12-16) > [warn] +- sparkkafka-project:sparkkafka-project_2.11:1.0 > [trace] Stack trace suppressed: run last *:update for the full output. > [error] (*:update) sbt.ResolveException: unresolved dependency: > org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found > [info] Updating > {file:/Users/karanalang/Documents/Technology/Coursera_ > spark_scala/spark_kafka_code/}spark_kafka_code... > [info] Resolving org.apache.spark#spark-streaming-kafka_2.11;2.1.0 ... > [warn] module not found: > org.apache.spark#spark-streaming-kafka_2.11;2.1.0 > [warn] ==== local: tried > [warn] > /Users/karanalang/.ivy2/local/org.apache.spark/spark- > streaming-kafka_2.11/2.1.0/ivys/ivy.xml > [warn] ==== activator-launcher-local: tried > [warn] > /Users/karanalang/.activator/repository/org.apache.spark/ > spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml > [warn] ==== activator-local: tried > [warn] > /Users/karanalang/Documents/Technology/SCALA/activator- > dist-1.3.10/repository/org.apache.spark/spark-streaming- > kafka_2.11/2.1.0/ivys/ivy.xml > [warn] ==== public: tried > [warn] > https://repo1.maven.org/maven2/org/apache/spark/spark- > streaming-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom > [warn] ==== typesafe-releases: tried > [warn] > http://repo.typesafe.com/typesafe/releases/org/apache/ > spark/spark-streaming-kafka_2.11/2.1.0/spark-streaming- > kafka_2.11-2.1.0.pom > [warn] ==== typesafe-ivy-releasez: tried > [warn] > http://repo.typesafe.com/typesafe/ivy-releases/org. > apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml > [info] Resolving jline#jline;2.12.1 ... > [warn] :::::::::::::::::::::::::::::::::::::::::::::: > [warn] :: UNRESOLVED DEPENDENCIES :: > [warn] :::::::::::::::::::::::::::::::::::::::::::::: > [warn] :: org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found > [warn] :::::::::::::::::::::::::::::::::::::::::::::: > [warn] > [warn] Note: Unresolved dependencies path: > [warn] org.apache.spark:spark-streaming-kafka_2.11:2.1.0 > (/Users/karanalang/Documents/Technology/Coursera_spark_ > scala/spark_kafka_code/sparkkafka.sbt#L12-16) > [warn] +- sparkkafka-project:sparkkafka-project_2.11:1.0 > [trace] Stack trace suppressed: run last *:update for the full output. > [error] (*:update) sbt.ResolveException: unresolved dependency: > org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found > [error] Could not create Eclipse project files: > [error] Error evaluating task "scalacOptions": error > [error] Error evaluating task "externalDependencyClasspath": error > > >