It looks like you're attempting to mix Scala versions, so that's going to cause some problems. If you really want to use Scala 2.11.5, you must also use Spark package versions built for Scala 2.11 rather than 2.10. Anyway, that's not quite the correct way to specify Scala dependencies in build.sbt. Instead of placing the Scala version after the artifactId (like "spark-core_2.10"), what you actually want is to use just "spark-core" with two percent signs before it. Using two percent signs will make it use the version of Scala that matches your declared scalaVersion. For example:
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.3.0" % "provided" libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.3.0" % "provided" libraryDependencies += "org.apache.spark" %% "spark-streaming-kinesis-asl" % "1.3.0" I think that may get you a little closer, though I think you're probably going to run into the same problems I ran into in this thread: https://www.mail-archive.com/user@spark.apache.org/msg23891.html I never really got an answer for that, and I temporarily moved on to other things for now. ~ Jonathan Kelly From: 'Vadim Bichutskiy' <vadim.bichuts...@gmail.com<mailto:vadim.bichuts...@gmail.com>> Date: Thursday, April 2, 2015 at 9:53 AM To: "user@spark.apache.org<mailto:user@spark.apache.org>" <user@spark.apache.org<mailto:user@spark.apache.org>> Subject: Spark + Kinesis Hi all, I am trying to write an Amazon Kinesis consumer Scala app that processes data in the Kinesis stream. Is this the correct way to specify build.sbt: ------- import AssemblyKeys._ name := "Kinesis Consumer" version := "1.0" organization := "com.myconsumer" scalaVersion := "2.11.5" libraryDependencies ++= Seq("org.apache.spark" % "spark-core_2.10" % "1.3.0" % "provided", "org.apache.spark" % "spark-streaming_2.10" % "1.3.0" "org.apache.spark" % "spark-streaming-kinesis-asl_2.10" % "1.3.0") assemblySettings jarName in assembly := "consumer-assembly.jar" assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala=false) -------- In project/assembly.sbt I have only the following line: addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.13.0") I am using sbt 0.13.7. I adapted Example 7.7 in the Learning Spark book. Thanks, Vadim [https://mailfoogae.appspot.com/t?sender=admFkaW0uYmljaHV0c2tpeUBnbWFpbC5jb20%3D&type=zerocontent&guid=ccce8efd-9f9f-4140-8f31-28b661c06314]ᐧ
