Thanks Jonathan. Helpful. VB
> On Apr 2, 2015, at 1:15 PM, Kelly, Jonathan <jonat...@amazon.com> wrote: > > It looks like you're attempting to mix Scala versions, so that's going to > cause some problems. If you really want to use Scala 2.11.5, you must also > use Spark package versions built for Scala 2.11 rather than 2.10. Anyway, > that's not quite the correct way to specify Scala dependencies in build.sbt. > Instead of placing the Scala version after the artifactId (like > "spark-core_2.10"), what you actually want is to use just "spark-core" with > two percent signs before it. Using two percent signs will make it use the > version of Scala that matches your declared scalaVersion. For example: > > libraryDependencies += "org.apache.spark" %% "spark-core" % "1.3.0" % > "provided" > > libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.3.0" % > "provided" > > libraryDependencies += "org.apache.spark" %% "spark-streaming-kinesis-asl" % > "1.3.0" > > I think that may get you a little closer, though I think you're probably > going to run into the same problems I ran into in this thread: > https://www.mail-archive.com/user@spark.apache.org/msg23891.html I never > really got an answer for that, and I temporarily moved on to other things for > now. > > ~ Jonathan Kelly > > From: 'Vadim Bichutskiy' <vadim.bichuts...@gmail.com> > Date: Thursday, April 2, 2015 at 9:53 AM > To: "user@spark.apache.org" <user@spark.apache.org> > Subject: Spark + Kinesis > > Hi all, > > I am trying to write an Amazon Kinesis consumer Scala app that processes data > in the > Kinesis stream. Is this the correct way to specify build.sbt: > > ------- > import AssemblyKeys._ > name := "Kinesis Consumer" > version := "1.0" > organization := "com.myconsumer" > scalaVersion := "2.11.5" > > libraryDependencies ++= Seq("org.apache.spark" % "spark-core_2.10" % "1.3.0" > % "provided", > "org.apache.spark" % "spark-streaming_2.10" % "1.3.0" > "org.apache.spark" % "spark-streaming-kinesis-asl_2.10" % "1.3.0") > > assemblySettings > jarName in assembly := "consumer-assembly.jar" > assemblyOption in assembly := (assemblyOption in > assembly).value.copy(includeScala=false) > -------- > > In project/assembly.sbt I have only the following line: > > addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.13.0") > > I am using sbt 0.13.7. I adapted Example 7.7 in the Learning Spark book. > > Thanks, > Vadim > > ᐧ
