Assembly settings have an option to exclude jars. You need something
similar to:
assemblyExcludedJars in assembly <<= (fullClasspath in assembly) map { cp =>
val excludes = Set(
"minlog-1.2.jar"
)
cp filter { jar => excludes(jar.data.getName) }
}
in your build file (may need to be refactored into a .scala file)
On Fri, Apr 3, 2015 at 12:57 PM, Vadim Bichutskiy <
vadim.bichuts...@gmail.com> wrote:
> Remove provided and got the following error:
>
> [error] (*:assembly) deduplicate: different file contents found in the
> following:
>
> [error]
> /Users/vb/.ivy2/cache/com.esotericsoftware.kryo/kryo/bundles/kryo-2.21.jar:com/esotericsoftware/minlog/Log$Logger.class
>
> [error]
> /Users/vb/.ivy2/cache/com.esotericsoftware.minlog/minlog/jars/minlog-1.2.jar:com/esotericsoftware/minlog/Log$Logger.class
> ᐧ
>
> On Fri, Apr 3, 2015 at 3:48 PM, Tathagata Das <t...@databricks.com> wrote:
>
>> Just remove "provided" for spark-streaming-kinesis-asl
>>
>> libraryDependencies += "org.apache.spark" %%
>> "spark-streaming-kinesis-asl" % "1.3.0"
>>
>> On Fri, Apr 3, 2015 at 12:45 PM, Vadim Bichutskiy <
>> vadim.bichuts...@gmail.com> wrote:
>>
>>> Thanks. So how do I fix it?
>>> ᐧ
>>>
>>> On Fri, Apr 3, 2015 at 3:43 PM, Kelly, Jonathan <jonat...@amazon.com>
>>> wrote:
>>>
>>>> spark-streaming-kinesis-asl is not part of the Spark distribution on
>>>> your cluster, so you cannot have it be just a "provided" dependency. This
>>>> is also why the KCL and its dependencies were not included in the assembly
>>>> (but yes, they should be).
>>>>
>>>>
>>>> ~ Jonathan Kelly
>>>>
>>>> From: Vadim Bichutskiy <vadim.bichuts...@gmail.com>
>>>> Date: Friday, April 3, 2015 at 12:26 PM
>>>> To: Jonathan Kelly <jonat...@amazon.com>
>>>> Cc: "user@spark.apache.org" <user@spark.apache.org>
>>>> Subject: Re: Spark + Kinesis
>>>>
>>>> Hi all,
>>>>
>>>> Good news! I was able to create a Kinesis consumer and assemble it
>>>> into an "uber jar" following
>>>> http://spark.apache.org/docs/latest/streaming-kinesis-integration.html
>>>> and example
>>>> https://github.com/apache/spark/blob/master/extras/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala
>>>> .
>>>>
>>>> However when I try to spark-submit it I get the following exception:
>>>>
>>>> *Exception in thread "main" java.lang.NoClassDefFoundError:
>>>> com/amazonaws/auth/AWSCredentialsProvider*
>>>>
>>>> Do I need to include KCL dependency in *build.sbt*, here's what it
>>>> looks like currently:
>>>>
>>>> import AssemblyKeys._
>>>> name := "Kinesis Consumer"
>>>> version := "1.0"
>>>> organization := "com.myconsumer"
>>>> scalaVersion := "2.11.5"
>>>>
>>>> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.3.0" %
>>>> "provided"
>>>> libraryDependencies += "org.apache.spark" %% "spark-streaming" %
>>>> "1.3.0" % "provided"
>>>> libraryDependencies += "org.apache.spark" %%
>>>> "spark-streaming-kinesis-asl" % "1.3.0" % "provided"
>>>>
>>>> assemblySettings
>>>> jarName in assembly := "consumer-assembly.jar"
>>>> assemblyOption in assembly := (assemblyOption in
>>>> assembly).value.copy(includeScala=false)
>>>>
>>>> Any help appreciated.
>>>>
>>>> Thanks,
>>>> Vadim
>>>>
>>>> On Thu, Apr 2, 2015 at 1:15 PM, Kelly, Jonathan <jonat...@amazon.com>
>>>> wrote:
>>>>
>>>>> It looks like you're attempting to mix Scala versions, so that's
>>>>> going to cause some problems. If you really want to use Scala 2.11.5, you
>>>>> must also use Spark package versions built for Scala 2.11 rather than
>>>>> 2.10. Anyway, that's not quite the correct way to specify Scala
>>>>> dependencies in build.sbt. Instead of placing the Scala version after the
>>>>> artifactId (like "spark-core_2.10"), what you actually want is to use just
>>>>> "spark-core" with two percent signs before it. Using two percent signs
>>>>> will make it use the version of Scala that matches your declared
>>>>> scalaVersion. For example:
>>>>>
>>>>> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.3.0"
>>>>> % "provided"
>>>>>
>>>>> libraryDependencies += "org.apache.spark" %% "spark-streaming" %
>>>>> "1.3.0" % "provided"
>>>>>
>>>>> libraryDependencies += "org.apache.spark" %%
>>>>> "spark-streaming-kinesis-asl" % "1.3.0"
>>>>>
>>>>> I think that may get you a little closer, though I think you're
>>>>> probably going to run into the same problems I ran into in this thread:
>>>>> https://www.mail-archive.com/user@spark.apache.org/msg23891.html I
>>>>> never really got an answer for that, and I temporarily moved on to other
>>>>> things for now.
>>>>>
>>>>>
>>>>> ~ Jonathan Kelly
>>>>>
>>>>> From: 'Vadim Bichutskiy' <vadim.bichuts...@gmail.com>
>>>>> Date: Thursday, April 2, 2015 at 9:53 AM
>>>>> To: "user@spark.apache.org" <user@spark.apache.org>
>>>>> Subject: Spark + Kinesis
>>>>>
>>>>> Hi all,
>>>>>
>>>>> I am trying to write an Amazon Kinesis consumer Scala app that
>>>>> processes data in the
>>>>> Kinesis stream. Is this the correct way to specify *build.sbt*:
>>>>>
>>>>> -------
>>>>> *import AssemblyKeys._*
>>>>> *name := "Kinesis Consumer"*
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *version := "1.0" organization := "com.myconsumer" scalaVersion :=
>>>>> "2.11.5" libraryDependencies ++= Seq("org.apache.spark" %
>>>>> "spark-core_2.10"
>>>>> % "1.3.0" % "provided", "org.apache.spark" % "spark-streaming_2.10" %
>>>>> "1.3.0" "org.apache.spark" % "spark-streaming-kinesis-asl_2.10" %
>>>>> "1.3.0")*
>>>>>
>>>>>
>>>>>
>>>>> * assemblySettings jarName in assembly := "consumer-assembly.jar"
>>>>> assemblyOption in assembly := (assemblyOption in
>>>>> assembly).value.copy(includeScala=false)*
>>>>> --------
>>>>>
>>>>> In *project/assembly.sbt* I have only the following line:
>>>>>
>>>>> *addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.13.0")*
>>>>>
>>>>> I am using sbt 0.13.7. I adapted Example 7.7 in the Learning Spark
>>>>> book.
>>>>>
>>>>> Thanks,
>>>>> Vadim
>>>>>
>>>>>
>>>>
>>>
>>
>
--
*Dan Osipov*
*Shazam*
2114 Broadway Street, Redwood City, CA 94063
Please consider the environment before printing this document
Shazam Entertainment Limited is incorporated in England and Wales under
company number 3998831 and its registered office is at 26-28 Hammersmith
Grove, London W6 7HA. Shazam Media Services Inc is a member of the Shazam
Entertainment Limited group of companies.
