I got the error during run time. It was for mongo-spark-connector class
files.
My build.sbt is like this
name := "Test Advice Project"
version := "1.0"
scalaVersion := "2.10.6"
libraryDependencies ++= Seq(
"org.mongodb.spark" %% "mongo-spark-connector" % "1.0.0",
"org.apache.spark" %% "spark-core" % "1.6.1" % "provided",
"org.apache.spark" %% "spark-sql" % "1.6.1" % "provided"
)
assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs @ _*) => MergeStrategy.discard
case x => MergeStrategy.first
}
and I create the fat jar using sbt assembly
I have also added addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")
in ~/.sbt/0.13/plugins/plugins.sbt for sbt assembly to work.
I think when you use provided it does not include those jars in your fat
jar.
I am using spark 1.6
Thanks
Sachin
On Wed, Jul 20, 2016 at 11:23 PM, Marco Mistroni <mmistr...@gmail.com>
wrote:
> that will work but ideally you should not include any of the
> spark-releated jars as they are provided to you by the spark environment
> whenever you launch your app via spark-submit (this will prevent unexpected
> errors e.g. when you kick off your app using a different version of spark
> where some of the classes has been renamd or movedaround -tbh i don't think
> this is a case that happen often)
>
> Btw did you get the NoClassDefFoundException at compile time or run
> time?if at run time, what is your Spark Version and what is the spark
> libraries version you used in your sbt?
> are you using a Spark version pre 1.4?
>
> kr
> marco
>
>
>
>
>
>
> On Wed, Jul 20, 2016 at 6:13 PM, Sachin Mittal <sjmit...@gmail.com> wrote:
>
>> NoClassDefFound error was for spark classes like say SparkConext.
>> When running a standalone spark application I was not passing external
>> jars using --jars option.
>>
>> However I have fixed this by making a fat jar using sbt assembly plugin.
>>
>> Now all the dependencies are included in that jar and I use that jar in
>> spark-submit
>>
>> Thanks
>> Sachin
>>
>>
>> On Wed, Jul 20, 2016 at 9:42 PM, Marco Mistroni <mmistr...@gmail.com>
>> wrote:
>>
>>> Hello Sachin
>>> pls paste the NoClassDefFound Exception so we can see what's failing,
>>> aslo please advise how are you running your Spark App
>>> For an extremely simple case, let's assume you have your
>>> MyFirstSparkApp packaged in your myFirstSparkApp.jar
>>> Then all you need to do would be to kick off
>>>
>>> spark-submit --class MyFirstSparkApp myFirstSparkApp.jar
>>>
>>> if you have any external dependencies (not spark , let's assume you are
>>> using common-utils.jar) then you should be able to kick it off via
>>>
>>> spark-submit --class MyFirstSparkApp --jars common-utiils.jar
>>> myFirstSparkApp.jar
>>>
>>> I paste below the build.sbt i am using for my SparkExamples apps, hope
>>> this helps.
>>> kr
>>> marco
>>>
>>> name := "SparkExamples"
>>>
>>> version := "1.0"
>>>
>>> scalaVersion := "2.10.5"
>>>
>>>
>>> // Add a single dependency
>>> libraryDependencies += "junit" % "junit" % "4.8" % "test"
>>> libraryDependencies += "org.mockito" % "mockito-core" % "1.9.5"
>>> libraryDependencies ++= Seq("org.slf4j" % "slf4j-api" % "1.7.5",
>>> "org.slf4j" % "slf4j-simple" % "1.7.5",
>>> "org.clapper" %% "grizzled-slf4j" % "1.0.2")
>>> libraryDependencies += "org.powermock" %
>>> "powermock-mockito-release-full" % "1.5.4" % "test"
>>> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1" %
>>> "provided"
>>> libraryDependencies += "org.apache.spark" %% "spark-streaming" %
>>> "1.6.1" % "provided"
>>> libraryDependencies += "org.apache.spark" %% "spark-mllib" % "1.6.1"
>>> % "provided"
>>> libraryDependencies += "org.apache.spark" %% "spark-streaming-flume" %
>>> "1.3.0" % "provided"
>>> resolvers += "softprops-maven" at "
>>> http://dl.bintray.com/content/softprops/maven";
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Jul 20, 2016 at 3:39 PM, Mich Talebzadeh <
>>> mich.talebza...@gmail.com> wrote:
>>>
>>>> you need an uber jar file.
>>>>
>>>> Have you actually followed the dependencies and project sub-directory
>>>> build?
>>>>
>>>> check this.
>>>>
>>>>
>>>> http://stackoverflow.com/questions/28459333/how-to-build-an-uber-jar-fat-jar-using-sbt-within-intellij-idea
>>>>
>>>> under three answers the top one.
>>>>
>>>> I started reading the official SBT tutorial
>>>> <http://www.scala-sbt.org/0.13/tutorial/>. .....
>>>>
>>>> HTH
>>>>
>>>> Dr Mich Talebzadeh
>>>>
>>>>
>>>>
>>>> LinkedIn *
>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>
>>>>
>>>>
>>>> http://talebzadehmich.wordpress.com
>>>>
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>>> On 20 July 2016 at 09:54, Sachin Mittal <sjmit...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>> I am following the example under
>>>>> https://spark.apache.org/docs/latest/quick-start.html
>>>>> For standalone scala application.
>>>>>
>>>>> I added all my dependencies via build.sbt (one dependency is under lib
>>>>> folder).
>>>>>
>>>>> When I run sbt package I see the jar created under
>>>>> target/scala-2.10/
>>>>>
>>>>> So compile seems to be working fine. However when I inspect that jar,
>>>>> it only contains my scala class.
>>>>> Unlike in java application we build a standalone jar, which contains
>>>>> all the dependencies inside that jar, here all the dependencies are
>>>>> missing.
>>>>>
>>>>> So as expected when I run the application via spark-submit I get the
>>>>> NoClassDefFoundError.
>>>>>
>>>>> Here is my build.sbt
>>>>>
>>>>> name := "Test Advice Project"
>>>>> version := "1.0"
>>>>> scalaVersion := "2.10.6"
>>>>> libraryDependencies ++= Seq(
>>>>> "org.apache.spark" %% "spark-core" % "1.6.1",
>>>>> "org.apache.spark" %% "spark-sql" % "1.6.1"
>>>>> )
>>>>>
>>>>> Can anyone please guide me to as what is going wrong and why sbt
>>>>> package is not including all the dependencies jar classes in the new jar.
>>>>>
>>>>> Thanks
>>>>> Sachin
>>>>>
>>>>>
>>>>> On Tue, Jul 19, 2016 at 8:23 PM, Andrew Ehrlich <and...@aehrlich.com>
>>>>> wrote:
>>>>>
>>>>>> Yes, spark-core will depend on Hadoop and several other jars. Here’s
>>>>>> the list of dependencies:
>>>>>> https://github.com/apache/spark/blob/master/core/pom.xml#L35
>>>>>>
>>>>>> Whether you need spark-sql depends on whether you will use the
>>>>>> DataFrame API. Without spark-sql, you will just have the RDD API.
>>>>>>
>>>>>> On Jul 19, 2016, at 7:09 AM, Sachin Mittal <sjmit...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>> Can someone please guide me what all jars I need to place in my lib
>>>>>> folder of the project to build a standalone scala application via sbt.
>>>>>>
>>>>>> Note I need to provide static dependencies and I cannot download the
>>>>>> jars using libraryDependencies.
>>>>>> So I need to provide all the jars upfront.
>>>>>>
>>>>>> So far I found that we need:
>>>>>> spark-core_<version>.jar
>>>>>>
>>>>>> Do we also need
>>>>>> spark-sql_<version>.jar
>>>>>> and
>>>>>> hadoop-core-<version>.jar
>>>>>>
>>>>>> Is there any jar from spark side I may be missing? What I found that
>>>>>> spark-core needs hadoop-core classes and if I don't add them then sbt was
>>>>>> giving me this error:
>>>>>> [error] bad symbolic reference. A signature in SparkContext.class
>>>>>> refers to term hadoop
>>>>>> [error] in package org.apache which is not available.
>>>>>>
>>>>>> So I was just confused on library dependency part when building an
>>>>>> application via sbt. Any inputs here would be helpful.
>>>>>>
>>>>>> Thanks
>>>>>> Sachin
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
