Thanks all Actually I have a generic shell script that for a given Scala program creates jar file using Maven or SBT.
I modified that one to create a uber jar file as well using SBT assembly. The root directory structure in uber has one more sub-directory. In addition SBT relies on a single sbt file. Maven relies on pom.xml. In contrast SBT assembly requires a build.sbt file plus assembly,sbt file The assembly sbt file asssenly.sbt simply has one liner addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.12.0") The buid.sbt has the following in my case (a generic one) lazy val root = (project in file(".")). settings( name := "${APPLICATION}", version := "1.0", scalaVersion := "2.10.4", mainClass in Compile := Some("myPackage.${APPLICATION}") ) libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1" % "provided" libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1" % "provided" libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1" % "provided" libraryDependencies += "junit" % "junit" % "4.12" libraryDependencies += "org.scala-sbt" % "test-interface" % "1.0" libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.6.1" % "provided" libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka" % "1.6.1" // META-INF discarding mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) => { case PathList("META-INF", xs @ _*) => MergeStrategy.discard case x => MergeStrategy.first } } So if I want to create an sbt assembly jar file and assuming that the Scala file is called TestStream_assembly then I just run the shell script passing the Application 'A' and type 'T' to the shell ./generic.ksh -A TestStream_assembly -T assembly For simple sbt that will be ./generic.ksh -A TestStream -T sbt and for maven it will be ./generic.ksh -A TestStream -T mvn Cheers, Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com On 4 April 2016 at 16:30, vetal king <greenve...@gmail.com> wrote: > > ---------- Forwarded message ---------- > From: vetal king <greenve...@gmail.com> > Date: Mon, Apr 4, 2016 at 8:59 PM > Subject: Re: All inclusive uber-jar > To: Mich Talebzadeh <mich.talebza...@gmail.com> > > > Not sure how to create uber jar using sbt, but this is how you can do it > using maven. > > <plugins> > .... > <plugin> > <groupId>org.apache.maven.plugins</groupId> > <artifactId>maven-shade-plugin</artifactId> > <version>2.4.3</version> > <executions> > <execution> > <phase>package</phase> > <goals> > <goal>shade</goal> > </goals> > <configuration> > <artifactSet> > <includes> > * <include>*:*</include>* > </includes> > </artifactSet> > </configuration> > </execution> > </executions> > </plugin> > > </plugins> > > But instead of creating a uber jar, consider providing maven coordinates > with the help of sark-submit's --packages and --repositories options > > > > On Mon, Apr 4, 2016 at 7:06 PM, Mich Talebzadeh <mich.talebza...@gmail.com > > wrote: > >> Hi, >> >> >> When one builds a project for Spark in this case Spark streaming with >> SBT, as usual I add dependencies as follows: >> >> libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.6.1" >> libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka" % >> "1.6.1" >> >> However when I submit it through spark-submit I need to put the package >> containing KafkaUtils the same way I do it in spark-shell >> >> ${SPARK_HOME}/bin/spark-submit \ >> --jars >> /home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar \ >> ..... >> >> Now if I want to distribute this as all-in-one package so that it can be >> run from any node, I have been told that I need to create an uber-jar. I >> have not done this before so I assume an uber-jar will be totally self >> contained with all the classes etc. >> >> Can someone elaborate on this please? >> >> Thanks >> >> Dr Mich Talebzadeh >> >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> > > >