@Sonal - makes sense. Is the maven shade plugin runnable within sbt ? If so would you care to share those build.sbt (or .scala) lines? If not, are you aware of a similar plugin for sbt?
2014-05-11 23:53 GMT-07:00 Sonal Goyal <sonalgoy...@gmail.com>: > Hi Stephen, > > I am using maven shade plugin for creating my uber jar. I have marked > spark dependencies as provided. > > Best Regards, > Sonal > Nube Technologies <http://www.nubetech.co> > > <http://in.linkedin.com/in/sonalgoyal> > > > > > On Mon, May 12, 2014 at 1:04 AM, Stephen Boesch <java...@gmail.com> wrote: > >> HI Sonal, >> Yes I am working towards that same idea. How did you go about >> creating the non-spark-jar dependencies ? The way I am doing it is a >> separate straw-man project that does not include spark but has the external >> third party jars included. Then running sbt compile:managedClasspath and >> reverse engineering the lib jars from it. That is obviously not ideal. >> >> The maven "run" will be useful for other projects built by maven: i will >> keep in my notes. >> >> AFA sbt run-example, it requires additional libraries to be added for my >> external dependencies. I tried several items including ADD_JARS, >> --driver-class-path and combinations of extraClassPath. I have deferred >> that ad-hoc approach to finding a systematic one. >> >> >> >> >> 2014-05-08 5:26 GMT-07:00 Sonal Goyal <sonalgoy...@gmail.com>: >> >> I am creating a jar with only my dependencies and run spark-submit >>> through my project mvn build. I have configured the mvn exec goal to the >>> location of the script. Here is how I have set it up for my app. The >>> mainClass is my driver program, and I am able to send my custom args too. >>> Hope this helps. >>> >>> <plugin> >>> <groupId>org.codehaus.mojo</groupId> >>> <artifactId>exec-maven-plugin</artifactId> >>> <executions> >>> <execution> >>> <goals> >>> <goal>exec</goal> >>> </goals> >>> </execution> >>> </executions> >>> <configuration> >>> <executable>/home/sgoyal/spark/bin/spark-submit</executable> >>> <arguments> >>> <argument>${jars}</argument> >>> <argument>--class</argument> >>> <argument>${mainClass}</argument> >>> <argument>--arg</argument> >>> <argument>${spark.master}</argument> >>> <argument>--arg</argument> >>> <argument>${my app arg 1}</argument> >>> <argument>--arg</argument> >>> <argument>${my arg 2}</argument> >>> </arguments> >>> </configuration> >>> </plugin> >>> >>> >>> Best Regards, >>> Sonal >>> Nube Technologies <http://www.nubetech.co> >>> >>> <http://in.linkedin.com/in/sonalgoyal> >>> >>> >>> >>> >>> On Wed, May 7, 2014 at 6:57 AM, Tathagata Das < >>> tathagata.das1...@gmail.com> wrote: >>> >>>> Doesnt the run-example script work for you? Also, are you on the latest >>>> commit of branch-1.0 ? >>>> >>>> TD >>>> >>>> >>>> On Mon, May 5, 2014 at 7:51 PM, Soumya Simanta < >>>> soumya.sima...@gmail.com> wrote: >>>> >>>>> >>>>> >>>>> Yes, I'm struggling with a similar problem where my class are not >>>>> found on the worker nodes. I'm using 1.0.0_SNAPSHOT. I would really >>>>> appreciate if someone can provide some documentation on the usage of >>>>> spark-submit. >>>>> >>>>> Thanks >>>>> >>>>> > On May 5, 2014, at 10:24 PM, Stephen Boesch <java...@gmail.com> >>>>> wrote: >>>>> > >>>>> > >>>>> > I have a spark streaming application that uses the external >>>>> streaming modules (e.g. kafka, mqtt, ..) as well. It is not clear how to >>>>> properly invoke the spark-submit script: what are the ---driver-class-path >>>>> and/or -Dspark.executor.extraClassPath parameters required? >>>>> > >>>>> > For reference, the following error is proving difficult to resolve: >>>>> > >>>>> > java.lang.ClassNotFoundException: >>>>> org.apache.spark.streaming.examples.StreamingExamples >>>>> > >>>>> >>>> >>>> >>> >> >
