HI Sonal,
Yes I am working towards that same idea. How did you go about creating
the non-spark-jar dependencies ? The way I am doing it is a separate
straw-man project that does not include spark but has the external third
party jars included. Then running sbt compile:managedClasspath and reverse
engineering the lib jars from it. That is obviously not ideal.
The maven "run" will be useful for other projects built by maven: i will
keep in my notes.
AFA sbt run-example, it requires additional libraries to be added for my
external dependencies. I tried several items including ADD_JARS,
--driver-class-path and combinations of extraClassPath. I have deferred
that ad-hoc approach to finding a systematic one.
2014-05-08 5:26 GMT-07:00 Sonal Goyal <[email protected]>:
> I am creating a jar with only my dependencies and run spark-submit through
> my project mvn build. I have configured the mvn exec goal to the location
> of the script. Here is how I have set it up for my app. The mainClass is my
> driver program, and I am able to send my custom args too. Hope this helps.
>
> <plugin>
> <groupId>org.codehaus.mojo</groupId>
> <artifactId>exec-maven-plugin</artifactId>
> <executions>
> <execution>
> <goals>
> <goal>exec</goal>
> </goals>
> </execution>
> </executions>
> <configuration>
> <executable>/home/sgoyal/spark/bin/spark-submit</executable>
> <arguments>
> <argument>${jars}</argument>
> <argument>--class</argument>
> <argument>${mainClass}</argument>
> <argument>--arg</argument>
> <argument>${spark.master}</argument>
> <argument>--arg</argument>
> <argument>${my app arg 1}</argument>
> <argument>--arg</argument>
> <argument>${my arg 2}</argument>
> </arguments>
> </configuration>
> </plugin>
>
>
> Best Regards,
> Sonal
> Nube Technologies <http://www.nubetech.co>
>
> <http://in.linkedin.com/in/sonalgoyal>
>
>
>
>
> On Wed, May 7, 2014 at 6:57 AM, Tathagata Das <[email protected]
> > wrote:
>
>> Doesnt the run-example script work for you? Also, are you on the latest
>> commit of branch-1.0 ?
>>
>> TD
>>
>>
>> On Mon, May 5, 2014 at 7:51 PM, Soumya Simanta
>> <[email protected]>wrote:
>>
>>>
>>>
>>> Yes, I'm struggling with a similar problem where my class are not found
>>> on the worker nodes. I'm using 1.0.0_SNAPSHOT. I would really appreciate
>>> if someone can provide some documentation on the usage of spark-submit.
>>>
>>> Thanks
>>>
>>> > On May 5, 2014, at 10:24 PM, Stephen Boesch <[email protected]> wrote:
>>> >
>>> >
>>> > I have a spark streaming application that uses the external streaming
>>> modules (e.g. kafka, mqtt, ..) as well. It is not clear how to properly
>>> invoke the spark-submit script: what are the ---driver-class-path and/or
>>> -Dspark.executor.extraClassPath parameters required?
>>> >
>>> > For reference, the following error is proving difficult to resolve:
>>> >
>>> > java.lang.ClassNotFoundException:
>>> org.apache.spark.streaming.examples.StreamingExamples
>>> >
>>>
>>
>>
>
