I am using standalone deployment, with spark 1.4.1
When I submit the job, I get no error at the submission terminal. Then I
check the webui, I can find the driver section which has a my driver
submission, with this error: java.io.FileNotFoundException ... which point
the full path of my jar as it is on my computer (where I run spark-submit).
Now if I create this full path on the worker(s) with my jar inside, the
application get processed.
Thanks!
On Thu, Oct 1, 2015 at 10:51 AM, Saisai Shao <sai.sai.s...@gmail.com> wrote:
> Are you running on standalone deploy mode, what Spark version are you
> running?
>
> Can you explain a little more specifically what exception occurs, how to
> provide the jar to Spark?
>
> I tried in my local machine with command:
>
> ./bin/spark-submit --verbose --master spark://hw12100.local:7077
> --deploy-mode cluster --class org.apache.spark.examples.SparkPi
> examples/target/scala-2.10/spark-examples-1.5.1-hadoop2.7.1.jar
>
> Seems Spark will upload this examples jar automatically, don't need to
> handle it manually.
>
> Thanks
> Saisai
>
>
>
> On Thu, Oct 1, 2015 at 8:36 AM, Christophe Schmitz <cofcof...@gmail.com>
> wrote:
>
>> Hi Saisai
>>
>> I am using this command:
>> spark-submit --deploy-mode cluster --properties-file file.conf --class
>> myclass test-assembly-1.0.jar
>>
>> The application start only if I manually copy test-assembly-1.0.jar in
>> all the worer (or the master, I don't remember) and provide the full path
>> of the file.
>>
>> On the other hand with --deploy-mode client I don't need to do that, but
>> then I need to accept incoming connection in my client to serve the jar
>> (which is not possible behind a firewall I don't control)
>>
>> Thanks,
>>
>> Christophe
>>
>>
>> On Wed, Sep 30, 2015 at 5:19 PM, Saisai Shao <sai.sai.s...@gmail.com>
>> wrote:
>>
>>> As I remembered you don't need to upload application jar manually, Spark
>>> will do it for you when you use Spark submit. Would you mind posting out
>>> your command of Spark submit?
>>>
>>>
>>> On Wed, Sep 30, 2015 at 3:13 PM, Christophe Schmitz <cofcof...@gmail.com
>>> > wrote:
>>>
>>>> Hi there,
>>>>
>>>> I am trying to use the "--deploy-mode cluster" option to submit my job
>>>> (spark 1.4.1). When I do that, the spark-driver (on the cluster) is looking
>>>> for my application jar. I can manually copy my application jar on all the
>>>> workers, but I was wondering if there is a way to submit the application
>>>> jar when running spark-submit.
>>>>
>>>> Thanks!
>>>>
>>>
>>>
>>
>
