Re: --jars vs --spark.executor.extraClassPath vs --spark.driver.extraClassPath

[Jason Nerothin]

Hi Rajat,

A little more color:

The executor classpath will be used by the spark workers/slaves. For
example, all JVMs that are started with $SPARK_HOME/sbin/start-slave.sh. If
you run with --deploy-mode cluster, then the driver itself will be run from
on the cluster (with executor classpath).

If you run with --deploy-mode client, then the Driver will have its own
classpath (and JVM) on the host that you start it from (similar to running
from an IDE).

If you are NOT running from the shell, then it's usually best to build an
uber-jar containing all required jars and use spark-submit to send the
entire classpath to the cluster. Using --packages like this

is
also a good option for jars that are in a repository (and also resolves
local paths during development).

To run driver code from an IDE, make sure your run/debug configuration is
picking up the spark libs you need as "provided" dependencies (sbt or
maven). This simulates the classpath that's provided by the Spark runtime.
I say 'simulates' because Spark (2.4.1) has access to 226 jar files in
$SPARK_HOME/jars and usually you're implementing against just a few of the
essential ones like spark-sql.

--jars is what to use for spark-shell.

Final related tidbit: If you're implementing in Scala, make sure your jars
are version-compatible with the scala compiler version (2.1.1 as of Spark
2.4.1).

HTH

Jason

On Sat, Apr 20, 2019 at 4:34 AM Subash Prabakar 
wrote:

> Hey Rajat,
>
> The documentation page is self explanatory..
>
> You can refer this for more configs
>
> https://spark.apache.org/docs/2.0.0/configuration.html
>  or any version of Spark documentation
>
> Thanks.
> Subash
>
> On Sat, 20 Apr 2019 at 16:04, rajat kumar 
> wrote:
>
>> Hi,
>>
>> Can anyone pls explain ?
>>
>>
>> On Mon, 15 Apr 2019, 09:31 rajat kumar >
>>> Hi All,
>>>
>>> I came across different parameters in spark submit
>>>
>>> --jars , --spark.executor.extraClassPath , --spark.driver.extraClassPath
>>>
>>> What are the differences between them? When to use which one? Will it
>>> differ
>>> if I use following:
>>>
>>> --master yarn --deploy-mode client
>>> --master yarn --deploy-mode cluster
>>>
>>>
>>> Thanks
>>> Rajat
>>>
>>

-- 
Thanks,
Jason

Re: --jars vs --spark.executor.extraClassPath vs --spark.driver.extraClassPath

[Subash Prabakar]

Hey Rajat,

The documentation page is self explanatory..

You can refer this for more configs

https://spark.apache.org/docs/2.0.0/configuration.html
 or any version of Spark documentation

Thanks.
Subash

On Sat, 20 Apr 2019 at 16:04, rajat kumar 
wrote:

> Hi,
>
> Can anyone pls explain ?
>
>
> On Mon, 15 Apr 2019, 09:31 rajat kumar 
>> Hi All,
>>
>> I came across different parameters in spark submit
>>
>> --jars , --spark.executor.extraClassPath , --spark.driver.extraClassPath
>>
>> What are the differences between them? When to use which one? Will it
>> differ
>> if I use following:
>>
>> --master yarn --deploy-mode client
>> --master yarn --deploy-mode cluster
>>
>>
>> Thanks
>> Rajat
>>
>

Re: --jars vs --spark.executor.extraClassPath vs --spark.driver.extraClassPath

[rajat kumar]

Hi,

Can anyone pls explain ?

On Mon, 15 Apr 2019, 09:31 rajat kumar  Hi All,
>
> I came across different parameters in spark submit
>
> --jars , --spark.executor.extraClassPath , --spark.driver.extraClassPath
>
> What are the differences between them? When to use which one? Will it
> differ
> if I use following:
>
> --master yarn --deploy-mode client
> --master yarn --deploy-mode cluster
>
>
> Thanks
> Rajat
>