Hi Rajat, A little more color:
The executor classpath will be used by the spark workers/slaves. For example, all JVMs that are started with $SPARK_HOME/sbin/start-slave.sh. If you run with --deploy-mode cluster, then the driver itself will be run from on the cluster (with executor classpath). If you run with --deploy-mode client, then the Driver will have its own classpath (and JVM) on the host that you start it from (similar to running from an IDE). If you are NOT running from the shell, then it's usually best to build an uber-jar containing all required jars and use spark-submit to send the entire classpath to the cluster. Using --packages like this <https://stackoverflow.com/questions/33928029/how-to-specify-multiple-dependencies-using-packages-for-spark-submit> is also a good option for jars that are in a repository (and also resolves local paths during development). To run driver code from an IDE, make sure your run/debug configuration is picking up the spark libs you need as "provided" dependencies (sbt or maven). This simulates the classpath that's provided by the Spark runtime. I say 'simulates' because Spark (2.4.1) has access to 226 jar files in $SPARK_HOME/jars and usually you're implementing against just a few of the essential ones like spark-sql. --jars is what to use for spark-shell. Final related tidbit: If you're implementing in Scala, make sure your jars are version-compatible with the scala compiler version (2.1.1 as of Spark 2.4.1). HTH Jason On Sat, Apr 20, 2019 at 4:34 AM Subash Prabakar <subashpraba...@gmail.com> wrote: > Hey Rajat, > > The documentation page is self explanatory.. > > You can refer this for more configs > > https://spark.apache.org/docs/2.0.0/configuration.html > or any version of Spark documentation > > Thanks. > Subash > > On Sat, 20 Apr 2019 at 16:04, rajat kumar <kumar.rajat20...@gmail.com> > wrote: > >> Hi, >> >> Can anyone pls explain ? >> >> >> On Mon, 15 Apr 2019, 09:31 rajat kumar <kumar.rajat20...@gmail.com wrote: >> >>> Hi All, >>> >>> I came across different parameters in spark submit >>> >>> --jars , --spark.executor.extraClassPath , --spark.driver.extraClassPath >>> >>> What are the differences between them? When to use which one? Will it >>> differ >>> if I use following: >>> >>> --master yarn --deploy-mode client >>> --master yarn --deploy-mode cluster >>> >>> >>> Thanks >>> Rajat >>> >> -- Thanks, Jason
