I take you don't want to use the --jars option to avoid moving them every time?
On Tue, 27 Dec 2016, 10:33 Mich Talebzadeh, <mich.talebza...@gmail.com> wrote: > When one runs in Local mode (one JVM) on an edge host (the host user > accesses the cluster), it is possible to put additional jar file say > accessing Oracle RDBMS tables in $SPARK_CLASSPATH. This works > > export SPARK_CLASSPATH=~/user_jars/ojdbc6.jar > > Normally a group of users can have read access to a shared directory like > above and once they log in their shell will invoke an environment file that > will have the above classpath plus additional parameters like $JAVA_HOME > etc are set up for them. > > However, if the user chooses to run spark through spark-submit with yarn, > then the only way this will work in my research is to add the jar file as > follows on every node of Spark cluster > > in $SPARK_HOME/conf/spark-defaults.conf > > Add the jar path to the following: > > spark.executor.extraClassPath /user_jars/ojdbc6.jar > > Note that setting both spark.executor.extraClassPath and SPARK_CLASSPATH > will cause initialisation error > > ERROR SparkContext: Error initializing SparkContext. > org.apache.spark.SparkException: Found both spark.executor.extraClassPath > and SPARK_CLASSPATH. Use only the former. > > I was wondering if there are other ways of making this work in YARN mode, > where every node of cluster will require this JAR file? > > Thanks >
