Thanks.
The problem is that with spark-shell --jars does not work! This is Spark 2
accessing Oracle 12c
spark-shell --jars /home/hduser/jars/ojdbc6.jar
It comes back with
java.sql.SQLException: No suitable driver
unfortunately
and spark-shell uses spark-submit under the bonnet if you look at the shell
file
"${SPARK_HOME}"/bin/spark-submit --class org.apache.spark.repl.Main --name
"Spark shell" "$@"
hm
Dr Mich Talebzadeh
LinkedIn *
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 27 December 2016 at 09:52, Deepak Sharma <deepakmc...@gmail.com> wrote:
> Hi Mich
> You can copy the jar to shared location and use --jars command line
> argument of spark-submit.
> Who so ever needs access to this jar , can refer to the shared path and
> access it using --jars argument.
>
> Thanks
> Deepak
>
> On Tue, Dec 27, 2016 at 3:03 PM, Mich Talebzadeh <
> mich.talebza...@gmail.com> wrote:
>
>> When one runs in Local mode (one JVM) on an edge host (the host user
>> accesses the cluster), it is possible to put additional jar file say
>> accessing Oracle RDBMS tables in $SPARK_CLASSPATH. This works
>>
>> export SPARK_CLASSPATH=~/user_jars/ojdbc6.jar
>>
>> Normally a group of users can have read access to a shared directory like
>> above and once they log in their shell will invoke an environment file that
>> will have the above classpath plus additional parameters like $JAVA_HOME
>> etc are set up for them.
>>
>> However, if the user chooses to run spark through spark-submit with yarn,
>> then the only way this will work in my research is to add the jar file as
>> follows on every node of Spark cluster
>>
>> in $SPARK_HOME/conf/spark-defaults.conf
>>
>> Add the jar path to the following:
>>
>> spark.executor.extraClassPath /user_jars/ojdbc6.jar
>>
>> Note that setting both spark.executor.extraClassPath and SPARK_CLASSPATH
>> will cause initialisation error
>>
>> ERROR SparkContext: Error initializing SparkContext.
>> org.apache.spark.SparkException: Found both
>> spark.executor.extraClassPath and SPARK_CLASSPATH. Use only the former.
>>
>> I was wondering if there are other ways of making this work in YARN mode,
>> where every node of cluster will require this JAR file?
>>
>> Thanks
>>
>
>
>
> --
> Thanks
> Deepak
> www.bigdatabig.com
> www.keosha.net
>
