You'd have to use spark.{driver,executor}.extraClassPath to modify the
system class loader. But that also means you have to manually
distribute the jar to the nodes in your cluster, into a common
location.
On Thu, Apr 23, 2015 at 6:38 PM, Night Wolf <nightwolf...@gmail.com> wrote:
> Hi guys,
>
> Having a problem build a DataFrame in Spark SQL from a JDBC data source when
> running with --master yarn-client and adding the JDBC driver JAR with
> --jars. If I run with a local[*] master all works fine.
>
> ./bin/spark-shell --jars /tmp/libs/mysql-jdbc.jar --master yarn-client
>
> sqlContext.load("jdbc", Map("url" ->
> "jdbc:mysql://mysqlsvr:3306/MyDB;user=usr;password=pwd", "driver" ->
> "com.mysql.jdbc.Driver", "dbtable" -> "MY_TBL”))
>
>
> This throws a class not found exception when running with Spark SQL. But
> when trying to get the driver class on the workers or driver the class is
> found no problems. So I'm guessing this is some problem with the primordial
> class loader/Java security in the DriverManager and the class loader used in
> Spark SQL when running on YARN.
>
> Any ideas? The only thing I have found that works is merging my mysql adbc
> driver into the Spark assembly JAR thats shipped to YARN. Because adding
> with --jars doesn't work.
>
> Cheers!
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
