Hi
Actually, I'm using spark 1.5.1. I have it in standalone mode on my laptop
for testing purposes at the moment. I have no doubt that if I were on a
cluster, I'd have to put the jar on all the workers although I will point
out that is really irritating to have to do that having built a fat jar
containing all dependencies.
Anyway, I have solved my issue and it turned out to be something utterly
trivial. The spark-submit script requires settings, like --jars and
--driver-class-path, to come as the first parameter. So my submit command
had to be:
./bin/spark-submit --driver-class-path ~/path/to/mysql-connector-
java-5.1.37-bin.jar ~/path/to/scala/project/target/scala-2.10/complete.jar
Thanks
Dean
On 29 October 2015 at 11:14, Deenar Toraskar
wrote:
> Hi Dean
>
> I guess you are using Spark 1.3.
>
>
>- The JDBC driver class must be visible to the primordial class loader
>on the client session and on all executors. This is because Java’s
>DriverManager class does a security check that results in it ignoring all
>drivers not visible to the primordial class loader when one goes to open a
>connection. One convenient way to do this is to modify compute_classpath.sh
>on all worker nodes to include your driver JARs.
>
> Take a look at this https://issues.apache.org/jira/browse/SPARK-6913 and
> see
> http://stackoverflow.com/questions/30221677/spark-sql-postgresql-jdbc-classpath-issues
> .
>
>
> Regards
> Deenar
>
> *Think Reactive Ltd*
> deenar.toras...@thinkreactive.co.uk
> 07714140812
>
>
>
> On 29 October 2015 at 10:34, dean.wood wrote:
>
>> I'm having a problem building a spark jar with scala. It's a really simple
>> thing, I want to programatically access a mysql server via JDBC and load
>> it
>>
>
> On 29 October 2015 at 10:34, dean.wood wrote:
>
>> I'm having a problem building a spark jar with scala. It's a really simple
>> thing, I want to programatically access a mysql server via JDBC and load
>> it
>> in to a spark data frame. I can get this to work in the spark shell but I
>> cannot package a jar that works with spark submit. It will package but
>> when
>> running, fails with
>>
>> Exception in thread "main" java.sql.SQLException: No suitable driver found
>> for jdbc:mysql://localhost:3310/100million
>> My spark-submit command is
>>
>> ./bin/spark-submit ~/path/to/scala/project/target/scala-2.10/complete.jar
>> --driver-class-path ~/path/to/mysql-connector-java-5.1.37-bin.jar
>>
>> My build.sbt looks like
>>
>> name := "sql_querier"
>>
>> version := "1.0"
>>
>> scalaVersion := "2.10.4"
>>
>> sbtVersion := "0.13.7"
>>
>> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1" %
>> "provided"
>>
>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1" %
>> "provided"
>>
>> libraryDependencies += "mysql" % "mysql-connector-java" % "5.1.37"
>>
>> assemblyJarName in assembly := "complete.jar"
>>
>> mainClass in assembly := Some("sql_querier")
>>
>> offline := true
>> and my very simple code is
>>
>> import org.apache.spark.SparkContext
>> import org.apache.spark.SparkConf
>> import org.apache.spark.sql.SQLContext
>>
>> object sql_querier{
>>
>> def main(args: Array[String]) {
>>
>> val sc = new org.apache.spark.SparkContext()
>> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
>> val url="jdbc:mysql://databaseurl:portno/database"
>>
>> val prop = new java.util.Properties
>> prop.setProperty("user","myuser")
>> prop.setProperty("password","mydatabase")
>> val cats=sqlContext.read.jdbc(url, "categories", prop)
>> cats.show
>> }
>> }
>> Where I've hidden the real values for user password and database url. I've
>> also got a file in projects that adds the sbt assembly plugin and there is
>> nothing wrong with this. I've successfully used sbt assembly before with
>> this configuration. When starting a spark shell with the
>> --driver-class-path
>> option pointing to the mysql jar, I can run the commands and extract data
>> from the mysql database.
>>
>> I've tried version 5.1.34 and 5.0.8 and neither have worked. I've also
>> tried
>> changing --driver-class-path for --jar in the spark submit command and
>> adding the lines
>>
>>
>>
>> sc.addJar("/Users/dean.wood/data_science/scala/sqlconn/mysql-connector-java-5.0.8-bin.jar")
>> Class.forName("com.mysql.jdbc.Driver")
>>
>> to the scala code.
>>
>> Any clue what I am doing wrong with the build would be greatly
>> appreciated.
>>
>> Dean
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Packaging-a-jar-for-a-jdbc-connection-using-sbt-assembly-and-scala-tp25225.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>> For addition
