Packaging a jar for a jdbc connection using sbt assembly and scala.

[dean.wood]

I'm having a problem building a spark jar with scala. It's a really simple
thing, I want to programatically access a mysql server via JDBC and load it
in to a spark data frame. I can get this to work in the spark shell but I
cannot package a jar that works with spark submit. It will package but when
running, fails with

Exception in thread "main" java.sql.SQLException: No suitable driver found
for jdbc:mysql://localhost:3310/100million
My spark-submit command is

./bin/spark-submit ~/path/to/scala/project/target/scala-2.10/complete.jar
--driver-class-path ~/path/to/mysql-connector-java-5.1.37-bin.jar

My build.sbt looks like

name := "sql_querier"

version := "1.0"

scalaVersion := "2.10.4"

sbtVersion := "0.13.7"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1" %
"provided"

libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1" %
"provided"

libraryDependencies += "mysql" % "mysql-connector-java" % "5.1.37"

assemblyJarName in assembly := "complete.jar"

mainClass in assembly := Some("sql_querier")

offline := true
and my very simple code is

import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
import org.apache.spark.sql.SQLContext

object sql_querier{

        def main(args: Array[String]) {

                val sc = new org.apache.spark.SparkContext()
                val sqlContext = new org.apache.spark.sql.SQLContext(sc)
                val url="jdbc:mysql://databaseurl:portno/database"

                val prop = new java.util.Properties
                prop.setProperty("user","myuser")
                prop.setProperty("password","mydatabase")
                val cats=sqlContext.read.jdbc(url, "categories", prop)
                cats.show
         }
 }
Where I've hidden the real values for user password and database url. I've
also got a file in projects that adds the sbt assembly plugin and there is
nothing wrong with this. I've successfully used sbt assembly before with
this configuration. When starting a spark shell with the --driver-class-path
option pointing to the mysql jar, I can run the commands and extract data
from the mysql database.

I've tried version 5.1.34 and 5.0.8 and neither have worked. I've also tried
changing --driver-class-path for --jar in the spark submit command and
adding the lines

               
sc.addJar("/Users/dean.wood/data_science/scala/sqlconn/mysql-connector-java-5.0.8-bin.jar")
                Class.forName("com.mysql.jdbc.Driver")

to the scala code.

Any clue what I am doing wrong with the build would be greatly appreciated.

Dean



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Packaging-a-jar-for-a-jdbc-connection-using-sbt-assembly-and-scala-tp25225.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org