Hi Actually, I'm using spark 1.5.1. I have it in standalone mode on my laptop for testing purposes at the moment. I have no doubt that if I were on a cluster, I'd have to put the jar on all the workers although I will point out that is really irritating to have to do that having built a fat jar containing all dependencies.
Anyway, I have solved my issue and it turned out to be something utterly trivial. The spark-submit script requires settings, like --jars and --driver-class-path, to come as the first parameter. So my submit command had to be: ./bin/spark-submit --driver-class-path ~/path/to/mysql-connector- java-5.1.37-bin.jar ~/path/to/scala/project/target/scala-2.10/complete.jar Thanks Dean On 29 October 2015 at 11:14, Deenar Toraskar <deenar.toras...@gmail.com> wrote: > Hi Dean > > I guess you are using Spark 1.3. > > > - The JDBC driver class must be visible to the primordial class loader > on the client session and on all executors. This is because Java’s > DriverManager class does a security check that results in it ignoring all > drivers not visible to the primordial class loader when one goes to open a > connection. One convenient way to do this is to modify compute_classpath.sh > on all worker nodes to include your driver JARs. > > Take a look at this https://issues.apache.org/jira/browse/SPARK-6913 and > see > http://stackoverflow.com/questions/30221677/spark-sql-postgresql-jdbc-classpath-issues > . > > > Regards > Deenar > > *Think Reactive Ltd* > deenar.toras...@thinkreactive.co.uk > 07714140812 > > > > On 29 October 2015 at 10:34, dean.wood <dean.w...@sparkol.com> wrote: > >> I'm having a problem building a spark jar with scala. It's a really simple >> thing, I want to programatically access a mysql server via JDBC and load >> it >> > > On 29 October 2015 at 10:34, dean.wood <dean.w...@sparkol.com> wrote: > >> I'm having a problem building a spark jar with scala. It's a really simple >> thing, I want to programatically access a mysql server via JDBC and load >> it >> in to a spark data frame. I can get this to work in the spark shell but I >> cannot package a jar that works with spark submit. It will package but >> when >> running, fails with >> >> Exception in thread "main" java.sql.SQLException: No suitable driver found >> for jdbc:mysql://localhost:3310/100million >> My spark-submit command is >> >> ./bin/spark-submit ~/path/to/scala/project/target/scala-2.10/complete.jar >> --driver-class-path ~/path/to/mysql-connector-java-5.1.37-bin.jar >> >> My build.sbt looks like >> >> name := "sql_querier" >> >> version := "1.0" >> >> scalaVersion := "2.10.4" >> >> sbtVersion := "0.13.7" >> >> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1" % >> "provided" >> >> libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1" % >> "provided" >> >> libraryDependencies += "mysql" % "mysql-connector-java" % "5.1.37" >> >> assemblyJarName in assembly := "complete.jar" >> >> mainClass in assembly := Some("sql_querier") >> >> offline := true >> and my very simple code is >> >> import org.apache.spark.SparkContext >> import org.apache.spark.SparkConf >> import org.apache.spark.sql.SQLContext >> >> object sql_querier{ >> >> def main(args: Array[String]) { >> >> val sc = new org.apache.spark.SparkContext() >> val sqlContext = new org.apache.spark.sql.SQLContext(sc) >> val url="jdbc:mysql://databaseurl:portno/database" >> >> val prop = new java.util.Properties >> prop.setProperty("user","myuser") >> prop.setProperty("password","mydatabase") >> val cats=sqlContext.read.jdbc(url, "categories", prop) >> cats.show >> } >> } >> Where I've hidden the real values for user password and database url. I've >> also got a file in projects that adds the sbt assembly plugin and there is >> nothing wrong with this. I've successfully used sbt assembly before with >> this configuration. When starting a spark shell with the >> --driver-class-path >> option pointing to the mysql jar, I can run the commands and extract data >> from the mysql database. >> >> I've tried version 5.1.34 and 5.0.8 and neither have worked. I've also >> tried >> changing --driver-class-path for --jar in the spark submit command and >> adding the lines >> >> >> >> sc.addJar("/Users/dean.wood/data_science/scala/sqlconn/mysql-connector-java-5.0.8-bin.jar") >> Class.forName("com.mysql.jdbc.Driver") >> >> to the scala code. >> >> Any clue what I am doing wrong with the build would be greatly >> appreciated. >> >> Dean >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Packaging-a-jar-for-a-jdbc-connection-using-sbt-assembly-and-scala-tp25225.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> > -- Dean Wood Data Scientist, Sparkol Sparkol <http://www.sparkol.com/> | +44 1275 400262 | Yeo Bank, Kenn Road, Clevedon, BS21 6UW Facebook <https://www.facebook.com/SparkolHQ> | YouTube <https://www.youtube.com/user/VideoScribetv?feature=mhee> | Twitter <https://twitter.com/sparkolhq> Download Tawe free for your desktop <http://tawe.co/?_sacid=tawdskemaemaegedm1509support+tawe> Make an instant video from a simple image
