JDBC Drivers need to be on the system classpath. try passing --jars /path/to/local/mysql-connector.jar when you submit the job.
this will also copy the jars to each of the worker nodes and should set you straight. On Tue, Dec 22, 2015 at 11:42 AM, Igor Berman <igor.ber...@gmail.com> wrote: > imho, if you succeeded to fetch something from your mysql with same jar in > classpath, then Manifest is ok and you indeed should look at your spark sql > - jdbc configs > > On 22 December 2015 at 12:21, David Yerrington <da...@yerrington.net> > wrote: > >> Igor, I think it's available. After I extract the jar file, I see a >> directory with class files that look very relevant in "/com/mysql/jdbc". >> >> After reading this, I started to wonder if MySQL connector was really the >> problem. Perhaps it's something to do with SQLcontext? I just wired a >> test endpoint to run a very basic mysql query, outside of Spark, and it >> worked just fine (yay!). I copied and pasted this example to verify my >> MySQL connector availability, and it worked just fine: >> https://mkaz.github.io/2011/05/27/using-scala-with-jdbc-to-connect-to-mysql/ >> >> As far as the Maven manifest goes, I'm really not sure. I will research >> it though. Now I'm wondering if my mergeStrategy is to blame? I'm going >> to try there next. >> >> Thank you for the help! >> >> On Tue, Dec 22, 2015 at 1:18 AM, Igor Berman <igor.ber...@gmail.com> >> wrote: >> >>> David, can you verify that mysql connector classes indeed in your single >>> jar? >>> open it with zip tool available at your platform >>> >>> another options that might be a problem - if there is some dependency in >>> MANIFEST(not sure though this is the case of mysql connector) then it might >>> be broken after preparing single jar >>> so you need to verify that it's ok(in maven usually it's possible to >>> define merging policy for resources while creating single jar) >>> >>> On 22 December 2015 at 10:04, Vijay Kiran <m...@vijaykiran.com> wrote: >>> >>>> Can you paste your libraryDependencies from build.sbt ? >>>> >>>> ./Vijay >>>> >>>> > On 22 Dec 2015, at 06:12, David Yerrington <da...@yerrington.net> >>>> wrote: >>>> > >>>> > Hi Everyone, >>>> > >>>> > I'm building a prototype that fundamentally grabs data from a MySQL >>>> instance, crunches some numbers, and then moves it on down the pipeline. >>>> I've been using SBT with assembly tool to build a single jar for >>>> deployment. >>>> > >>>> > I've gone through the paces of stomping out many dependency problems >>>> and have come down to one last (hopefully) zinger. >>>> > >>>> > java.lang.ClassNotFoundException: Failed to load class for data >>>> source: jdbc. >>>> > >>>> > at >>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:67) >>>> > >>>> > at >>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:87) >>>> > >>>> > at >>>> org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:114) >>>> > >>>> > at org.apache.spark.sql.SQLContext.load(SQLContext.scala:1203) >>>> > >>>> > at her.recommender.getDataframe(her.recommender.scala:45) >>>> > >>>> > at her.recommender.getRecommendations(her.recommender.scala:60) >>>> > >>>> > >>>> > I'm assuming this has to do with mysql-connector because this is the >>>> problem I run into when I'm working with spark-shell and I forget to >>>> include my classpath with my mysql-connect jar file. >>>> > >>>> > I've tried: >>>> > • Using different versions of mysql-connector-java in my >>>> build.sbt file >>>> > • Copying the connector jar to my_project/src/main/lib >>>> > • Copying the connector jar to my_project/lib <-- (this is >>>> where I keep my build.sbt) >>>> > Everything loads fine and works, except my call that does >>>> "sqlContext.load("jdbc", myOptions)". I know this is a total newbie >>>> question but in my defense, I'm fairly new to Scala, and this is my first >>>> go at deploying a fat jar with sbt-assembly. >>>> > >>>> > Thanks for any advice! >>>> > >>>> > -- >>>> > David Yerrington >>>> > yerrington.net >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>> For additional commands, e-mail: user-h...@spark.apache.org >>>> >>>> >>> >> >> >> -- >> David Yerrington >> yerrington.net >> > > -- *Chris Fregly* Principal Data Solutions Engineer IBM Spark Technology Center, San Francisco, CA http://spark.tc | http://advancedspark.com
