Igor, I think it's available. After I extract the jar file, I see a directory with class files that look very relevant in "/com/mysql/jdbc".
After reading this, I started to wonder if MySQL connector was really the problem. Perhaps it's something to do with SQLcontext? I just wired a test endpoint to run a very basic mysql query, outside of Spark, and it worked just fine (yay!). I copied and pasted this example to verify my MySQL connector availability, and it worked just fine: https://mkaz.github.io/2011/05/27/using-scala-with-jdbc-to-connect-to-mysql/ As far as the Maven manifest goes, I'm really not sure. I will research it though. Now I'm wondering if my mergeStrategy is to blame? I'm going to try there next. Thank you for the help! On Tue, Dec 22, 2015 at 1:18 AM, Igor Berman <igor.ber...@gmail.com> wrote: > David, can you verify that mysql connector classes indeed in your single > jar? > open it with zip tool available at your platform > > another options that might be a problem - if there is some dependency in > MANIFEST(not sure though this is the case of mysql connector) then it might > be broken after preparing single jar > so you need to verify that it's ok(in maven usually it's possible to > define merging policy for resources while creating single jar) > > On 22 December 2015 at 10:04, Vijay Kiran <m...@vijaykiran.com> wrote: > >> Can you paste your libraryDependencies from build.sbt ? >> >> ./Vijay >> >> > On 22 Dec 2015, at 06:12, David Yerrington <da...@yerrington.net> >> wrote: >> > >> > Hi Everyone, >> > >> > I'm building a prototype that fundamentally grabs data from a MySQL >> instance, crunches some numbers, and then moves it on down the pipeline. >> I've been using SBT with assembly tool to build a single jar for deployment. >> > >> > I've gone through the paces of stomping out many dependency problems >> and have come down to one last (hopefully) zinger. >> > >> > java.lang.ClassNotFoundException: Failed to load class for data source: >> jdbc. >> > >> > at >> org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:67) >> > >> > at >> org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:87) >> > >> > at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:114) >> > >> > at org.apache.spark.sql.SQLContext.load(SQLContext.scala:1203) >> > >> > at her.recommender.getDataframe(her.recommender.scala:45) >> > >> > at her.recommender.getRecommendations(her.recommender.scala:60) >> > >> > >> > I'm assuming this has to do with mysql-connector because this is the >> problem I run into when I'm working with spark-shell and I forget to >> include my classpath with my mysql-connect jar file. >> > >> > I've tried: >> > • Using different versions of mysql-connector-java in my >> build.sbt file >> > • Copying the connector jar to my_project/src/main/lib >> > • Copying the connector jar to my_project/lib <-- (this is where >> I keep my build.sbt) >> > Everything loads fine and works, except my call that does >> "sqlContext.load("jdbc", myOptions)". I know this is a total newbie >> question but in my defense, I'm fairly new to Scala, and this is my first >> go at deploying a fat jar with sbt-assembly. >> > >> > Thanks for any advice! >> > >> > -- >> > David Yerrington >> > yerrington.net >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> > -- David Yerrington yerrington.net
