Re: Re: driver ClassNotFoundException when MySQL JDBC exceptions are thrown on executor
>> at $iwC$$iwC.(:34) >> >> at $iwC.(:36) >> >> at (:38) >> >> at .(:42) >> >> at .() >> >> at .(:7) >> >> at .() >> >> at $print() >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:606) >> >> at >> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065) >> >> at >> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340) >> >> at >> org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840) >> >> at >> org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) >> >> at >> org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819) >> >> at >> org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857) >> >> at >> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902) >> >> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814) >> >> at >> org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657) >> >> at >> org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665) >> >> at org.apache.spark.repl.SparkILoop.org >> $apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670) >> >> at >> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997) >> >> at >> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) >> >> at >> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) >> >> at >> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) >> >> at org.apache.spark.repl.SparkILoop.org >> $apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945) >> >> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059) >> >> at org.apache.spark.repl.Main$.main(Main.scala:31) >> >> at org.apache.spark.repl.Main.main(Main.scala) >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:606) >> >> at >> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674) >> >> at >> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) >> >> at >> org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) >> >> at >> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) >> >> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >> >> >> >> >> >> ThanksBest regards! >> San.Luo >> >> - 原始邮件 - >> 发件人:Xiao Li <gatorsm...@gmail.com> >> 收件人:Akhil Das <ak...@sigmoidanalytics.com> >> 抄送人:Hurshal Patel <hpatel...@gmail.com>, "user@spark.apache.org" < >> user@spark.apache.org> >> 主题:Re: driver ClassNotFoundException when MySQL JDBC exceptions are >> thrown on executor >> 日期:2015年10月22日 22点10分 >> >> A few months ago, I used the DB2 jdbc drivers. I hit a couple of issues >> when using --driver-class-path. At the end, I used the following command to >> bypass most of issues: >> >> ./bin/spark-submit --jars >> /Users/smile/db2driver/db2jcc.jar,/Users/smile/db2driver/db2jcc_license_cisuz.jar >> --master local[*] --class com.sparkEngine. >> /Users/smile/spark-1.3.1-bin-hadoop2.3/projects/SparkApps-master/spark-load-from-db/target/-1.0.jar >> >> Hopefully, it works for you. >> >> Xiao Li >> >> >> 2015-10-22 4:56 GMT-07:00 Akhil
Re: Re: driver ClassNotFoundException when MySQL JDBC exceptions are thrown on executor
arkIMain.scala:1065) > > at > org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340) > > at > org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840) > > at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) > > at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819) > > at > org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857) > > at > org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902) > > at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814) > > at > org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657) > > at > org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665) > > at org.apache.spark.repl.SparkILoop.org > $apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670) > > at > org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997) > > at > org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) > > at > org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) > > at > scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) > > at org.apache.spark.repl.SparkILoop.org > $apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945) > > at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059) > > at org.apache.spark.repl.Main$.main(Main.scala:31) > > at org.apache.spark.repl.Main.main(Main.scala) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674) > > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) > > at > org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) > > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) > > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > > > > > ThanksBest regards! > San.Luo > > - 原始邮件 - > 发件人:Xiao Li <gatorsm...@gmail.com> > 收件人:Akhil Das <ak...@sigmoidanalytics.com> > 抄送人:Hurshal Patel <hpatel...@gmail.com>, "user@spark.apache.org" < > user@spark.apache.org> > 主题:Re: driver ClassNotFoundException when MySQL JDBC exceptions are thrown > on executor > 日期:2015年10月22日 22点10分 > > A few months ago, I used the DB2 jdbc drivers. I hit a couple of issues > when using --driver-class-path. At the end, I used the following command to > bypass most of issues: > > ./bin/spark-submit --jars > /Users/smile/db2driver/db2jcc.jar,/Users/smile/db2driver/db2jcc_license_cisuz.jar > --master local[*] --class com.sparkEngine. > /Users/smile/spark-1.3.1-bin-hadoop2.3/projects/SparkApps-master/spark-load-from-db/target/-1.0.jar > > Hopefully, it works for you. > > Xiao Li > > > 2015-10-22 4:56 GMT-07:00 Akhil Das <ak...@sigmoidanalytics.com>: > > Did you try passing the mysql connector jar through --driver-class-path > > Thanks > Best Regards > > On Sat, Oct 17, 2015 at 6:33 AM, Hurshal Patel <hpatel...@gmail.com> > wrote: > > Hi all, > > I've been struggling with a particularly puzzling issue after upgrading to > Spark 1.5.1 from Spark 1.4.1. > > When I use the MySQL JDBC connector and an exception (e.g. > com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException) is thrown on > the executor, I get a ClassNotFoundException on the driver, which results > in this error (logs are abbreviated): > > 15/10/16 17:20:59 INFO SparkContext: Starting job: collect at > repro.scala:73 > ... > 15/10/16 17:20:59 INFO Executor: Running task 0.0 in stage 3.0 (TID 3) > 15/10/16 17:20:59 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID > 3) > com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException > at repro.Repro$$anonfun$main$3.apply$mcZI$sp(repro.scala:69) > ... > 15/10/16 17:20:59 WARN ThrowableSerializationWrapper: Task exception could > not be deserialized > java.lang.ClassNotFoundException: > com.mysql.jdb
Re: driver ClassNotFoundException when MySQL JDBC exceptions are thrown on executor
Did you try passing the mysql connector jar through --driver-class-path Thanks Best Regards On Sat, Oct 17, 2015 at 6:33 AM, Hurshal Patelwrote: > Hi all, > > I've been struggling with a particularly puzzling issue after upgrading to > Spark 1.5.1 from Spark 1.4.1. > > When I use the MySQL JDBC connector and an exception (e.g. > com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException) is thrown on > the executor, I get a ClassNotFoundException on the driver, which results > in this error (logs are abbreviated): > > 15/10/16 17:20:59 INFO SparkContext: Starting job: collect at > repro.scala:73 > ... > 15/10/16 17:20:59 INFO Executor: Running task 0.0 in stage 3.0 (TID 3) > 15/10/16 17:20:59 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID > 3) > com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException > at repro.Repro$$anonfun$main$3.apply$mcZI$sp(repro.scala:69) > ... > 15/10/16 17:20:59 WARN ThrowableSerializationWrapper: Task exception could > not be deserialized > java.lang.ClassNotFoundException: > com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > ... > 15/10/16 17:20:59 ERROR TaskResultGetter: Could not deserialize > TaskEndReason: ClassNotFound with classloader > org.apache.spark.util.MutableURLClassLoader@7f08a6b1 > 15/10/16 17:20:59 WARN TaskSetManager: Lost task 0.0 in stage 3.0 (TID 3, > localhost): UnknownReason > 15/10/16 17:20:59 ERROR TaskSetManager: Task 0 in stage 3.0 failed 1 > times; aborting job > 15/10/16 17:20:59 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks > have all completed, from pool > 15/10/16 17:20:59 INFO TaskSchedulerImpl: Cancelling stage 3 > 15/10/16 17:20:59 INFO DAGScheduler: ResultStage 3 (collect at > repro.scala:73) failed in 0.012 s > 15/10/16 17:20:59 INFO DAGScheduler: Job 3 failed: collect at > repro.scala:73, took 0.018694 s > > In Spark 1.4.1, I get the following (logs are abbreviated): > 15/10/16 17:42:41 INFO SparkContext: Starting job: collect at > repro.scala:53 > ... > 15/10/16 17:42:41 INFO Executor: Running task 0.0 in stage 2.0 (TID 2) > 15/10/16 17:42:41 ERROR Executor: Exception in task 0.0 in stage 2.0 (TID > 2) > com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException > at repro.Repro$$anonfun$main$2.apply$mcZI$sp(repro.scala:49) > ... > 15/10/16 17:42:41 WARN TaskSetManager: Lost task 0.0 in stage 2.0 (TID 2, > localhost): com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException > at repro.Repro$$anonfun$main$2.apply$mcZI$sp(repro.scala:49) > ... > > 15/10/16 17:42:41 ERROR TaskSetManager: Task 0 in stage 2.0 failed 1 > times; aborting job > 15/10/16 17:42:41 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks > have all completed, from pool > 15/10/16 17:42:41 INFO TaskSchedulerImpl: Cancelling stage 2 > 15/10/16 17:42:41 INFO DAGScheduler: ResultStage 2 (collect at > repro.scala:53) failed in 0.016 s > 15/10/16 17:42:41 INFO DAGScheduler: Job 2 failed: collect at > repro.scala:53, took 0.024584 s > > > I have seriously screwed up somewhere or this is a change in behavior that > I have not been able to find in the documentation. For those that are > interested, a full repro and logs follow. > > Hurshal > > --- > > I am running this on Spark 1.5.1+Hadoop 2.6. I have tried this in various > combinations of > * local/standalone mode > * putting mysql on the classpath with --jars/building a fat jar with > mysql in it/manually running sc.addJar on the mysql jar > * --deploy-mode client/--deploy-mode cluster > but nothing seems to change. > > > > Here is an example invocation, and the accompanying source code: > > $ ./bin/spark-submit --master local --deploy-mode client --class > repro.Repro /home/nix/repro/target/scala-2.10/repro-assembly-0.0.1.jar > Using Spark's default log4j profile: > org/apache/spark/log4j-defaults.properties > 15/10/16 17:40:53 INFO SparkContext: Running Spark version 1.5.1 > 15/10/16 17:40:53 WARN NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 15/10/16 17:40:53 WARN Utils: Your hostname, choochootrain resolves to a > loopback address: 127.0.1.1; using 10.0.1.97 instead (on interface wlan0) > 15/10/16 17:40:53 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to > another address > 15/10/16 17:40:53 INFO SecurityManager: Changing view acls to: root > 15/10/16 17:40:53 INFO SecurityManager: Changing modify acls to: root > 15/10/16 17:40:53 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(root); users > with modify permissions: Set(root) > 15/10/16 17:40:54 INFO Slf4jLogger: Slf4jLogger started > 15/10/16 17:40:54 INFO Remoting: Starting remoting > 15/10/16 17:40:54 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://sparkDriver@10.0.1.97:48116] > 15/10/16 17:40:54 INFO Utils: Successfully started service 'sparkDriver' > on port 48116. >
Re: driver ClassNotFoundException when MySQL JDBC exceptions are thrown on executor
A few months ago, I used the DB2 jdbc drivers. I hit a couple of issues when using --driver-class-path. At the end, I used the following command to bypass most of issues: ./bin/spark-submit --jars /Users/smile/db2driver/db2jcc.jar,/Users/smile/db2driver/db2jcc_license_cisuz.jar --master local[*] --class com.sparkEngine. /Users/smile/spark-1.3.1-bin-hadoop2.3/projects/SparkApps-master/spark-load-from-db/target/-1.0.jar Hopefully, it works for you. Xiao Li 2015-10-22 4:56 GMT-07:00 Akhil Das: > Did you try passing the mysql connector jar through --driver-class-path > > Thanks > Best Regards > > On Sat, Oct 17, 2015 at 6:33 AM, Hurshal Patel > wrote: > >> Hi all, >> >> I've been struggling with a particularly puzzling issue after upgrading >> to Spark 1.5.1 from Spark 1.4.1. >> >> When I use the MySQL JDBC connector and an exception (e.g. >> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException) is thrown on >> the executor, I get a ClassNotFoundException on the driver, which results >> in this error (logs are abbreviated): >> >> 15/10/16 17:20:59 INFO SparkContext: Starting job: collect at >> repro.scala:73 >> ... >> 15/10/16 17:20:59 INFO Executor: Running task 0.0 in stage 3.0 (TID 3) >> 15/10/16 17:20:59 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID >> 3) >> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException >> at repro.Repro$$anonfun$main$3.apply$mcZI$sp(repro.scala:69) >> ... >> 15/10/16 17:20:59 WARN ThrowableSerializationWrapper: Task exception >> could not be deserialized >> java.lang.ClassNotFoundException: >> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException >> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >> ... >> 15/10/16 17:20:59 ERROR TaskResultGetter: Could not deserialize >> TaskEndReason: ClassNotFound with classloader >> org.apache.spark.util.MutableURLClassLoader@7f08a6b1 >> 15/10/16 17:20:59 WARN TaskSetManager: Lost task 0.0 in stage 3.0 (TID 3, >> localhost): UnknownReason >> 15/10/16 17:20:59 ERROR TaskSetManager: Task 0 in stage 3.0 failed 1 >> times; aborting job >> 15/10/16 17:20:59 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose >> tasks have all completed, from pool >> 15/10/16 17:20:59 INFO TaskSchedulerImpl: Cancelling stage 3 >> 15/10/16 17:20:59 INFO DAGScheduler: ResultStage 3 (collect at >> repro.scala:73) failed in 0.012 s >> 15/10/16 17:20:59 INFO DAGScheduler: Job 3 failed: collect at >> repro.scala:73, took 0.018694 s >> >> In Spark 1.4.1, I get the following (logs are abbreviated): >> 15/10/16 17:42:41 INFO SparkContext: Starting job: collect at >> repro.scala:53 >> ... >> 15/10/16 17:42:41 INFO Executor: Running task 0.0 in stage 2.0 (TID 2) >> 15/10/16 17:42:41 ERROR Executor: Exception in task 0.0 in stage 2.0 (TID >> 2) >> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException >> at repro.Repro$$anonfun$main$2.apply$mcZI$sp(repro.scala:49) >> ... >> 15/10/16 17:42:41 WARN TaskSetManager: Lost task 0.0 in stage 2.0 (TID 2, >> localhost): com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException >> at repro.Repro$$anonfun$main$2.apply$mcZI$sp(repro.scala:49) >> ... >> >> 15/10/16 17:42:41 ERROR TaskSetManager: Task 0 in stage 2.0 failed 1 >> times; aborting job >> 15/10/16 17:42:41 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose >> tasks have all completed, from pool >> 15/10/16 17:42:41 INFO TaskSchedulerImpl: Cancelling stage 2 >> 15/10/16 17:42:41 INFO DAGScheduler: ResultStage 2 (collect at >> repro.scala:53) failed in 0.016 s >> 15/10/16 17:42:41 INFO DAGScheduler: Job 2 failed: collect at >> repro.scala:53, took 0.024584 s >> >> >> I have seriously screwed up somewhere or this is a change in behavior >> that I have not been able to find in the documentation. For those that are >> interested, a full repro and logs follow. >> >> Hurshal >> >> --- >> >> I am running this on Spark 1.5.1+Hadoop 2.6. I have tried this in various >> combinations of >> * local/standalone mode >> * putting mysql on the classpath with --jars/building a fat jar with >> mysql in it/manually running sc.addJar on the mysql jar >> * --deploy-mode client/--deploy-mode cluster >> but nothing seems to change. >> >> >> >> Here is an example invocation, and the accompanying source code: >> >> $ ./bin/spark-submit --master local --deploy-mode client --class >> repro.Repro /home/nix/repro/target/scala-2.10/repro-assembly-0.0.1.jar >> Using Spark's default log4j profile: >> org/apache/spark/log4j-defaults.properties >> 15/10/16 17:40:53 INFO SparkContext: Running Spark version 1.5.1 >> 15/10/16 17:40:53 WARN NativeCodeLoader: Unable to load native-hadoop >> library for your platform... using builtin-java classes where applicable >> 15/10/16 17:40:53 WARN Utils: Your hostname, choochootrain resolves to a >> loopback address: 127.0.1.1; using 10.0.1.97 instead (on interface wlan0) >> 15/10/16 17:40:53 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to >> another address >> 15/10/16