Spark 1.1.0 ClassNotFoundException issue when submit with multi jars using CLUSTER MODE

[xing_bing]

HI

 

       I am using Spark 1.1.0  config with STANDALONE clusterManager and
CLUSTER deployMode. The logic is I want to submit multi jars with
spark-submit , using the �C-jars optional, I got an ClassNotFoundException
, 　by the way in my code I also use thread  context class loader to load
custom class . 

 

Strange things is that when I use CLIENT deployMode. the exception is not
throws. 

 

Can anyone explain the class loader logic of spark or the issue when using
cluster mode ?   

 

 

 

 

/10/24 14:18:30 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)

java.lang.RuntimeException: Cannot load class:
cn.cekasp.al.demo.SimpleInputFormat3

                    at
cn.cekasp.algorithm.util.ReflectUtil.findClass(ReflectUtil.java:12)

                    at
cn.cekasp.algorithm.util.ReflectUtil.newInstance(ReflectUtil.java:18)

                    at
cn.cekasp.algorithm.reader.JdbcSourceReader$1.call(JdbcSourceReader.java:95)

                    at
cn.cekasp.algorithm.reader.JdbcSourceReader$1.call(JdbcSourceReader.java:90)

                    at
org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction$1.apply(JavaP
airRDD.scala:923)

                    at
scala.collection.Iterator$$anon$11.next(Iterator.scala:328)

                    at
org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1167)

                    at
org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:904)

                    at
org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:904)

                    at
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:112
1)

                    at
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:112
1)

                    at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)

                    at org.apache.spark.scheduler.Task.run(Task.scala:54)

                    at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)

                    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)

                    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)

                    at java.lang.Thread.run(Thread.java:745)

   Caused by: java.lang.ClassNotFoundException:
cn.cekasp.al.demo.SimpleInputFormat3
                    at
java.net.URLClassLoader$1.run(URLClassLoader.java:366)
                    at
java.net.URLClassLoader$1.run(URLClassLoader.java:355)
                    at java.security.AccessController.doPrivileged(Native
Method)
                    at
java.net.URLClassLoader.findClass(URLClassLoader.java:354)
                    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
                    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
                    at java.lang.Class.forName0(Native Method)
                    at java.lang.Class.forName(Class.java:270)
                    at
cn.cekasp.algorithm.util.ReflectUtil.findClass(ReflectUtil.java:10)
                    ... 16 more