Hi, I'm trying to access my stand alone spark app from spark-shell. I tried starting the shell by:
MASTER=local[2] ADD_JARS=/path/to/my/jar ./spark-shell The log shows that the jar file was loaded. Also, I can access and create a new instance of mypackage.MyClass. The problem is that myRDD.collect() returns RDD[MyClass], and that throws this exception: java.lang.ClassNotFoundException: mypackage.MyClass at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:423) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:356) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:622) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1593) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1514) at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1642) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1341) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369) at org.apache.spark.util.Utils$.deserialize(Utils.scala:59) at org.apache.spark.SparkContext$$anonfun$objectFile$1.apply(SparkContext.scala:573) at org.apache.spark.SparkContext$$anonfun$objectFile$1.apply(SparkContext.scala:573) at scala.collection.Iterator$$anon$21.hasNext(Iterator.scala:440) at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:702) at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:698) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:872) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:872) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:107) at org.apache.spark.scheduler.Task.run(Task.scala:53) at org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:215) at org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:50) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:182) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Does this mean that my jar was not shipped to the workers? Is this a known issue, or am I doing something wrong here?
