I cherry-picked the fix for SPARK-5470 and the problem has gone away. On Fri, May 1, 2015 at 9:15 AM, Akshat Aranya <aara...@gmail.com> wrote: > Yes, this class is present in the jar that was loaded in the classpath > of the executor Java process -- it wasn't even lazily added as a part > of the task execution. Schema$MyRow is a protobuf-generated class. > > After doing some digging around, I think I might be hitting up against > SPARK-5470, the fix for which hasn't been merged into 1.2, as far as I > can tell. > > On Fri, May 1, 2015 at 9:05 AM, Ted Yu <yuzhih...@gmail.com> wrote: >> bq. Caused by: java.lang.ClassNotFoundException: com.example.Schema$MyRow >> >> So the above class is in the jar which was in the classpath ? >> Can you tell us a bit more about Schema$MyRow ? >> >> On Fri, May 1, 2015 at 8:05 AM, Akshat Aranya <aara...@gmail.com> wrote: >>> >>> Hi, >>> >>> I'm getting a ClassNotFoundException at the executor when trying to >>> register a class for Kryo serialization: >>> >>> java.lang.reflect.InvocationTargetException >>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>> Method) >>> at >>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) >>> at >>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526) >>> at org.apache.spark.SparkEnv$.instantiateClass$1(SparkEnv.scala:243) >>> at >>> org.apache.spark.SparkEnv$.instantiateClassFromConf$1(SparkEnv.scala:254) >>> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:257) >>> at org.apache.spark.SparkEnv$.createExecutorEnv(SparkEnv.scala:182) >>> at org.apache.spark.executor.Executor.<init>(Executor.scala:87) >>> at >>> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$receiveWithLogging$1.applyOrElse(CoarseGrainedExecutorBackend.scala:61) >>> at >>> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33) >>> at >>> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33) >>> at >>> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25) >>> at >>> org.apache.spark.util.ActorLogReceive$$anon$1.apply(ActorLogReceive.scala:53) >>> at >>> org.apache.spark.util.ActorLogReceive$$anon$1.apply(ActorLogReceive.scala:42) >>> at >>> scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118) >>> at >>> org.apache.spark.util.ActorLogReceive$$anon$1.applyOrElse(ActorLogReceive.scala:42) >>> at akka.actor.Actor$class.aroundReceive(Actor.scala:465) >>> at >>> org.apache.spark.executor.CoarseGrainedExecutorBackend.aroundReceive(CoarseGrainedExecutorBackend.scala:36) >>> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) >>> at akka.actor.ActorCell.invoke(ActorCell.scala:487) >>> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) >>> at akka.dispatch.Mailbox.run(Mailbox.scala:220) >>> at >>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) >>> at >>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >>> at >>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >>> at >>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >>> at >>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >>> Caused by: org.apache.spark.SparkException: Failed to load class to >>> register with Kryo >>> at >>> org.apache.spark.serializer.KryoSerializer$$anonfun$2.apply(KryoSerializer.scala:66) >>> at >>> org.apache.spark.serializer.KryoSerializer$$anonfun$2.apply(KryoSerializer.scala:61) >>> at >>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) >>> at >>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) >>> at >>> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) >>> at >>> scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) >>> at >>> scala.collection.TraversableLike$class.map(TraversableLike.scala:244) >>> at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108) >>> at >>> org.apache.spark.serializer.KryoSerializer.<init>(KryoSerializer.scala:61) >>> ... 28 more >>> Caused by: java.lang.ClassNotFoundException: com.example.Schema$MyRow >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>> at java.lang.Class.forName0(Native Method) >>> at java.lang.Class.forName(Class.java:190) >>> at >>> org.apache.spark.serializer.KryoSerializer$$anonfun$2.apply(KryoSerializer.scala:63) >>> >>> I have verified that when the executor process is launched, my jar is in >>> the classpath of the command line of the executor. I expect the class to be >>> found by the default classloader being used at KryoSerializer.scala:63 >>> >>> Any ideas? >> >>
--------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org