when i set spark.files.userClassPathFirst=true, i get java serialization
errors in my tasks, see below. when i set userClassPathFirst back to its
default of false, the serialization errors are gone. my spark.serializer is
KryoSerializer.
the class org.apache.hadoop.fs.Path is in the spark assembly jar, but not
in my task jars (the ones i added to the SparkConf). so looks like the
ClosureSerializer is having trouble with this class once the
ChildExecutorURLClassLoader is used? thats me just guessing.
Exception in thread "main" org.apache.spark.SparkException: Job aborted due
to stage failure: Task 1.0:5 failed 4 times, most recent failure: Exception
failure in TID 31 on host node05.tresata.com:
java.lang.NoClassDefFoundError: org/apache/hadoop/fs/Path
java.lang.Class.getDeclaredConstructors0(Native Method)
java.lang.Class.privateGetDeclaredConstructors(Class.java:2398)
java.lang.Class.getDeclaredConstructors(Class.java:1838)
java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1697)
java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:50)
java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:203)
java.security.AccessController.doPrivileged(Native Method)
java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass.java:200)
java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:556)
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1580)
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1493)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1729)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1326)
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1950)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1874)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1756)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1326)
java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
scala.collection.immutable.$colon$colon.readObject(List.scala:362)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:969)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1852)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1756)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1326)
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1950)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1874)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1756)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1326)
java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:60)
org.apache.spark.scheduler.ShuffleMapTask$.deserializeInfo(ShuffleMapTask.scala:66)
org.apache.spark.scheduler.ShuffleMapTask.readExternal(ShuffleMapTask.scala:139)
java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1795)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1754)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1326)
java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:60)
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:82)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:190)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
java.lang.Thread.run(Thread.java:662)
