[ https://issues.apache.org/jira/browse/SPARK-36627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Samuel Souza updated SPARK-36627: --------------------------------- Description: In JavaSerializer.JavaDeserializationStream we override resolveClass of ObjectInputStream to use the threads' contextClassLoader. However, we do not override resolveProxyClass, which is used when deserializing Java proxy objects, which makes spark use the wrong classloader when deserializing objects, which causes the job to fail with the following exception: {code} Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 4, <host>, executor 1): java.lang.ClassNotFoundException: <class> at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:398) at java.base/java.io.ObjectInputStream.resolveProxyClass(ObjectInputStream.java:829) at java.base/java.io.ObjectInputStream.readProxyDesc(ObjectInputStream.java:1917) ... at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76) {code} was: In JavaSerializer.JavaDeserializationStream we override resolveClass of ObjectInputStream to use the threads' contextClassLoader. However, we do not override resolveProxyClass, which is used when deserializing Java proxy objects, which makes spark use the wrong classloader when deserializing objects, which causes the job to fail with the following exception: Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 4, <host>, executor 1): java.lang.ClassNotFoundException: <class> at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:398) at java.base/java.io.ObjectInputStream.resolveProxyClass(ObjectInputStream.java:829) at java.base/java.io.ObjectInputStream.readProxyDesc(ObjectInputStream.java:1917) ... at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76) > Tasks with Java proxy objects fail to deserialize > ------------------------------------------------- > > Key: SPARK-36627 > URL: https://issues.apache.org/jira/browse/SPARK-36627 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 3.0.3 > Reporter: Samuel Souza > Priority: Minor > > In JavaSerializer.JavaDeserializationStream we override resolveClass of > ObjectInputStream to use the threads' contextClassLoader. However, we do not > override resolveProxyClass, which is used when deserializing Java proxy > objects, which makes spark use the wrong classloader when deserializing > objects, which causes the job to fail with the following exception: > {code} > Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: > Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in > stage 1.0 (TID 4, <host>, executor 1): java.lang.ClassNotFoundException: > <class> > at > java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) > at > java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) > at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) > at java.base/java.lang.Class.forName0(Native Method) > at java.base/java.lang.Class.forName(Class.java:398) > at > java.base/java.io.ObjectInputStream.resolveProxyClass(ObjectInputStream.java:829) > at > java.base/java.io.ObjectInputStream.readProxyDesc(ObjectInputStream.java:1917) > ... > at > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org