[
https://issues.apache.org/jira/browse/SPARK-1863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
koert kuipers updated SPARK-1863:
---------------------------------
Description:
See here:
http://apache-spark-user-list.1001560.n3.nabble.com/java-serialization-errors-with-spark-files-userClassPathFirst-true-td5832.html
The issue seems to be that within ChildExecutorURLClassLoader userClassLoader
has no visibility on classes managed by parentClassLoader because their is no
parent/child relationship. What this means that if a class is loaded by
userClassLoader and it refers to a class loaded by parentClassLoader you get a
NoClassDefFoundError.
was:
See here:
http://apache-spark-user-list.1001560.n3.nabble.com/java-serialization-errors-with-spark-files-userClassPathFirst-true-td5832.html
The issue seems to be that within ChildExecutorURLClassLoader userClassLoader
has no visibility on classes managed by parentClassLoader because their is no
parent/child relationship. What this means that if a class is loaded by
userClassLoader and it refers to a class loaded by parentClassLoader you get a
NoClassDefFoundError.
When i addresses this by creating a new version of ChildExecutorURLClassLoader
that does have the proper parent-child relationship and reverses the loading
order inside loadClass the class loading seemed to work fine but now classes
like SparkEnv were loaded by ChildExecutorURLClassLoader leading to NPEs on
SparkEnv.get()
To verify that the issue was that SparkEnv was now loaded by
ChildExecutorURLClassLoader i forced SparkEnv to be loaded by the parent
classloader. That didnt help. Then i forced all spark classes to be loaded by
parent classloader and that did help. But it causes even bigger problems:
java.lang.LinkageError: loader constraint violation: when resolving overridden
method
"myclass.MyRDD.compute(Lorg/apache/spark/Partition;Lorg/apache/spark/TaskContext;)Lscala/collection/Iterator;"
the class loader (instance of
org/apache/spark/executor/ChildExecutorURLClassLoader) of the current class,
myclass/MyRDD, and its superclass loader (instance of
sun/misc/Launcher$AppClassLoader), have different Class objects for the type
TaskContext;)Lscala/collection/Iterator; used in the signature
> Allowing user jars to take precedence over Spark jars does not work as
> expected
> -------------------------------------------------------------------------------
>
> Key: SPARK-1863
> URL: https://issues.apache.org/jira/browse/SPARK-1863
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.0.0
> Reporter: koert kuipers
> Priority: Minor
>
> See here:
> http://apache-spark-user-list.1001560.n3.nabble.com/java-serialization-errors-with-spark-files-userClassPathFirst-true-td5832.html
> The issue seems to be that within ChildExecutorURLClassLoader userClassLoader
> has no visibility on classes managed by parentClassLoader because their is no
> parent/child relationship. What this means that if a class is loaded by
> userClassLoader and it refers to a class loaded by parentClassLoader you get
> a NoClassDefFoundError.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
