Thomas Graves created SPARK-10910:
-------------------------------------
Summary: spark.{executor,driver}.userClassPathFirst don't work for
kryo (probably others)
Key: SPARK-10910
URL: https://issues.apache.org/jira/browse/SPARK-10910
Project: Spark
Issue Type: Bug
Components: Spark Core, YARN
Affects Versions: 1.5.1
Reporter: Thomas Graves
Trying to use spark.{executor,driver}.userClassPathFirst to put a newer version
of kryo in doesn't work. Note I was running on YARN.
There is a bug in kryo 1.21 that spark is using which is fixed in kryo 1.24. A
customer tried to use the spark.{executor,driver}.userClassPathFirst to include
the newer version of kryo but it threw the following exception:
15/09/29 21:36:43 ERROR yarn.ApplicationMaster: User class threw exception:
java.lang.LinkageError: loader constraint violation: loader (instance of
org/apache/spark/util/ChildFirstURLClassLoader) previously initiated loading
for a different type with name "com/esotericsoftware/kryo/Kryo"
java.lang.LinkageError: loader constraint violation: loader (instance of
org/apache/spark/util/ChildFirstURLClassLoader) previously initiated loading
for a different type with name "com/esotericsoftware/kryo/Kryo"
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
The issue here is that the Spark Driver instantiates a kryo class in SparkEnv:
val serializer = instantiateClassFromConf[Serializer](
"spark.serializer", "org.apache.spark.serializer.JavaSerializer")
logDebug(s"Using serializer: ${serializer.getClass}")
It uses whatever version is in the spark assembly jar.
Then on YARN in the ApplicationMaster code before it starts the user
application is handles the user classpath first to be the
ChildFirstURLClassLoader, which is later used when kryo is needed. This tries
to load the newer version of kryo from the user jar and it throws the exception
above.
I'm sure this could happen with any number of other classes that got loaded by
Spark before we try to run the user application code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
