Repository: spark
Updated Branches:
refs/heads/branch-1.3 4ff8855e8 -> 8007a4f20
[SPARK-5470][Core]use defaultClassLoader to load classes in KryoSerializer
Now KryoSerializer load classes of classesToRegister at the time of its
initialization. when we set spark.kryo.classesToRegister=class1, it will throw
SparkException("Failed to load class to register with Kryo".
because in KryoSerializer's initialization, classLoader cannot include class of
user's jars.
we need to use defaultClassLoader of Serializer in newKryo(), because executor
will reset defaultClassLoader of Serializer after Serializer's initialization.
thank zzcclp for reporting it to me.
Author: lianhuiwang <lianhuiwan...@gmail.com>
Closes #4258 from lianhuiwang/SPARK-5470 and squashes the following commits:
73b719f [lianhuiwang] do the splitting and filtering during initialization
64cf306 [lianhuiwang] use defaultClassLoader to load classes of
classesToRegister in KryoSerializer
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8007a4f2
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/8007a4f2
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/8007a4f2
Branch: refs/heads/branch-1.3
Commit: 8007a4f206ecf2c9e1823cb2bd5fff7a2dfd02ce
Parents: 4ff8855
Author: lianhuiwang <lianhuiwan...@gmail.com>
Authored: Fri Feb 6 11:00:35 2015 +0000
Committer: Josh Rosen <joshro...@databricks.com>
Committed: Fri Feb 6 10:50:51 2015 -0800
----------------------------------------------------------------------
.../org/apache/spark/serializer/KryoSerializer.scala | 11 ++---------
1 file changed, 2 insertions(+), 9 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/8007a4f2/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
----------------------------------------------------------------------
diff --git
a/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
b/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
index d56e23c..02158aa 100644
--- a/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
+++ b/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
@@ -58,14 +58,6 @@ class KryoSerializer(conf: SparkConf)
private val classesToRegister = conf.get("spark.kryo.classesToRegister", "")
.split(',')
.filter(!_.isEmpty)
- .map { className =>
- try {
- Class.forName(className)
- } catch {
- case e: Exception =>
- throw new SparkException("Failed to load class to register with
Kryo", e)
- }
- }
def newKryoOutput() = new KryoOutput(bufferSize, math.max(bufferSize,
maxBufferSize))
@@ -97,7 +89,8 @@ class KryoSerializer(conf: SparkConf)
// Use the default classloader when calling the user registrator.
Thread.currentThread.setContextClassLoader(classLoader)
// Register classes given through spark.kryo.classesToRegister.
- classesToRegister.foreach { clazz => kryo.register(clazz) }
+ classesToRegister
+ .foreach { className => kryo.register(Class.forName(className, true,
classLoader)) }
// Allow the user to register their own classes by setting
spark.kryo.registrator.
userRegistrator
.map(Class.forName(_, true,
classLoader).newInstance().asInstanceOf[KryoRegistrator])
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
