[
https://issues.apache.org/jira/browse/SPARK-28112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-28112:
------------------------------------
Assignee: Josh Rosen (was: Apache Spark)
> Fix Kryo exception perf. bottleneck in tests due to absence of ML/MLlib
> classes
> -------------------------------------------------------------------------------
>
> Key: SPARK-28112
> URL: https://issues.apache.org/jira/browse/SPARK-28112
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, Tests
> Affects Versions: 3.0.0
> Reporter: Xiao Li
> Assignee: Josh Rosen
> Priority: Major
>
> In a nutshell, it looks like the absence of ML / MLlib classes on the
> classpath causes code in KryoSerializer to throw and catch
> ClassNotFoundExceptions whenever instantiating a new serializer in
> {{newInstance()}}. This isn't a performance problem in production (since
> MLlib is on the classpath there) but it's a huge issue in tests and appears
> to account for an enormous amount of test time
> We can address this problem by reducing the total number of
> ClassNotFoundExceptions by performing the class existence checks once and
> storing the results in KryoSerializer instances rather than repeating the
> checks on each {{newInstance()}} call.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
