Patrick Brown created SPARK-25839:
-------------------------------------
Summary: Implement use of KryoPool in KryoSerializer
Key: SPARK-25839
URL: https://issues.apache.org/jira/browse/SPARK-25839
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 2.3.2, 2.3.1, 2.0.2
Reporter: Patrick Brown
The current implementation of KryoSerializer does not use KryoPool, which is
recommended by Kryo due to the creation of a Kryo instance being slow.
The current implementation references the KryoSerializerInstance private
variable cachedKryo as effectively being a pool of size 1. However (in my
admittedly somewhat limited research) it seems that frequently (such as in the
ClosureCleaner ensureSerializable method) a new instance of
KryoSerializerInstance is created, which in turn forces a new instance of Kryo
itself to be created, this instance is then dropped from scope, causing the
"pool" not to be re-used.
I have a small set of proposed changes we have been using on an internal
production application (running 24x7 for 6+ months, processing 10k+ jobs a day)
which implements using a KryoPool inside KryoSerializer which is then used by
each KryoSerializerInstance to borrow a Kryo instance.
I believe this is mainly a performance improvement for applications processing
a large number of small jobs, where the cost of instantiating Kryo instances is
a larger portion of execution time compared to larger jobs.
I have discussed this proposed change in the dev mailing list and it was
suggested I create this issue and a PR. It was also suggested I accompany that
with some performance metrics, which it is my plan to do.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
