GitHub user patrickbrownsync opened a pull request:
https://github.com/apache/spark/pull/22855
[SPARK-25839] [Core] Implement use of KryoPool in KryoSerializer
## What changes were proposed in this pull request?
* Implement (optional) use of KryoPool in KryoSerializer, an alternative to
the existing implementation of caching a Kryo instance inside
KryoSerializerInstance
* Add config key & documentation of spark.kryo.pool in order to turn this on
* Add benchmark KryoSerializerBenchmark to compare new and old
implementation
* Add results of benchmark
## How was this patch tested?
Added new tests inside KryoSerializerSuite to test the pool implementation
as well as added the pool option to the existing regression testing for
SPARK-7766
This is my original work and I license the work to the project under the
projectâs open source license.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Blyncs/spark kryo-pool
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22855.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22855
----
commit ce5d13e0673f8574b5795d0d3df59da03118038a
Author: Patrick Brown <patrick.brown@...>
Date: 2018-04-06T18:19:52Z
use kryo pool for kryo serializer
commit a4ba88eed6d18d2df5fab609bc28210df9b5a716
Author: Patrick Brown <patrick.brown@...>
Date: 2018-04-09T21:20:16Z
fix pool serializable issue
commit 3f1c41ccc451af868a13616065676a5667d597fa
Author: Patrick Brown <patrick.brown@...>
Date: 2018-10-26T18:52:07Z
Add option to KryoSerializer to use new KryoPool based implentation
Add tests for new implementation to KryoSerializerSuite.scala
Add benchmark new vs old implemtation in KryoSerializerBenchmark.scala
Add option in Benchmark base for afterAll() shutdown code to facilitate
clean benchmark shutdown
Add documentation for spark.kryo.pool configuration option
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
