[ https://issues.apache.org/jira/browse/SPARK-928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265805#comment-15265805 ]
Sandeep Singh commented on SPARK-928: ------------------------------------- [~joshrosen] I've started working on it. I tried benchmarking the difference between unsafe kryo and our current impl. and then we can have a spark.kryo.useUnsafe flag as Matei has mentioned. {code:title=Without Kryo UnSafe|borderStyle=solid} Java HotSpot(TM) 64-Bit Server VM 1.8.0_60-b27 on Mac OS X 10.11.4 Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz Serialize and then deserialize: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------- primitive:Long 1 / 4 11223.1 0.1 1.0X primitive:Double 1 / 1 19409.0 0.1 1.7X Array:Long 38 / 49 412.4 2.4 0.0X Array:Double 25 / 35 631.4 1.6 0.1X Map of string->Double 2651 / 2766 5.9 168.6 0.0X {code} {code:title=With Kryo UnSafe|borderStyle=solid} Java HotSpot(TM) 64-Bit Server VM 1.8.0_60-b27 on Mac OS X 10.11.4 Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz Serialize and then deserialize: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------- primitive:Long 1 / 3 15872.0 0.1 1.0X primitive:Double 1 / 1 17769.7 0.1 1.1X Array:Long 24 / 42 642.3 1.6 0.0X Array:Double 22 / 26 719.4 1.4 0.0X Map of string->Double 2560 / 2582 6.1 162.8 0.0X {code} You can find the code for benchmarking here (https://github.com/techaddict/spark/commit/46fa44141c849ca15bbd6136cea2fa52bd927da2), very ugly right now but will improve it(add more benchmarks) before creating a PR. > Add support for Unsafe-based serializer in Kryo 2.22 > ---------------------------------------------------- > > Key: SPARK-928 > URL: https://issues.apache.org/jira/browse/SPARK-928 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Reporter: Matei Zaharia > Priority: Minor > Labels: starter > > This can reportedly be quite a bit faster, but it also requires Chill to > update its Kryo dependency. Once that happens we should add a > spark.kryo.useUnsafe flag. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org