[
https://issues.apache.org/jira/browse/SPARK-27367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823178#comment-16823178
]
Liang-Chi Hsieh commented on SPARK-27367:
-----------------------------------------
So I think the new serde API has performance advantage over old API. However,
the advantage only shows when serialized bytes are relatively big. I can see
the advantage begins to show when at least there are 100000000 partitions in
HighlyCompressedMapStatus. Seems to me it is not easy to seen such case.
> Faster RoaringBitmap Serialization with v0.8.0
> ----------------------------------------------
>
> Key: SPARK-27367
> URL: https://issues.apache.org/jira/browse/SPARK-27367
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Imran Rashid
> Priority: Major
>
> RoaringBitmap 0.8.0 adds faster serde, but also requires us to change how we
> call the serde routines slightly to take advantage of it. This is probably a
> worthwhile optimization as the every shuffle map task with a large # of
> partitions generates these bitmaps, and the driver especially has to
> deserialize many of these messages.
> See
> * https://github.com/apache/spark/pull/24264#issuecomment-479675572
> * https://github.com/RoaringBitmap/RoaringBitmap/pull/325
> * https://github.com/RoaringBitmap/RoaringBitmap/issues/319
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
