[
https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807400#comment-16807400
]
Lantao Jin commented on SPARK-27216:
------------------------------------
[~irashid] Not yet. It's petty challenging to find out it without JIRA This
fixing is much important than performance since it causes job failed (2.4 and
above) or data quality problem (2.1~2.3 AFAIK) when using unsafe
KryoSerializer. I don't know which version is the proper one, so I simply use
latest version. Anyway, Spark is not a heavy user on this lib. Why not replace
it is due to compatibility consideration.
> Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue
> -----------------------------------------------------------------
>
> Key: SPARK-27216
> URL: https://issues.apache.org/jira/browse/SPARK-27216
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.3.3, 2.4.0, 3.0.0
> Reporter: Lantao Jin
> Priority: Major
>
> HighlyCompressedMapStatus uses RoaringBitmap to record the empty blocks. But
> RoaringBitmap-0.5.11 couldn't be ser/deser with unsafe KryoSerializer.
> We can use below UT to reproduce:
> {code}
> test("kryo serialization with RoaringBitmap") {
> val bitmap = new RoaringBitmap
> bitmap.add(1787)
> val safeSer = new KryoSerializer(conf).newInstance()
> val bitmap2 : RoaringBitmap =
> safeSer.deserialize(safeSer.serialize(bitmap))
> assert(bitmap2.equals(bitmap))
> conf.set("spark.kryo.unsafe", "true")
> val unsafeSer = new KryoSerializer(conf).newInstance()
> val bitmap3 : RoaringBitmap =
> unsafeSer.deserialize(unsafeSer.serialize(bitmap))
> assert(bitmap3.equals(bitmap)) // this will fail
> }
> {code}
> Upgrade to latest version 0.7.45 to fix it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
