[ https://issues.apache.org/jira/browse/SPARK-5949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14345259#comment-14345259 ]
Apache Spark commented on SPARK-5949: ------------------------------------- User 'squito' has created a pull request for this issue: https://github.com/apache/spark/pull/4877 > Driver program has to register roaring bitmap classes used by spark with Kryo > when number of partitions is greater than 2000 > ---------------------------------------------------------------------------------------------------------------------------- > > Key: SPARK-5949 > URL: https://issues.apache.org/jira/browse/SPARK-5949 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.2.0 > Reporter: Peter Torok > Assignee: Imran Rashid > Labels: kryo, partitioning, serialization > > When more than 2000 partitions are being used with Kryo, the following > classes need to be registered by driver program: > - org.apache.spark.scheduler.HighlyCompressedMapStatus > - org.roaringbitmap.RoaringBitmap > - org.roaringbitmap.RoaringArray > - org.roaringbitmap.ArrayContainer > - org.roaringbitmap.RoaringArray$Element > - org.roaringbitmap.RoaringArray$Element[] > - short[] > Our project doesn't have dependency on roaring bitmap and > HighlyCompressedMapStatus is intended for internal spark usage. Spark should > take care of this registration when Kryo is used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org