Problem is noted here: https://issues.apache.org/jira/browse/SPARK-5949

I tried this as a workaround:

import org.apache.spark.scheduler._
import org.roaringbitmap._

...


kryo.register(classOf[org.roaringbitmap.RoaringBitmap])
    kryo.register(classOf[org.roaringbitmap.RoaringArray])
    kryo.register(classOf[org.roaringbitmap.ArrayContainer])

kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus])
    kryo.register(classOf[org.roaringbitmap.RoaringArray$Element])
    kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]])
    kryo.register(classOf[short[]])


in build file:

libraryDependencies += "org.roaringbitmap" % "RoaringBitmap" % "0.4.8"


This fails to compile:

...:53: identifier expected but ']' found.

[error]     kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]])

also:

:54: identifier expected but ']' found.

[error]     kryo.register(classOf[short[]])
also:

:51: class HighlyCompressedMapStatus in package scheduler cannot be
accessed in package org.apache.spark.scheduler
[error]
kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus])


Suggestions?

Arun

Reply via email to