Does anyone know how to get the HighlyCompressedMapStatus to compile? I will try turning off kryo in 1.2.0 and hope things don't break. I want to benefit from the MapOutputTracker fix in 1.2.0.
On Tue, Mar 3, 2015 at 5:41 AM, Imran Rashid <iras...@cloudera.com> wrote: > the scala syntax for arrays is Array[T], not T[], so you want to use > something: > > kryo.register(classOf[Array[org.roaringbitmap.RoaringArray$Element]]) > kryo.register(classOf[Array[Short]]) > > nonetheless, the spark should take care of this itself. I'll look into it > later today. > > > On Mon, Mar 2, 2015 at 2:55 PM, Arun Luthra <arun.lut...@gmail.com> wrote: > >> I think this is a Java vs scala syntax issue. Will check. >> >> On Thu, Feb 26, 2015 at 8:17 PM, Arun Luthra <arun.lut...@gmail.com> >> wrote: >> >>> Problem is noted here: https://issues.apache.org/jira/browse/SPARK-5949 >>> >>> I tried this as a workaround: >>> >>> import org.apache.spark.scheduler._ >>> import org.roaringbitmap._ >>> >>> ... >>> >>> >>> kryo.register(classOf[org.roaringbitmap.RoaringBitmap]) >>> kryo.register(classOf[org.roaringbitmap.RoaringArray]) >>> kryo.register(classOf[org.roaringbitmap.ArrayContainer]) >>> >>> kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus]) >>> kryo.register(classOf[org.roaringbitmap.RoaringArray$Element]) >>> kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]]) >>> kryo.register(classOf[short[]]) >>> >>> >>> in build file: >>> >>> libraryDependencies += "org.roaringbitmap" % "RoaringBitmap" % "0.4.8" >>> >>> >>> This fails to compile: >>> >>> ...:53: identifier expected but ']' found. >>> >>> [error] >>> kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]]) >>> >>> also: >>> >>> :54: identifier expected but ']' found. >>> >>> [error] kryo.register(classOf[short[]]) >>> also: >>> >>> :51: class HighlyCompressedMapStatus in package scheduler cannot be >>> accessed in package org.apache.spark.scheduler >>> [error] >>> kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus]) >>> >>> >>> Suggestions? >>> >>> Arun >>> >> >> >
