Giving a bit more detail on the error would make it a lot easier for others to help you out. Eg., in this case, it would have helped if included your actual compile error.
In any case, I'm assuming your issue is b/c that class if private to spark. You can sneak around that by using Class.forName("stringOfClassName") instead: scala> classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus] > <console>:8: error: class HighlyCompressedMapStatus in package scheduler > cannot be accessed in package org.apache.spark.scheduler > classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus] > ^ > scala> > Class.forName("org.apache.spark.scheduler.HighlyCompressedMapStatus") > res1: Class[_] = class org.apache.spark.scheduler.HighlyCompressedMapStatus hope this helps, Imran On Thu, Mar 12, 2015 at 12:47 PM, Arun Luthra <arun.lut...@gmail.com> wrote: > I'm using a pre-built Spark; I'm not trying to compile Spark. > > The compile error appears when I try to register HighlyCompressedMapStatus > in my program: > > kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus]) > > If I don't register it, I get a runtime error saying that it needs to be > registered (the error is only when I turn on kryo). > > However the code is running smoothly with kryo turned off. > > On Wed, Mar 11, 2015 at 5:38 PM, Imran Rashid <iras...@cloudera.com> > wrote: > >> I'm not sure what you mean. Are you asking how you can recompile all of >> spark and deploy it, instead of using one of the pre-built versions? >> >> https://spark.apache.org/docs/latest/building-spark.html >> >> Or are you seeing compile problems specifically w/ >> HighlyCompressedMapStatus? The code compiles fine, so I'm not sure what >> problem you are running into -- we'd need a lot more info to help >> >> On Tue, Mar 10, 2015 at 6:54 PM, Arun Luthra <arun.lut...@gmail.com> >> wrote: >> >>> Does anyone know how to get the HighlyCompressedMapStatus to compile? >>> >>> I will try turning off kryo in 1.2.0 and hope things don't break. I >>> want to benefit from the MapOutputTracker fix in 1.2.0. >>> >>> On Tue, Mar 3, 2015 at 5:41 AM, Imran Rashid <iras...@cloudera.com> >>> wrote: >>> >>>> the scala syntax for arrays is Array[T], not T[], so you want to use >>>> something: >>>> >>>> kryo.register(classOf[Array[org.roaringbitmap.RoaringArray$Element]]) >>>> kryo.register(classOf[Array[Short]]) >>>> >>>> nonetheless, the spark should take care of this itself. I'll look into >>>> it later today. >>>> >>>> >>>> On Mon, Mar 2, 2015 at 2:55 PM, Arun Luthra <arun.lut...@gmail.com> >>>> wrote: >>>> >>>>> I think this is a Java vs scala syntax issue. Will check. >>>>> >>>>> On Thu, Feb 26, 2015 at 8:17 PM, Arun Luthra <arun.lut...@gmail.com> >>>>> wrote: >>>>> >>>>>> Problem is noted here: >>>>>> https://issues.apache.org/jira/browse/SPARK-5949 >>>>>> >>>>>> I tried this as a workaround: >>>>>> >>>>>> import org.apache.spark.scheduler._ >>>>>> import org.roaringbitmap._ >>>>>> >>>>>> ... >>>>>> >>>>>> >>>>>> kryo.register(classOf[org.roaringbitmap.RoaringBitmap]) >>>>>> kryo.register(classOf[org.roaringbitmap.RoaringArray]) >>>>>> kryo.register(classOf[org.roaringbitmap.ArrayContainer]) >>>>>> >>>>>> kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus]) >>>>>> kryo.register(classOf[org.roaringbitmap.RoaringArray$Element]) >>>>>> kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]]) >>>>>> kryo.register(classOf[short[]]) >>>>>> >>>>>> >>>>>> in build file: >>>>>> >>>>>> libraryDependencies += "org.roaringbitmap" % "RoaringBitmap" % >>>>>> "0.4.8" >>>>>> >>>>>> >>>>>> This fails to compile: >>>>>> >>>>>> ...:53: identifier expected but ']' found. >>>>>> >>>>>> [error] >>>>>> kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]]) >>>>>> >>>>>> also: >>>>>> >>>>>> :54: identifier expected but ']' found. >>>>>> >>>>>> [error] kryo.register(classOf[short[]]) >>>>>> also: >>>>>> >>>>>> :51: class HighlyCompressedMapStatus in package scheduler cannot be >>>>>> accessed in package org.apache.spark.scheduler >>>>>> [error] >>>>>> kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus]) >>>>>> >>>>>> >>>>>> Suggestions? >>>>>> >>>>>> Arun >>>>>> >>>>> >>>>> >>>> >>> >> >