I'm not sure what you mean. Are you asking how you can recompile all of spark and deploy it, instead of using one of the pre-built versions?
https://spark.apache.org/docs/latest/building-spark.html Or are you seeing compile problems specifically w/ HighlyCompressedMapStatus? The code compiles fine, so I'm not sure what problem you are running into -- we'd need a lot more info to help On Tue, Mar 10, 2015 at 6:54 PM, Arun Luthra <arun.lut...@gmail.com> wrote: > Does anyone know how to get the HighlyCompressedMapStatus to compile? > > I will try turning off kryo in 1.2.0 and hope things don't break. I want > to benefit from the MapOutputTracker fix in 1.2.0. > > On Tue, Mar 3, 2015 at 5:41 AM, Imran Rashid <iras...@cloudera.com> wrote: > >> the scala syntax for arrays is Array[T], not T[], so you want to use >> something: >> >> kryo.register(classOf[Array[org.roaringbitmap.RoaringArray$Element]]) >> kryo.register(classOf[Array[Short]]) >> >> nonetheless, the spark should take care of this itself. I'll look into >> it later today. >> >> >> On Mon, Mar 2, 2015 at 2:55 PM, Arun Luthra <arun.lut...@gmail.com> >> wrote: >> >>> I think this is a Java vs scala syntax issue. Will check. >>> >>> On Thu, Feb 26, 2015 at 8:17 PM, Arun Luthra <arun.lut...@gmail.com> >>> wrote: >>> >>>> Problem is noted here: https://issues.apache.org/jira/browse/SPARK-5949 >>>> >>>> I tried this as a workaround: >>>> >>>> import org.apache.spark.scheduler._ >>>> import org.roaringbitmap._ >>>> >>>> ... >>>> >>>> >>>> kryo.register(classOf[org.roaringbitmap.RoaringBitmap]) >>>> kryo.register(classOf[org.roaringbitmap.RoaringArray]) >>>> kryo.register(classOf[org.roaringbitmap.ArrayContainer]) >>>> >>>> kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus]) >>>> kryo.register(classOf[org.roaringbitmap.RoaringArray$Element]) >>>> kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]]) >>>> kryo.register(classOf[short[]]) >>>> >>>> >>>> in build file: >>>> >>>> libraryDependencies += "org.roaringbitmap" % "RoaringBitmap" % "0.4.8" >>>> >>>> >>>> This fails to compile: >>>> >>>> ...:53: identifier expected but ']' found. >>>> >>>> [error] >>>> kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]]) >>>> >>>> also: >>>> >>>> :54: identifier expected but ']' found. >>>> >>>> [error] kryo.register(classOf[short[]]) >>>> also: >>>> >>>> :51: class HighlyCompressedMapStatus in package scheduler cannot be >>>> accessed in package org.apache.spark.scheduler >>>> [error] >>>> kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus]) >>>> >>>> >>>> Suggestions? >>>> >>>> Arun >>>> >>> >>> >> >
