Re: Registering classes with KryoSerializer

[Imran Rashid]

hmm, I dunno why IntelliJ is unhappy, but you can always fall back to
getting a class from the String:

Class.forName(scala.reflect.ClassTag$$anon$1)

perhaps the class is package private or something, and the repl somehow
subverts it ...

On Tue, Apr 14, 2015 at 5:44 PM, Arun Lists lists.a...@gmail.com wrote:

 Hi Imran,

 Thanks for the response! However, I am still not there yet.

 In the Scala interpreter, I can do:

 scala classOf[scala.reflect.ClassTag$$anon$1]

 but when I try to do this in my program in IntelliJ, it indicates an error:

 Cannot resolve symbol ClassTag$$anon$1

 Hence I am not any closer to making this work. If you have any further
 suggestions, they would be most welcome.

 arun


 On Tue, Apr 14, 2015 at 2:33 PM, Imran Rashid iras...@cloudera.com
 wrote:

 Hi Arun,

 It can be hard to use kryo with required registration because of issues
 like this -- there isn't a good way to register all the classes that you
 need transitively.  In this case, it looks like one of your classes has a
 reference to a ClassTag, which in turn has a reference to some anonymous
 inner class.  I'd suggest

 (a) figuring out whether you really want to be serializing this thing --
 its possible you're serializing an RDD which keeps a ClassTag, but normally
 you wouldn't want to serialize your RDDs
 (b) you might want to bring this up w/ chill -- spark offloads most of
 the kryo setup for all the scala internals to chill, I'm surprised they
 don't handle this already.  Looks like they still handle ClassManifests
 which are from pre-scala 2.10:
 https://github.com/twitter/chill/blob/master/chill-scala/src/main/scala/com/twitter/chill/ScalaKryoInstantiator.scala#L189

 (c) you can always register these classes yourself, despite the crazy
 names, though you'll just need to knock these out one-by-one:

 scala classOf[scala.reflect.ClassTag$$anon$1]

 res0: Class[scala.reflect.ClassTag[T]{def unapply(x$1:
 scala.runtime.BoxedUnit): Option[_]; def arrayClass(x$1: Class[_]):
 Class[_]}] = class scala.reflect.ClassTag$$anon$1

 On Mon, Apr 13, 2015 at 6:09 PM, Arun Lists lists.a...@gmail.com wrote:

 Hi,

 I am trying to register classes with KryoSerializer. This has worked
 with other programs. Usually the error messages are helpful in indicating
 which classes need to be registered. But with my current program, I get the
 following cryptic error message:

 *Caused by: java.lang.IllegalArgumentException: Class is not registered:
 scala.reflect.ClassTag$$anon$1*

 *Note: To register this class use:
 kryo.register(scala.reflect.ClassTag$$anon$1.class);*

 How do I find out which class needs to be registered? I looked at my
 program and registered all classes used in RDDs. But clearly more classes
 remain to be registered if I can figure out which classes.

 Thanks for your help!

 arun

Re: Registering classes with KryoSerializer

[Arun Lists]

Wow, it all works now! Thanks, Imran!

In case someone else finds this useful, here are the additional classes
that I had to register (in addition to my application specific classes):

val tuple3ArrayClass = classOf[Array[Tuple3[Any, Any, Any]]]
val anonClass = Class.forName(scala.reflect.ClassTag$$anon$1)
val javaClassClass = classOf[java.lang.Class[Any]]

arun

On Tue, Apr 14, 2015 at 6:23 PM, Imran Rashid iras...@cloudera.com wrote:

 hmm, I dunno why IntelliJ is unhappy, but you can always fall back to
 getting a class from the String:

 Class.forName(scala.reflect.ClassTag$$anon$1)

 perhaps the class is package private or something, and the repl somehow
 subverts it ...

 On Tue, Apr 14, 2015 at 5:44 PM, Arun Lists lists.a...@gmail.com wrote:

 Hi Imran,

 Thanks for the response! However, I am still not there yet.

 In the Scala interpreter, I can do:

 scala classOf[scala.reflect.ClassTag$$anon$1]

 but when I try to do this in my program in IntelliJ, it indicates an
 error:

 Cannot resolve symbol ClassTag$$anon$1

 Hence I am not any closer to making this work. If you have any further
 suggestions, they would be most welcome.

 arun


 On Tue, Apr 14, 2015 at 2:33 PM, Imran Rashid iras...@cloudera.com
 wrote:

 Hi Arun,

 It can be hard to use kryo with required registration because of issues
 like this -- there isn't a good way to register all the classes that you
 need transitively.  In this case, it looks like one of your classes has a
 reference to a ClassTag, which in turn has a reference to some anonymous
 inner class.  I'd suggest

 (a) figuring out whether you really want to be serializing this thing --
 its possible you're serializing an RDD which keeps a ClassTag, but normally
 you wouldn't want to serialize your RDDs
 (b) you might want to bring this up w/ chill -- spark offloads most of
 the kryo setup for all the scala internals to chill, I'm surprised they
 don't handle this already.  Looks like they still handle ClassManifests
 which are from pre-scala 2.10:
 https://github.com/twitter/chill/blob/master/chill-scala/src/main/scala/com/twitter/chill/ScalaKryoInstantiator.scala#L189

 (c) you can always register these classes yourself, despite the crazy
 names, though you'll just need to knock these out one-by-one:

 scala classOf[scala.reflect.ClassTag$$anon$1]

 res0: Class[scala.reflect.ClassTag[T]{def unapply(x$1:
 scala.runtime.BoxedUnit): Option[_]; def arrayClass(x$1: Class[_]):
 Class[_]}] = class scala.reflect.ClassTag$$anon$1

 On Mon, Apr 13, 2015 at 6:09 PM, Arun Lists lists.a...@gmail.com
 wrote:

 Hi,

 I am trying to register classes with KryoSerializer. This has worked
 with other programs. Usually the error messages are helpful in indicating
 which classes need to be registered. But with my current program, I get the
 following cryptic error message:

 *Caused by: java.lang.IllegalArgumentException: Class is not
 registered: scala.reflect.ClassTag$$anon$1*

 *Note: To register this class use:
 kryo.register(scala.reflect.ClassTag$$anon$1.class);*

 How do I find out which class needs to be registered? I looked at my
 program and registered all classes used in RDDs. But clearly more classes
 remain to be registered if I can figure out which classes.

 Thanks for your help!

 arun

Re: Registering classes with KryoSerializer

[Imran Rashid]

Hi Arun,

It can be hard to use kryo with required registration because of issues
like this -- there isn't a good way to register all the classes that you
need transitively.  In this case, it looks like one of your classes has a
reference to a ClassTag, which in turn has a reference to some anonymous
inner class.  I'd suggest

(a) figuring out whether you really want to be serializing this thing --
its possible you're serializing an RDD which keeps a ClassTag, but normally
you wouldn't want to serialize your RDDs
(b) you might want to bring this up w/ chill -- spark offloads most of the
kryo setup for all the scala internals to chill, I'm surprised they don't
handle this already.  Looks like they still handle ClassManifests which are
from pre-scala 2.10:
https://github.com/twitter/chill/blob/master/chill-scala/src/main/scala/com/twitter/chill/ScalaKryoInstantiator.scala#L189

(c) you can always register these classes yourself, despite the crazy
names, though you'll just need to knock these out one-by-one:

scala classOf[scala.reflect.ClassTag$$anon$1]

res0: Class[scala.reflect.ClassTag[T]{def unapply(x$1:
scala.runtime.BoxedUnit): Option[_]; def arrayClass(x$1: Class[_]):
Class[_]}] = class scala.reflect.ClassTag$$anon$1

On Mon, Apr 13, 2015 at 6:09 PM, Arun Lists lists.a...@gmail.com wrote:

 Hi,

 I am trying to register classes with KryoSerializer. This has worked with
 other programs. Usually the error messages are helpful in indicating which
 classes need to be registered. But with my current program, I get the
 following cryptic error message:

 *Caused by: java.lang.IllegalArgumentException: Class is not registered:
 scala.reflect.ClassTag$$anon$1*

 *Note: To register this class use:
 kryo.register(scala.reflect.ClassTag$$anon$1.class);*

 How do I find out which class needs to be registered? I looked at my
 program and registered all classes used in RDDs. But clearly more classes
 remain to be registered if I can figure out which classes.

 Thanks for your help!

 arun

Re: Registering classes with KryoSerializer

[Arun Lists]

Hi Imran,

Thanks for the response! However, I am still not there yet.

In the Scala interpreter, I can do:

scala classOf[scala.reflect.ClassTag$$anon$1]

but when I try to do this in my program in IntelliJ, it indicates an error:

Cannot resolve symbol ClassTag$$anon$1

Hence I am not any closer to making this work. If you have any further
suggestions, they would be most welcome.

arun


On Tue, Apr 14, 2015 at 2:33 PM, Imran Rashid iras...@cloudera.com wrote:

 Hi Arun,

 It can be hard to use kryo with required registration because of issues
 like this -- there isn't a good way to register all the classes that you
 need transitively.  In this case, it looks like one of your classes has a
 reference to a ClassTag, which in turn has a reference to some anonymous
 inner class.  I'd suggest

 (a) figuring out whether you really want to be serializing this thing --
 its possible you're serializing an RDD which keeps a ClassTag, but normally
 you wouldn't want to serialize your RDDs
 (b) you might want to bring this up w/ chill -- spark offloads most of the
 kryo setup for all the scala internals to chill, I'm surprised they don't
 handle this already.  Looks like they still handle ClassManifests which are
 from pre-scala 2.10:
 https://github.com/twitter/chill/blob/master/chill-scala/src/main/scala/com/twitter/chill/ScalaKryoInstantiator.scala#L189

 (c) you can always register these classes yourself, despite the crazy
 names, though you'll just need to knock these out one-by-one:

 scala classOf[scala.reflect.ClassTag$$anon$1]

 res0: Class[scala.reflect.ClassTag[T]{def unapply(x$1:
 scala.runtime.BoxedUnit): Option[_]; def arrayClass(x$1: Class[_]):
 Class[_]}] = class scala.reflect.ClassTag$$anon$1

 On Mon, Apr 13, 2015 at 6:09 PM, Arun Lists lists.a...@gmail.com wrote:

 Hi,

 I am trying to register classes with KryoSerializer. This has worked with
 other programs. Usually the error messages are helpful in indicating which
 classes need to be registered. But with my current program, I get the
 following cryptic error message:

 *Caused by: java.lang.IllegalArgumentException: Class is not registered:
 scala.reflect.ClassTag$$anon$1*

 *Note: To register this class use:
 kryo.register(scala.reflect.ClassTag$$anon$1.class);*

 How do I find out which class needs to be registered? I looked at my
 program and registered all classes used in RDDs. But clearly more classes
 remain to be registered if I can figure out which classes.

 Thanks for your help!

 arun

Re: Registering classes with KryoSerializer

[Imran Rashid]

Those funny class names come from scala's specialization -- its compiling a
different version of OpenHashMap for each primitive you stick in the type
parameter.  Here's a super simple example:

*➜  **~ * more Foo.scala



class Foo[@specialized X]

*➜  **~ * scalac Foo.scala



*➜  **~ * ls Foo*.class



Foo$mcB$sp.class Foo$mcC$sp.class Foo$mcD$sp.class Foo$mcF$sp.class
Foo$mcI$sp.class Foo$mcJ$sp.class Foo$mcS$sp.class Foo$mcV$sp.class
Foo$mcZ$sp.class Foo.class

Sadly, I'm not sure of a foolproof way of getting all those specialized
versions registered except for registering with these strange names.
Here's an example of how its done by chill for Tuples (which is what spark
is relying on for its own registration of tuples):

https://github.com/twitter/chill/blob/6d03f6976f33f6e2e16b8e254fead1625720c281/chill-scala/src/main/scala/com/twitter/chill/TupleSerializers.scala#L861

On Mon, Mar 30, 2015 at 3:59 PM, Arun Lists lists.a...@gmail.com wrote:

 I am trying to register classes with KryoSerializer. I get the following
 error message:

 How do I find out what class is being referred to by: *OpenHashMap$mcI$sp
 ?*

 *com.esotericsoftware.kryo.KryoException:
 java.lang.IllegalArgumentException: Class is not registered:
 com.comp.common.base.OpenHashMap$mcI$sp*

 *Note: To register this class use: *
 *kryo.register(com.dtex.common.base.OpenHashMap$mcI$sp.class);*

 I have registered other classes with it by using:

 sparkConf.registerKryoClasses(Array(

   classOf[MyClass]

 ))


 Thanks,

 arun