Re: Kryo NPE with Array
I finally solved this issue. The problem was that: 1. I defined a case class with a Buffer[MyType] field. 2. I instantiated the class with the field set to the value given by an implicit conversion from a Java list, which is supposedly a Buffer. 3. However, the underlying type of that field was instead scala.collection.convert.Wrappers.JListWrapper, as noted in the exception above. This type was not registered with Kryo and so that's why I got the exception. Registering the type did not solve the problem. However, an additional call to .toBuffer did solve the problem, since the Buffer class is registered through the Chill AllScalaRegistrar which is called by the Spark Kryo serializer. I thought I'd document this in case somebody else is running into a similar issue. Simone Franzini, PhD http://www.linkedin.com/in/simonefranzini On Wed, Nov 26, 2014 at 7:40 PM, Simone Franzini wrote: > I guess I already have the answer of what I have to do here, which is to > configure the kryo object with the strategy as above. > Now the question becomes: how can I pass this custom kryo configuration to > the spark kryo serializer / kryo registrator? > I've had a look at the code but I am still fairly new to Scala and I can't > see how I would do this. In the worst case, could I override the newKryo > method and put my configuration there? It appears to me that method is the > one where the kryo instance is created. > > Simone Franzini, PhD > > http://www.linkedin.com/in/simonefranzini > > On Tue, Nov 25, 2014 at 2:38 PM, Simone Franzini > wrote: > >> I am running into the following NullPointerException: >> >> com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException >> Serialization trace: >> underlying (scala.collection.convert.Wrappers$JListWrapper) >> myArrayField (MyCaseClass) >> at >> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) >> >> I have been running into similar issues when using avro classes, that I >> was able to resolve by registering them with a Kryo serializer that uses >> chill-avro. However, in this case the field is in a case class and it seems >> that registering the class does not help. >> >> I found this stack overflow that seems to be relevant: >> >> http://stackoverflow.com/questions/23962796/kryo-readobject-cause-nullpointerexception-with-arraylist >> I have this line of code translated to Scala, that supposedly solves the >> issue: >> >> val kryo = new Kryo() >> kryo.getInstantiatorStrategy().asInstanceOf[Kryo.DefaultInstantiatorStrategy].setFallbackInstantiatorStrategy(new >> StdInstantiatorStrategy()) >> >> However, I am not sure where this line should be placed to take effect. >> >> I already have the following, should it go somewhere in here? >> class MyRegistrator extends KryoRegistrator { >> override def registerClasses(kryo: Kryo) { >> kryo.register(...) >> } >> } >> >> >> Simone Franzini, PhD >> >> http://www.linkedin.com/in/simonefranzini >> > >
Re: Kryo NPE with Array
I guess I already have the answer of what I have to do here, which is to configure the kryo object with the strategy as above. Now the question becomes: how can I pass this custom kryo configuration to the spark kryo serializer / kryo registrator? I've had a look at the code but I am still fairly new to Scala and I can't see how I would do this. In the worst case, could I override the newKryo method and put my configuration there? It appears to me that method is the one where the kryo instance is created. Simone Franzini, PhD http://www.linkedin.com/in/simonefranzini On Tue, Nov 25, 2014 at 2:38 PM, Simone Franzini wrote: > I am running into the following NullPointerException: > > com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException > Serialization trace: > underlying (scala.collection.convert.Wrappers$JListWrapper) > myArrayField (MyCaseClass) > at > com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) > > I have been running into similar issues when using avro classes, that I > was able to resolve by registering them with a Kryo serializer that uses > chill-avro. However, in this case the field is in a case class and it seems > that registering the class does not help. > > I found this stack overflow that seems to be relevant: > > http://stackoverflow.com/questions/23962796/kryo-readobject-cause-nullpointerexception-with-arraylist > I have this line of code translated to Scala, that supposedly solves the > issue: > > val kryo = new Kryo() > kryo.getInstantiatorStrategy().asInstanceOf[Kryo.DefaultInstantiatorStrategy].setFallbackInstantiatorStrategy(new > StdInstantiatorStrategy()) > > However, I am not sure where this line should be placed to take effect. > > I already have the following, should it go somewhere in here? > class MyRegistrator extends KryoRegistrator { > override def registerClasses(kryo: Kryo) { > kryo.register(...) > } > } > > > Simone Franzini, PhD > > http://www.linkedin.com/in/simonefranzini >
Kryo NPE with Array
I am running into the following NullPointerException: com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException Serialization trace: underlying (scala.collection.convert.Wrappers$JListWrapper) myArrayField (MyCaseClass) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) I have been running into similar issues when using avro classes, that I was able to resolve by registering them with a Kryo serializer that uses chill-avro. However, in this case the field is in a case class and it seems that registering the class does not help. I found this stack overflow that seems to be relevant: http://stackoverflow.com/questions/23962796/kryo-readobject-cause-nullpointerexception-with-arraylist I have this line of code translated to Scala, that supposedly solves the issue: val kryo = new Kryo() kryo.getInstantiatorStrategy().asInstanceOf[Kryo.DefaultInstantiatorStrategy].setFallbackInstantiatorStrategy(new StdInstantiatorStrategy()) However, I am not sure where this line should be placed to take effect. I already have the following, should it go somewhere in here? class MyRegistrator extends KryoRegistrator { override def registerClasses(kryo: Kryo) { kryo.register(...) } } Simone Franzini, PhD http://www.linkedin.com/in/simonefranzini