Hi Andrea,
What version of Spark are you using? There were some improvements in how
Spark uses Kryo in 0.9.1 and to-be 1.0 that I would expect to improve this.
Also, can you share your registrator's code?
Another possibility is that Kryo can have some difficulty serializing very
large objects. Do you have a sense of how large the serialized items in
your RDD are?
Andrew
On Sat, May 10, 2014 at 6:32 AM, Andrea Esposito <and1...@gmail.com> wrote:
> UP, doesn't anyone know something about it? ^^
>
>
> 2014-05-06 12:05 GMT+02:00 Andrea Esposito <and1...@gmail.com>:
>
> Hi there,
>>
>> sorry if i'm posting a lot lately.
>>
>> i'm trying to add the KryoSerializer but i receive this exception:
>> 2014 - 05 - 06 11: 45: 23 WARN TaskSetManager: 62 - Loss was due to
>> java.io.EOFException
>> java.io.EOFException
>> at
>> org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:
>> 105)
>> at org.apache.spark.broadcast.HttpBroadcast$.read(HttpBroadcast.scala:
>> 165)
>> at
>> org.apache.spark.broadcast.HttpBroadcast.readObject(HttpBroadcast.scala: 56)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>> 57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
>> 43)
>> at java.lang.reflect.Method.invoke(Method.java: 606)
>>
>> I set the serializer as:
>> System.setProperty("spark.serializer",
>> "org.apache.spark.serializer.KryoSerializer")
>> System.setProperty("spark.kryo.registrator", "test.TestKryoRegistrator")
>>
>> With or without register my custom registrator it throws the exception.
>>
>> Seems something related to broadcast.. but isn't Kryo already ok out of
>> the box just setting it as default serializer?
>>
>
>
