Matei, thanks for the reply. I've tried setting kryo.setRegistrationOptional by KryoRegistrator but it seems that some spark's internal class, such as o.a.spark.scheduler.MapStatus, need to be registered.
2013/10/17 Matei Zaharia <[email protected]> > This is a good idea, though you may also be able to do it in your > KryoRegistrator by just calling kryo.setRegistrationOptional there. (You > get full access to the Kryo object.) > > Omitting to register a class is not an error by the way, it just leads to > bigger data. > > Matei > > On Oct 16, 2013, at 3:28 AM, Han JU <[email protected]> wrote: > > > Hi, > > > > I think maybe it's better to have an option to force class registration > > when using Kryo, mainly to prevent human errors. > > Kryo has a setRegistrationRequired method and by default the check is > false > > (in kryo and in chill). Maybe we can have a system property for setting > > this? Like: > > > > System.setProperty("spark.kryo.registrationRequired", true) > > > > What you say about this? > > -- > > *JU Han* > > > > Data Engineer @ Botify.com > > > > +33 0619608888 > > -- *JU Han* Data Engineer @ Botify.com +33 0619608888
