Re: Kryo fails with avro having Arrays and unions, but succeeds with simple avro.

Thanks for the info frank. Twitter's-chill avro serializer looks great. But how does spark identifies it as serializer, as its not extending from KryoSerializer. (sorry scala is an alien lang for me). - Thanks Regards, Mohan -- View this message in context:

Re: Kryo fails with avro having Arrays and unions, but succeeds with simple avro.

Hi Mohan, It’s a bit convoluted to follow in their source, but they essentially typedef KSerializer as being a KryoSerializer, and then their serializers all extend KSerializer. Spark should identify them properly as Kryo Serializers, but I haven’t tried it myself. Regards, Frank Austin

Re: Kryo fails with avro having Arrays and unions, but succeeds with simple avro.

Hi frank, thanks for the info, thats great. but im not saying Avro serializer is failing. Kryo is failing but im using kryo serializer. and registering Avro generated classes with kryo. sparkConf.set(spark.serializer, org.apache.spark.serializer.KryoSerializer);