Hi, Yes, the AvroSerializer currently partially still uses Kryo for object copying. Also, right now, I think the AvroSerializer is only used when the type is recognized as a POJO, and that `isForceAvroEnabled` is set on the job configuration. I’m not sure if that is always possible. As mentioned in [1], we would probably need to improve the user experience for Avro usage.
For now, if you want to directly use Avro only for serializing your state, AFAIK the straightforward approach would be, as you mentioned, to extend a custom TypeSerializer that uses the Avro constructs. Flink’s AvroSerializer actually already sorts of does this, so you can refer to that implementation as a base line. Cheers, Gordon [1] http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Serialization-performance-td12019.html On 5 October 2017 at 4:39:10 PM, mrooding (ad...@webresource.nl) wrote: Gordon Thanks for the detailed response. I have verified your assumption and that is, unfortunately, the case. I also looked into creating a custom Kryo serializer but I got stuck on serializing arrays of complex types. It seems like this isn't trivial at all with Kryo. As an alternative, I've been looking into using Avro only for the Flink buffers. Basically, as a first step, we'd still be sending JSON messages through Kafka but we would use a custom TypeSerializer that converts the case classes to bytes using Avro and vice versa. Unfortunately, documentation is really scarce. In a different topic, http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Serialization-performance-td12019.html, it says that Avro is a bit of an odd citizen and that the AvroSerializer provided by Flink uses Kryo. This confirms what I've found by going through the source code of Flink myself. I hope that you can provide me with some pointers. Is extending TypeSerializer[T] the best way forward if we only want to use Avro for state buffers and thus utilize Avro's schema migration facilities? Any pointers would be greatly appreciated! Kind regards Marc -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
