Hi Yong, Looks good, thanks for creating this.
One comment - related to my recent email about Fury, I would love to see the v2 serialization decoupled from Kryo. As part of that, instead of using xxxKryo in methods, call them xxxGeneric. A more extreme change would be to totally rely on Fury (so no more POJO serializer). Fury is faster than the POJO serializer in my tests, but this would be a much bigger change. Though it could dramatically simplify the Flink serialization support. — Ken PS - a separate issue is how to migrate state from Kryo to something like Fury, which supports schema evolution. I think this might be possible, by having a smarter deserializer that identifies state as being created by Kryo, and using (shaded) Kryo to deserialize, while still writing as Fury. > On Dec 6, 2023, at 6:35 PM, Yong Fang <zjur...@gmail.com> wrote: > > Hi devs, > > I'd like to start a discussion about FLIP-398: Improve Serialization > Configuration And Usage In Flink [1]. > > Currently, users can register custom data types and serializers in Flink > jobs through various methods, including registration in code, > configuration, and annotations. These lead to difficulties in upgrading > Flink jobs and priority issues. > > In flink-2.0 we would like to manage job data types and serializers through > configurations. This FLIP will introduce a unified option for data type and > serializer and users can configure all custom data types and > pojo/kryo/custom serializers. In addition, this FLIP will add more built-in > serializers for complex data types such as List and Map, and optimize the > management of Avro Serializers. > > Looking forward to hearing from you, thanks! > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-398%3A+Improve+Serialization+Configuration+And+Usage+In+Flink > > Best, > Fang Yong -------------------------- Ken Krugler http://www.scaleunlimited.com Custom big data solutions Flink & Pinot
