Hi Xintong, I agree that decoupling from Kryo is a bigger topic, well beyond the scope of this FLIP.
The reason I’d brought up Fury is that this increases my confidence that Flink will want to decouple from Kryo sooner rather than later. So I feel it would be worth investing in a (minor) name change now, to improve that migration path in the future. Thus my suggestion for avoiding the explicit use of Kryo in method names. Regards, — Ken > On Dec 17, 2023, at 7:16 PM, Xintong Song <tonysong...@gmail.com> wrote: > > Hi Ken, > > I think the main purpose of this FLIP is to change how users interact with > the knobs for customizing the serialization behaviors, from requiring code > changes to working with pure configurations. Redesigning the knobs (i.e., > names, semantics, etc.), on the other hand, is not the purpose of this > FLIP. Preserving the existing names and semantics should also help minimize > the migration cost for existing users. Therefore, I'm in favor of not > changing them. > > Concerning decoupling from Kryo, and introducing other serialization > frameworks like Fury, I think that's a bigger topic that is worth further > discussion. At the moment, I'm not aware of any community consensus on > doing so. And even if in the future we decide to do so, the changes needed > should be the same w/ or w/o this FLIP. So I'd suggest not to block this > FLIP on these issues. > > WDYT? > > Best, > > Xintong > > > > On Fri, Dec 15, 2023 at 1:40 AM Ken Krugler <kkrugler_li...@transpac.com> > wrote: > >> Hi Yong, >> >> Looks good, thanks for creating this. >> >> One comment - related to my recent email about Fury, I would love to see >> the v2 serialization decoupled from Kryo. >> >> As part of that, instead of using xxxKryo in methods, call them xxxGeneric. >> >> A more extreme change would be to totally rely on Fury (so no more POJO >> serializer). Fury is faster than the POJO serializer in my tests, but this >> would be a much bigger change. >> >> Though it could dramatically simplify the Flink serialization support. >> >> — Ken >> >> PS - a separate issue is how to migrate state from Kryo to something like >> Fury, which supports schema evolution. I think this might be possible, by >> having a smarter deserializer that identifies state as being created by >> Kryo, and using (shaded) Kryo to deserialize, while still writing as Fury. >> >>> On Dec 6, 2023, at 6:35 PM, Yong Fang <zjur...@gmail.com> wrote: >>> >>> Hi devs, >>> >>> I'd like to start a discussion about FLIP-398: Improve Serialization >>> Configuration And Usage In Flink [1]. >>> >>> Currently, users can register custom data types and serializers in Flink >>> jobs through various methods, including registration in code, >>> configuration, and annotations. These lead to difficulties in upgrading >>> Flink jobs and priority issues. >>> >>> In flink-2.0 we would like to manage job data types and serializers >> through >>> configurations. This FLIP will introduce a unified option for data type >> and >>> serializer and users can configure all custom data types and >>> pojo/kryo/custom serializers. In addition, this FLIP will add more >> built-in >>> serializers for complex data types such as List and Map, and optimize the >>> management of Avro Serializers. >>> >>> Looking forward to hearing from you, thanks! >>> >>> [1] >>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-398%3A+Improve+Serialization+Configuration+And+Usage+In+Flink >>> >>> Best, >>> Fang Yong >> >> -------------------------- >> Ken Krugler >> http://www.scaleunlimited.com >> Custom big data solutions >> Flink & Pinot >> >> >> >> -------------------------- Ken Krugler http://www.scaleunlimited.com Custom big data solutions Flink & Pinot
