Hi Fang Yong, Thanks for the response, and I understand the desire to limit the impact of this FLIP.
I guess I should spend the time to start a new FLIP on switching to Fury, which could include cleaning up method names. In the context of “facilitate user understanding”, one aspect of this cleanup is the current ExecutionConfig.enable/disable/hasGenericTypes() methods. These are inconsistent with the current xxxKryo() methods, and cause confusion whenever I’m teaching a Flink course :) Regards, — Ken > On Jan 4, 2024, at 6:40 PM, Yong Fang <zjur...@gmail.com> wrote: > > Hi Ken, > > Sorry for the late reply. After discussing with @Xintong, we think it is > better to keep the method names in the FLIP mainly for the following reasons: > > 1. This FLIP is mainly to support the configurable serializer while keeping > consistent with Flink at the semantic layer. Keeping the existing naming > rules can facilitate user understanding. > > 2. In the future, if Flink can choose Fury as the generic serializer, we can > update the corresponding methods in that FLIP after the discussion of Fury is > completed. This will be a minor modification, and we can avoid over-design in > the current FLIP. > > Thanks for your feedback! > > Best, > Fang Yong > > On Fri, Dec 29, 2023 at 12:38 PM Ken Krugler <kkrugler_li...@transpac.com > <mailto:kkrugler_li...@transpac.com>> wrote: >> Hi Xintong, >> >> I agree that decoupling from Kryo is a bigger topic, well beyond the scope >> of this FLIP. >> >> The reason I’d brought up Fury is that this increases my confidence that >> Flink will want to decouple from Kryo sooner rather than later. >> >> So I feel it would be worth investing in a (minor) name change now, to >> improve that migration path in the future. Thus my suggestion for avoiding >> the explicit use of Kryo in method names. >> >> Regards, >> >> — Ken >> >> >> >> >> > On Dec 17, 2023, at 7:16 PM, Xintong Song <tonysong...@gmail.com >> > <mailto:tonysong...@gmail.com>> wrote: >> > >> > Hi Ken, >> > >> > I think the main purpose of this FLIP is to change how users interact with >> > the knobs for customizing the serialization behaviors, from requiring code >> > changes to working with pure configurations. Redesigning the knobs (i.e., >> > names, semantics, etc.), on the other hand, is not the purpose of this >> > FLIP. Preserving the existing names and semantics should also help minimize >> > the migration cost for existing users. Therefore, I'm in favor of not >> > changing them. >> > >> > Concerning decoupling from Kryo, and introducing other serialization >> > frameworks like Fury, I think that's a bigger topic that is worth further >> > discussion. At the moment, I'm not aware of any community consensus on >> > doing so. And even if in the future we decide to do so, the changes needed >> > should be the same w/ or w/o this FLIP. So I'd suggest not to block this >> > FLIP on these issues. >> > >> > WDYT? >> > >> > Best, >> > >> > Xintong >> > >> > >> > >> > On Fri, Dec 15, 2023 at 1:40 AM Ken Krugler <kkrugler_li...@transpac.com >> > <mailto:kkrugler_li...@transpac.com>> >> > wrote: >> > >> >> Hi Yong, >> >> >> >> Looks good, thanks for creating this. >> >> >> >> One comment - related to my recent email about Fury, I would love to see >> >> the v2 serialization decoupled from Kryo. >> >> >> >> As part of that, instead of using xxxKryo in methods, call them >> >> xxxGeneric. >> >> >> >> A more extreme change would be to totally rely on Fury (so no more POJO >> >> serializer). Fury is faster than the POJO serializer in my tests, but this >> >> would be a much bigger change. >> >> >> >> Though it could dramatically simplify the Flink serialization support. >> >> >> >> — Ken >> >> >> >> PS - a separate issue is how to migrate state from Kryo to something like >> >> Fury, which supports schema evolution. I think this might be possible, by >> >> having a smarter deserializer that identifies state as being created by >> >> Kryo, and using (shaded) Kryo to deserialize, while still writing as Fury. >> >> >> >>> On Dec 6, 2023, at 6:35 PM, Yong Fang <zjur...@gmail.com >> >>> <mailto:zjur...@gmail.com>> wrote: >> >>> >> >>> Hi devs, >> >>> >> >>> I'd like to start a discussion about FLIP-398: Improve Serialization >> >>> Configuration And Usage In Flink [1]. >> >>> >> >>> Currently, users can register custom data types and serializers in Flink >> >>> jobs through various methods, including registration in code, >> >>> configuration, and annotations. These lead to difficulties in upgrading >> >>> Flink jobs and priority issues. >> >>> >> >>> In flink-2.0 we would like to manage job data types and serializers >> >> through >> >>> configurations. This FLIP will introduce a unified option for data type >> >> and >> >>> serializer and users can configure all custom data types and >> >>> pojo/kryo/custom serializers. In addition, this FLIP will add more >> >> built-in >> >>> serializers for complex data types such as List and Map, and optimize the >> >>> management of Avro Serializers. >> >>> >> >>> Looking forward to hearing from you, thanks! >> >>> >> >>> [1] >> >>> >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-398%3A+Improve+Serialization+Configuration+And+Usage+In+Flink >> >>> >> >>> Best, >> >>> Fang Yong >> >> >> >> -------------------------- >> >> Ken Krugler >> >> http://www.scaleunlimited.com <http://www.scaleunlimited.com/> >> >> Custom big data solutions >> >> Flink & Pinot >> >> >> >> >> >> >> >> >> >> >> >> -------------------------- >> Ken Krugler >> http://www.scaleunlimited.com <http://www.scaleunlimited.com/> >> Custom big data solutions >> Flink & Pinot >> >> >> -------------------------- Ken Krugler http://www.scaleunlimited.com Custom big data solutions Flink & Pinot
