Hi Ken, I think the main reason is that currently Kryo is the only generic serializer in Flink. I'm looking forward to your FLIP of Fury, and we can continue to discuss this issue there.
If there are no other questions, I will close the voting for this FLIP. Thank you again. Best, Fang Yong On Sat, Jan 6, 2024 at 2:27 AM Ken Krugler <kkrugler_li...@transpac.com> wrote: > Hi Fang Yong, > > Thanks for the response, and I understand the desire to limit the impact > of this FLIP. > > I guess I should spend the time to start a new FLIP on switching to Fury, > which could include cleaning up method names. > > In the context of “facilitate user understanding”, one aspect of this > cleanup is the current ExecutionConfig.enable/disable/hasGenericTypes() > methods. > > These are inconsistent with the current xxxKryo() methods, and cause > confusion whenever I’m teaching a Flink course :) > > Regards, > > — Ken > > > > > On Jan 4, 2024, at 6:40 PM, Yong Fang <zjur...@gmail.com> wrote: > > Hi Ken, > > Sorry for the late reply. After discussing with @Xintong, we think it is > better to keep the method names in the FLIP mainly for the following > reasons: > > 1. This FLIP is mainly to support the configurable serializer while > keeping consistent with Flink at the semantic layer. Keeping the existing > naming rules can facilitate user understanding. > > 2. In the future, if Flink can choose Fury as the generic serializer, we > can update the corresponding methods in that FLIP after the discussion of > Fury is completed. This will be a minor modification, and we can avoid > over-design in the current FLIP. > > Thanks for your feedback! > > Best, > Fang Yong > > On Fri, Dec 29, 2023 at 12:38 PM Ken Krugler <kkrugler_li...@transpac.com> > wrote: > >> Hi Xintong, >> >> I agree that decoupling from Kryo is a bigger topic, well beyond the >> scope of this FLIP. >> >> The reason I’d brought up Fury is that this increases my confidence that >> Flink will want to decouple from Kryo sooner rather than later. >> >> So I feel it would be worth investing in a (minor) name change now, to >> improve that migration path in the future. Thus my suggestion for avoiding >> the explicit use of Kryo in method names. >> >> Regards, >> >> — Ken >> >> >> >> >> > On Dec 17, 2023, at 7:16 PM, Xintong Song <tonysong...@gmail.com> >> wrote: >> > >> > Hi Ken, >> > >> > I think the main purpose of this FLIP is to change how users interact >> with >> > the knobs for customizing the serialization behaviors, from requiring >> code >> > changes to working with pure configurations. Redesigning the knobs >> (i.e., >> > names, semantics, etc.), on the other hand, is not the purpose of this >> > FLIP. Preserving the existing names and semantics should also help >> minimize >> > the migration cost for existing users. Therefore, I'm in favor of not >> > changing them. >> > >> > Concerning decoupling from Kryo, and introducing other serialization >> > frameworks like Fury, I think that's a bigger topic that is worth >> further >> > discussion. At the moment, I'm not aware of any community consensus on >> > doing so. And even if in the future we decide to do so, the changes >> needed >> > should be the same w/ or w/o this FLIP. So I'd suggest not to block this >> > FLIP on these issues. >> > >> > WDYT? >> > >> > Best, >> > >> > Xintong >> > >> > >> > >> > On Fri, Dec 15, 2023 at 1:40 AM Ken Krugler < >> kkrugler_li...@transpac.com> >> > wrote: >> > >> >> Hi Yong, >> >> >> >> Looks good, thanks for creating this. >> >> >> >> One comment - related to my recent email about Fury, I would love to >> see >> >> the v2 serialization decoupled from Kryo. >> >> >> >> As part of that, instead of using xxxKryo in methods, call them >> xxxGeneric. >> >> >> >> A more extreme change would be to totally rely on Fury (so no more POJO >> >> serializer). Fury is faster than the POJO serializer in my tests, but >> this >> >> would be a much bigger change. >> >> >> >> Though it could dramatically simplify the Flink serialization support. >> >> >> >> — Ken >> >> >> >> PS - a separate issue is how to migrate state from Kryo to something >> like >> >> Fury, which supports schema evolution. I think this might be possible, >> by >> >> having a smarter deserializer that identifies state as being created by >> >> Kryo, and using (shaded) Kryo to deserialize, while still writing as >> Fury. >> >> >> >>> On Dec 6, 2023, at 6:35 PM, Yong Fang <zjur...@gmail.com> wrote: >> >>> >> >>> Hi devs, >> >>> >> >>> I'd like to start a discussion about FLIP-398: Improve Serialization >> >>> Configuration And Usage In Flink [1]. >> >>> >> >>> Currently, users can register custom data types and serializers in >> Flink >> >>> jobs through various methods, including registration in code, >> >>> configuration, and annotations. These lead to difficulties in >> upgrading >> >>> Flink jobs and priority issues. >> >>> >> >>> In flink-2.0 we would like to manage job data types and serializers >> >> through >> >>> configurations. This FLIP will introduce a unified option for data >> type >> >> and >> >>> serializer and users can configure all custom data types and >> >>> pojo/kryo/custom serializers. In addition, this FLIP will add more >> >> built-in >> >>> serializers for complex data types such as List and Map, and optimize >> the >> >>> management of Avro Serializers. >> >>> >> >>> Looking forward to hearing from you, thanks! >> >>> >> >>> [1] >> >>> >> >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-398%3A+Improve+Serialization+Configuration+And+Usage+In+Flink >> >>> >> >>> Best, >> >>> Fang Yong >> >> >> >> -------------------------- >> >> Ken Krugler >> >> http://www.scaleunlimited.com >> >> Custom big data solutions >> >> Flink & Pinot >> >> >> >> >> >> >> >> >> >> >> >> -------------------------- >> Ken Krugler >> http://www.scaleunlimited.com >> Custom big data solutions >> Flink & Pinot >> >> >> >> > -------------------------- > Ken Krugler > http://www.scaleunlimited.com > Custom big data solutions > Flink & Pinot > > > >
