Hi everyone,
Thanks for all the comments! Based on the discussion so far, I will proceed
to initiate the vote tomorrow if there are no further discussions or
objections.
Best regards,
Junrui
Junrui Lee <jrlee....@gmail.com> 于2023年10月9日周一 11:52写道：
> Hi, Chesnay and David, sorry for the late reply.
>
> Thanks for your feedback about this FLIP.
>
> @Chesnay
>
> >> Personally I'd just name it "config.yaml"
> Thanks for your suggestions, I agree that "config.yaml" is a simpler and
> more intuitive name. I have updated the FLIP accordingly.
>
> >> In terms of scope, is the migration of existing e2e tests and the
> docker image to the new parser part of the FLIP?
> The migration of existing end-to-end tests and the docker image to the new
> parser is indeed included in the FLIP. Taking into account the difficulty
> of using shell scripts to modify configuration files that adhere to
> standard YAML syntax, I propose utilizing the
> org.apache.flink.runtime.util.bash.BashJavaUtils class as a unified utility
> for modifying the configuration file.
>
> In addition, considering that users may not only configure properties in
> the configuration file, but also use dynamic parameters like '-D' or
> configure the "with" clauses in FLINK SQL. When the default parser is
> changed to the standard YAML parser, the configOption whose value is List
> or Map types also needs to be modified accordingly. This migration cost is
> relatively high because users have to modify their code or command lines.
> To reduce migration costs while considering compatibility, I propose to
> add compatibility support in FLINK-1.X versions to parse old format List
> and Map values using the standard YAML parser. In case of an error while
> parsing List or Map configurations with the standard YAML parser, a WARNING
> log will be printed, and an attempt will be made to parse using the old
> parser. And this support will be removed in the FLINK-2.0 version.
>
> Do you think the aforementioned compatibility handling is necessary in
> order to reduce migration costs? If you agree, I will also update this
> support in the FLIP under the migration section.
>
> @David
>
> >> could we write a migration tool, that would convert the old config
> files to the new format,
> I agree that providing a more user-friendly guidance for the migration is
> important. In the FLIP, we can include a user migration manual to help
> users with the migration process. This migration manual will be presented
> on the official FLINK website, making it easily accessible for users. And
> the migrated configuration files will adhere to standard YAML syntax, and
> users can validate them using various third-party YAML tools.
>
> Regarding the automated migration tool, I think it can be considered as a
> follow-up task rather than a requirement for the initial version. We can
> focus on implementing the migration manually first and then explore the
> possibility of automating it in future updates.
>
> Best,
> Junrui
>
> David Radley <david_rad...@uk.ibm.com> 于2023年10月3日周二 18:56写道：
>
>> Hi,
>> I agree this is a standardising, simplifying change for read,
>> simplifying programmatically authoring the config file as well. As you
>> know the mapping for the old config form to the new form, could we write a
>> migration tool, that would convert the old config files to the new format,
>> with errors and warnings if the content is such that the user would need to
>> manually fix up the file.
>> If there are minimal user fixups, we should consider automatically
>> migrating the config file to the new format,
>> Kind regards, David.
>>
>>
>>
>>
>> From: Chesnay Schepler <ches...@apache.org>
>> Date: Tuesday, 3 October 2023 at 11:17
>> To: dev@flink.apache.org <dev@flink.apache.org>
>> Subject: [EXTERNAL] Re: [Discuss] FLIP-366: Support standard YAML for
>> FLINK configuration
>> It is a unfortunate that we'll need a separate config file but the FLIP
>> does a good job justifying it.
>>
>> Personally I'd just name it "config.yaml"; I never quite understood why
>> there was a flink prefix to begin with, and the current proposal
>> ("flink-configuration.yaml") seems unnecessarily long.
>>
>> For the deprecation process we could consider logging a warning if the
>> old parser is used.
>>
>> In terms of scope, is the migration of existing e2e tests and the docker
>> image to the new parser part of the FLIP?
>>
>> On 22/09/2023 09:32, Jane Chan wrote:
>> > Hi, Junrui,
>> >
>> > Sorry for the late reply. The update looks good to me and thanks for
>> your
>> > effort!
>> >
>> > Best,
>> > Jane
>> >
>> > On Fri, Sep 22, 2023 at 2:25 PM Yuxin Tan <tanyuxinw...@gmail.com>
>> wrote:
>> >
>> >> Hi, Junrui
>> >>
>> >> +1 for the proposal.
>> >> Thanks for your effort.
>> >>
>> >> Best,
>> >> Yuxin
>> >>
>> >>
>> >> Samrat Deb <decordea...@gmail.com> 于2023年9月22日周五 13:23写道：
>> >>
>> >>> Hello Junrui,
>> >>>
>> >>> +1 for the proposal.
>> >>>
>> >>>
>> >>> Bests,
>> >>> Samrat
>> >>>
>> >>> On Fri, Sep 22, 2023 at 10:18 AM Shammon FY <zjur...@gmail.com>
>> wrote:
>> >>>
>> >>>> +1 for the proposal, thanks for driving.
>> >>>>
>> >>>> Bet,
>> >>>> Shammon FY
>> >>>>
>> >>>> On Fri, Sep 22, 2023 at 12:41 PM Yangze Guo <karma...@gmail.com>
>> >> wrote:
>> >>>>> Thanks for driving this, +1 for the proposal.
>> >>>>>
>> >>>>> Best,
>> >>>>> Yangze Guo
>> >>>>>
>> >>>>>
>> >>>>> On Fri, Sep 22, 2023 at 11:59 AM Lijie Wang <
>> >> wangdachui9...@gmail.com>
>> >>>>> wrote:
>> >>>>>> Hi Junrui,
>> >>>>>>
>> >>>>>> +1 for this proposal, thanks for driving.
>> >>>>>>
>> >>>>>> Best,
>> >>>>>> Lijie
>> >>>>>>
>> >>>>>> ConradJam <jam.gz...@gmail.com> 于2023年9月22日周五 10:07写道：
>> >>>>>>
>> >>>>>>> +1 Support for standard YAML format facilitates specification
>> >>>>>>>
>> >>>>>>> Jing Ge <j...@ververica.com.invalid> 于2023年9月22日周五 02:23写道：
>> >>>>>>>
>> >>>>>>>> Hi Junrui,
>> >>>>>>>>
>> >>>>>>>> +1 for following the standard. Thanks for your effort!
>> >>>>>>>>
>> >>>>>>>> Best regards,
>> >>>>>>>> Jing
>> >>>>>>>>
>> >>>>>>>> On Thu, Sep 21, 2023 at 5:09 AM Junrui Lee <
>> >> jrlee....@gmail.com>
>> >>>>> wrote:
>> >>>>>>>>> Hi Jane,
>> >>>>>>>>>
>> >>>>>>>>> Thank you for your valuable feedback and suggestions.
>> >>>>>>>>> I agree with your point about differentiating between
>> >>>>>>> "flink-config.yaml"
>> >>>>>>>>> and "flink-conf.yaml" to determine the standard syntax at a
>> >>>> glance.
>> >>>>>>>>> While I understand your suggestion of using
>> >>>>> "flink-conf-default.yaml"
>> >>>>>>> to
>> >>>>>>>>> represent the default YAML file for Flink 1.x, I have been
>> >>>>> considering
>> >>>>>>>>> the option of using "flink-configuration.yaml" as the file
>> >> name
>> >>>>> for the
>> >>>>>>>>> new configuration file.
>> >>>>>>>>> This name "flink-configuration.yaml" provides a clear
>> >>> distinction
>> >>>>>>> between
>> >>>>>>>>> the new and old configuration files based on their names, and
>> >>> it
>> >>>>> does
>> >>>>>>> not
>> >>>>>>>>> introduce any additional semantics. Moreover, this name
>> >>>>>>>>> "flink-configuration.yaml" can continue to be used in future
>> >>>>> versions
>> >>>>>>>>> FLINK-2.0.
>> >>>>>>>>>
>> >>>>>>>>> WDYT? If we can reach a consensus on this, I will update the
>> >>> FLIP
>> >>>>>>>>> documentation
>> >>>>>>>>> accordingly.
>> >>>>>>>>>
>> >>>>>>>>> Best regards,
>> >>>>>>>>> Junrui
>> >>>>>>>>>
>> >>>>>>>>> Jane Chan <qingyue....@gmail.com> 于2023年9月20日周三 23:38写道：
>> >>>>>>>>>
>> >>>>>>>>>> Hi Junrui,
>> >>>>>>>>>>
>> >>>>>>>>>> Thanks for driving this FLIP. +1 for adoption of the
>> >> standard
>> >>>>> YAML
>> >>>>>>>>> syntax.
>> >>>>>>>>>> I just have one minor suggestion. It's a little bit
>> >>> challenging
>> >>>>> to
>> >>>>>>>>>> differentiate between `flink-config.yaml` and
>> >>> `flink-conf.yaml`
>> >>>>> to
>> >>>>>>>>>> determine which one uses the standard syntax at a glance.
>> >> How
>> >>>>> about
>> >>>>>>>>>> using `flink-conf-default.yaml` to represent the default
>> >> yaml
>> >>>>> file
>> >>>>>>> for
>> >>>>>>>>>> Flink 1.x?
>> >>>>>>>>>>
>> >>>>>>>>>> Best,
>> >>>>>>>>>> Jane
>> >>>>>>>>>>
>> >>>>>>>>>> On Wed, Sep 20, 2023 at 11:06 AM Junrui Lee <
>> >>>> jrlee....@gmail.com
>> >>>>>>>> wrote:
>> >>>>>>>>>>> Hi devs,
>> >>>>>>>>>>>
>> >>>>>>>>>>> I would like to start a discussion about FLIP-366:
>> >>>>>>>>>>> Support standard YAML for FLINK configuration[1]
>> >>>>>>>>>>>
>> >>>>>>>>>>> The current flink-conf.yaml parser in FLINK is not a
>> >>> standard
>> >>>>> YAML
>> >>>>>>>>>> parser,
>> >>>>>>>>>>> which has some shortcomings.
>> >>>>>>>>>>> Firstly, it does not support nested structure
>> >> configuration
>> >>>>> items
>> >>>>>>> and
>> >>>>>>>>>> only
>> >>>>>>>>>>> supports key-value pairs, resulting in poor readability.
>> >>>>> Secondly,
>> >>>>>>> if
>> >>>>>>>>> the
>> >>>>>>>>>>> value is a collection type, such as a List or Map, users
>> >>> are
>> >>>>>>> required
>> >>>>>>>>> to
>> >>>>>>>>>>> write the value in a FLINK-specific pattern, which is
>> >>>>> inconvenient
>> >>>>>>> to
>> >>>>>>>>>> use.
>> >>>>>>>>>>> Additionally, the parser of FLINK has some differences in
>> >>>>> syntax
>> >>>>>>>>> compared
>> >>>>>>>>>>> to the standard YAML parser, such as the syntax for
>> >> parsing
>> >>>>>>> comments
>> >>>>>>>>> and
>> >>>>>>>>>>> null values. These inconsistencies can cause confusion
>> >> for
>> >>>>> users,
>> >>>>>>> as
>> >>>>>>>>> seen
>> >>>>>>>>>>> in FLINK-15358 and FLINK-32740.
>> >>>>>>>>>>>
>> >>>>>>>>>>> By supporting standard YAML, these issues can be
>> >> resolved,
>> >>>> and
>> >>>>>>> users
>> >>>>>>>>> can
>> >>>>>>>>>>> create a Flink configuration file using third-party tools
>> >>> and
>> >>>>>>>> leverage
>> >>>>>>>>>>> some advanced YAML features. Therefore, we propose to
>> >>> support
>> >>>>>>>> standard
>> >>>>>>>>>>> YAML for FLINK configuration.
>> >>>>>>>>>>>
>> >>>>>>>>>>> You can find more details in the FLIP-366[1]. Looking
>> >>> forward
>> >>>>> to
>> >>>>>>> your
>> >>>>>>>>>>> feedback.
>> >>>>>>>>>>>
>> >>>>>>>>>>> [1]
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration
>> >>>>>>>>>>> Best,
>> >>>>>>>>>>> Junrui
>> >>>>>>>>>>>
>> >>>>>>>
>> >>>>>>> --
>> >>>>>>> Best
>> >>>>>>>
>> >>>>>>> ConradJam
>> >>>>>>>
>>
>> Unless otherwise stated above:
>>
>> IBM United Kingdom Limited
>> Registered in England and Wales with number 741598
>> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>>
>
