Hi, Chesnay and David, sorry for the late reply.
Thanks for your feedback about this FLIP.
@Chesnay
>> Personally I'd just name it "config.yaml"
Thanks for your suggestions, I agree that "config.yaml" is a simpler and
more intuitive name. I have updated the FLIP accordingly.
>> In terms of scope, is the migration of existing e2e tests and the docker
image to the new parser part of the FLIP?
The migration of existing end-to-end tests and the docker image to the new
parser is indeed included in the FLIP. Taking into account the difficulty
of using shell scripts to modify configuration files that adhere to
standard YAML syntax, I propose utilizing the
org.apache.flink.runtime.util.bash.BashJavaUtils class as a unified utility
for modifying the configuration file.
In addition, considering that users may not only configure properties in
the configuration file, but also use dynamic parameters like '-D' or
configure the "with" clauses in FLINK SQL. When the default parser is
changed to the standard YAML parser, the configOption whose value is List
or Map types also needs to be modified accordingly. This migration cost is
relatively high because users have to modify their code or command lines.
To reduce migration costs while considering compatibility, I propose to add
compatibility support in FLINK-1.X versions to parse old format List and
Map values using the standard YAML parser. In case of an error while
parsing List or Map configurations with the standard YAML parser, a WARNING
log will be printed, and an attempt will be made to parse using the old
parser. And this support will be removed in the FLINK-2.0 version.
Do you think the aforementioned compatibility handling is necessary in
order to reduce migration costs? If you agree, I will also update this
support in the FLIP under the migration section.
@David
>> could we write a migration tool, that would convert the old config files
to the new format,
I agree that providing a more user-friendly guidance for the migration is
important. In the FLIP, we can include a user migration manual to help
users with the migration process. This migration manual will be presented
on the official FLINK website, making it easily accessible for users. And
the migrated configuration files will adhere to standard YAML syntax, and
users can validate them using various third-party YAML tools.
Regarding the automated migration tool, I think it can be considered as a
follow-up task rather than a requirement for the initial version. We can
focus on implementing the migration manually first and then explore the
possibility of automating it in future updates.
Best,
Junrui
David Radley <david_rad...@uk.ibm.com> 于2023年10月3日周二 18:56写道：
> Hi,
> I agree this is a standardising, simplifying change for read, simplifying
> programmatically authoring the config file as well. As you know the
> mapping for the old config form to the new form, could we write a migration
> tool, that would convert the old config files to the new format, with
> errors and warnings if the content is such that the user would need to
> manually fix up the file.
> If there are minimal user fixups, we should consider automatically
> migrating the config file to the new format,
> Kind regards, David.
>
>
>
>
> From: Chesnay Schepler <ches...@apache.org>
> Date: Tuesday, 3 October 2023 at 11:17
> To: dev@flink.apache.org <dev@flink.apache.org>
> Subject: [EXTERNAL] Re: [Discuss] FLIP-366: Support standard YAML for
> FLINK configuration
> It is a unfortunate that we'll need a separate config file but the FLIP
> does a good job justifying it.
>
> Personally I'd just name it "config.yaml"; I never quite understood why
> there was a flink prefix to begin with, and the current proposal
> ("flink-configuration.yaml") seems unnecessarily long.
>
> For the deprecation process we could consider logging a warning if the
> old parser is used.
>
> In terms of scope, is the migration of existing e2e tests and the docker
> image to the new parser part of the FLIP?
>
> On 22/09/2023 09:32, Jane Chan wrote:
> > Hi, Junrui,
> >
> > Sorry for the late reply. The update looks good to me and thanks for your
> > effort!
> >
> > Best,
> > Jane
> >
> > On Fri, Sep 22, 2023 at 2:25 PM Yuxin Tan <tanyuxinw...@gmail.com>
> wrote:
> >
> >> Hi, Junrui
> >>
> >> +1 for the proposal.
> >> Thanks for your effort.
> >>
> >> Best,
> >> Yuxin
> >>
> >>
> >> Samrat Deb <decordea...@gmail.com> 于2023年9月22日周五 13:23写道：
> >>
> >>> Hello Junrui,
> >>>
> >>> +1 for the proposal.
> >>>
> >>>
> >>> Bests,
> >>> Samrat
> >>>
> >>> On Fri, Sep 22, 2023 at 10:18 AM Shammon FY <zjur...@gmail.com> wrote:
> >>>
> >>>> +1 for the proposal, thanks for driving.
> >>>>
> >>>> Bet,
> >>>> Shammon FY
> >>>>
> >>>> On Fri, Sep 22, 2023 at 12:41 PM Yangze Guo <karma...@gmail.com>
> >> wrote:
> >>>>> Thanks for driving this, +1 for the proposal.
> >>>>>
> >>>>> Best,
> >>>>> Yangze Guo
> >>>>>
> >>>>>
> >>>>> On Fri, Sep 22, 2023 at 11:59 AM Lijie Wang <
> >> wangdachui9...@gmail.com>
> >>>>> wrote:
> >>>>>> Hi Junrui,
> >>>>>>
> >>>>>> +1 for this proposal, thanks for driving.
> >>>>>>
> >>>>>> Best,
> >>>>>> Lijie
> >>>>>>
> >>>>>> ConradJam <jam.gz...@gmail.com> 于2023年9月22日周五 10:07写道：
> >>>>>>
> >>>>>>> +1 Support for standard YAML format facilitates specification
> >>>>>>>
> >>>>>>> Jing Ge <j...@ververica.com.invalid> 于2023年9月22日周五 02:23写道：
> >>>>>>>
> >>>>>>>> Hi Junrui,
> >>>>>>>>
> >>>>>>>> +1 for following the standard. Thanks for your effort!
> >>>>>>>>
> >>>>>>>> Best regards,
> >>>>>>>> Jing
> >>>>>>>>
> >>>>>>>> On Thu, Sep 21, 2023 at 5:09 AM Junrui Lee <
> >> jrlee....@gmail.com>
> >>>>> wrote:
> >>>>>>>>> Hi Jane,
> >>>>>>>>>
> >>>>>>>>> Thank you for your valuable feedback and suggestions.
> >>>>>>>>> I agree with your point about differentiating between
> >>>>>>> "flink-config.yaml"
> >>>>>>>>> and "flink-conf.yaml" to determine the standard syntax at a
> >>>> glance.
> >>>>>>>>> While I understand your suggestion of using
> >>>>> "flink-conf-default.yaml"
> >>>>>>> to
> >>>>>>>>> represent the default YAML file for Flink 1.x, I have been
> >>>>> considering
> >>>>>>>>> the option of using "flink-configuration.yaml" as the file
> >> name
> >>>>> for the
> >>>>>>>>> new configuration file.
> >>>>>>>>> This name "flink-configuration.yaml" provides a clear
> >>> distinction
> >>>>>>> between
> >>>>>>>>> the new and old configuration files based on their names, and
> >>> it
> >>>>> does
> >>>>>>> not
> >>>>>>>>> introduce any additional semantics. Moreover, this name
> >>>>>>>>> "flink-configuration.yaml" can continue to be used in future
> >>>>> versions
> >>>>>>>>> FLINK-2.0.
> >>>>>>>>>
> >>>>>>>>> WDYT? If we can reach a consensus on this, I will update the
> >>> FLIP
> >>>>>>>>> documentation
> >>>>>>>>> accordingly.
> >>>>>>>>>
> >>>>>>>>> Best regards,
> >>>>>>>>> Junrui
> >>>>>>>>>
> >>>>>>>>> Jane Chan <qingyue....@gmail.com> 于2023年9月20日周三 23:38写道：
> >>>>>>>>>
> >>>>>>>>>> Hi Junrui,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for driving this FLIP. +1 for adoption of the
> >> standard
> >>>>> YAML
> >>>>>>>>> syntax.
> >>>>>>>>>> I just have one minor suggestion. It's a little bit
> >>> challenging
> >>>>> to
> >>>>>>>>>> differentiate between `flink-config.yaml` and
> >>> `flink-conf.yaml`
> >>>>> to
> >>>>>>>>>> determine which one uses the standard syntax at a glance.
> >> How
> >>>>> about
> >>>>>>>>>> using `flink-conf-default.yaml` to represent the default
> >> yaml
> >>>>> file
> >>>>>>> for
> >>>>>>>>>> Flink 1.x?
> >>>>>>>>>>
> >>>>>>>>>> Best,
> >>>>>>>>>> Jane
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Sep 20, 2023 at 11:06 AM Junrui Lee <
> >>>> jrlee....@gmail.com
> >>>>>>>> wrote:
> >>>>>>>>>>> Hi devs,
> >>>>>>>>>>>
> >>>>>>>>>>> I would like to start a discussion about FLIP-366:
> >>>>>>>>>>> Support standard YAML for FLINK configuration[1]
> >>>>>>>>>>>
> >>>>>>>>>>> The current flink-conf.yaml parser in FLINK is not a
> >>> standard
> >>>>> YAML
> >>>>>>>>>> parser,
> >>>>>>>>>>> which has some shortcomings.
> >>>>>>>>>>> Firstly, it does not support nested structure
> >> configuration
> >>>>> items
> >>>>>>> and
> >>>>>>>>>> only
> >>>>>>>>>>> supports key-value pairs, resulting in poor readability.
> >>>>> Secondly,
> >>>>>>> if
> >>>>>>>>> the
> >>>>>>>>>>> value is a collection type, such as a List or Map, users
> >>> are
> >>>>>>> required
> >>>>>>>>> to
> >>>>>>>>>>> write the value in a FLINK-specific pattern, which is
> >>>>> inconvenient
> >>>>>>> to
> >>>>>>>>>> use.
> >>>>>>>>>>> Additionally, the parser of FLINK has some differences in
> >>>>> syntax
> >>>>>>>>> compared
> >>>>>>>>>>> to the standard YAML parser, such as the syntax for
> >> parsing
> >>>>>>> comments
> >>>>>>>>> and
> >>>>>>>>>>> null values. These inconsistencies can cause confusion
> >> for
> >>>>> users,
> >>>>>>> as
> >>>>>>>>> seen
> >>>>>>>>>>> in FLINK-15358 and FLINK-32740.
> >>>>>>>>>>>
> >>>>>>>>>>> By supporting standard YAML, these issues can be
> >> resolved,
> >>>> and
> >>>>>>> users
> >>>>>>>>> can
> >>>>>>>>>>> create a Flink configuration file using third-party tools
> >>> and
> >>>>>>>> leverage
> >>>>>>>>>>> some advanced YAML features. Therefore, we propose to
> >>> support
> >>>>>>>> standard
> >>>>>>>>>>> YAML for FLINK configuration.
> >>>>>>>>>>>
> >>>>>>>>>>> You can find more details in the FLIP-366[1]. Looking
> >>> forward
> >>>>> to
> >>>>>>> your
> >>>>>>>>>>> feedback.
> >>>>>>>>>>>
> >>>>>>>>>>> [1]
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration
> >>>>>>>>>>> Best,
> >>>>>>>>>>> Junrui
> >>>>>>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> Best
> >>>>>>>
> >>>>>>> ConradJam
> >>>>>>>
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>
