Hi,
I agree this is a standardising, simplifying change for read, simplifying
programmatically authoring the config file as well. As you know the mapping
for the old config form to the new form, could we write a migration tool, that
would convert the old config files to the new format, with errors and warnings
if the content is such that the user would need to manually fix up the file.
If there are minimal user fixups, we should consider automatically migrating
the config file to the new format,
Kind regards, David.
From: Chesnay Schepler <ches...@apache.org>
Date: Tuesday, 3 October 2023 at 11:17
To: dev@flink.apache.org <dev@flink.apache.org>
Subject: [EXTERNAL] Re: [Discuss] FLIP-366: Support standard YAML for FLINK
configuration
It is a unfortunate that we'll need a separate config file but the FLIP
does a good job justifying it.
Personally I'd just name it "config.yaml"; I never quite understood why
there was a flink prefix to begin with, and the current proposal
("flink-configuration.yaml") seems unnecessarily long.
For the deprecation process we could consider logging a warning if the
old parser is used.
In terms of scope, is the migration of existing e2e tests and the docker
image to the new parser part of the FLIP?
On 22/09/2023 09:32, Jane Chan wrote:
> Hi, Junrui,
>
> Sorry for the late reply. The update looks good to me and thanks for your
> effort!
>
> Best,
> Jane
>
> On Fri, Sep 22, 2023 at 2:25 PM Yuxin Tan <tanyuxinw...@gmail.com> wrote:
>
>> Hi, Junrui
>>
>> +1 for the proposal.
>> Thanks for your effort.
>>
>> Best,
>> Yuxin
>>
>>
>> Samrat Deb <decordea...@gmail.com> 于2023年9月22日周五 13:23写道：
>>
>>> Hello Junrui,
>>>
>>> +1 for the proposal.
>>>
>>>
>>> Bests,
>>> Samrat
>>>
>>> On Fri, Sep 22, 2023 at 10:18 AM Shammon FY <zjur...@gmail.com> wrote:
>>>
>>>> +1 for the proposal, thanks for driving.
>>>>
>>>> Bet,
>>>> Shammon FY
>>>>
>>>> On Fri, Sep 22, 2023 at 12:41 PM Yangze Guo <karma...@gmail.com>
>> wrote:
>>>>> Thanks for driving this, +1 for the proposal.
>>>>>
>>>>> Best,
>>>>> Yangze Guo
>>>>>
>>>>>
>>>>> On Fri, Sep 22, 2023 at 11:59 AM Lijie Wang <
>> wangdachui9...@gmail.com>
>>>>> wrote:
>>>>>> Hi Junrui,
>>>>>>
>>>>>> +1 for this proposal, thanks for driving.
>>>>>>
>>>>>> Best,
>>>>>> Lijie
>>>>>>
>>>>>> ConradJam <jam.gz...@gmail.com> 于2023年9月22日周五 10:07写道：
>>>>>>
>>>>>>> +1 Support for standard YAML format facilitates specification
>>>>>>>
>>>>>>> Jing Ge <j...@ververica.com.invalid> 于2023年9月22日周五 02:23写道：
>>>>>>>
>>>>>>>> Hi Junrui,
>>>>>>>>
>>>>>>>> +1 for following the standard. Thanks for your effort!
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Jing
>>>>>>>>
>>>>>>>> On Thu, Sep 21, 2023 at 5:09 AM Junrui Lee <
>> jrlee....@gmail.com>
>>>>> wrote:
>>>>>>>>> Hi Jane,
>>>>>>>>>
>>>>>>>>> Thank you for your valuable feedback and suggestions.
>>>>>>>>> I agree with your point about differentiating between
>>>>>>> "flink-config.yaml"
>>>>>>>>> and "flink-conf.yaml" to determine the standard syntax at a
>>>> glance.
>>>>>>>>> While I understand your suggestion of using
>>>>> "flink-conf-default.yaml"
>>>>>>> to
>>>>>>>>> represent the default YAML file for Flink 1.x, I have been
>>>>> considering
>>>>>>>>> the option of using "flink-configuration.yaml" as the file
>> name
>>>>> for the
>>>>>>>>> new configuration file.
>>>>>>>>> This name "flink-configuration.yaml" provides a clear
>>> distinction
>>>>>>> between
>>>>>>>>> the new and old configuration files based on their names, and
>>> it
>>>>> does
>>>>>>> not
>>>>>>>>> introduce any additional semantics. Moreover, this name
>>>>>>>>> "flink-configuration.yaml" can continue to be used in future
>>>>> versions
>>>>>>>>> FLINK-2.0.
>>>>>>>>>
>>>>>>>>> WDYT? If we can reach a consensus on this, I will update the
>>> FLIP
>>>>>>>>> documentation
>>>>>>>>> accordingly.
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>> Junrui
>>>>>>>>>
>>>>>>>>> Jane Chan <qingyue....@gmail.com> 于2023年9月20日周三 23:38写道：
>>>>>>>>>
>>>>>>>>>> Hi Junrui,
>>>>>>>>>>
>>>>>>>>>> Thanks for driving this FLIP. +1 for adoption of the
>> standard
>>>>> YAML
>>>>>>>>> syntax.
>>>>>>>>>> I just have one minor suggestion. It's a little bit
>>> challenging
>>>>> to
>>>>>>>>>> differentiate between `flink-config.yaml` and
>>> `flink-conf.yaml`
>>>>> to
>>>>>>>>>> determine which one uses the standard syntax at a glance.
>> How
>>>>> about
>>>>>>>>>> using `flink-conf-default.yaml` to represent the default
>> yaml
>>>>> file
>>>>>>> for
>>>>>>>>>> Flink 1.x?
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Jane
>>>>>>>>>>
>>>>>>>>>> On Wed, Sep 20, 2023 at 11:06 AM Junrui Lee <
>>>> jrlee....@gmail.com
>>>>>>>> wrote:
>>>>>>>>>>> Hi devs,
>>>>>>>>>>>
>>>>>>>>>>> I would like to start a discussion about FLIP-366:
>>>>>>>>>>> Support standard YAML for FLINK configuration[1]
>>>>>>>>>>>
>>>>>>>>>>> The current flink-conf.yaml parser in FLINK is not a
>>> standard
>>>>> YAML
>>>>>>>>>> parser,
>>>>>>>>>>> which has some shortcomings.
>>>>>>>>>>> Firstly, it does not support nested structure
>> configuration
>>>>> items
>>>>>>> and
>>>>>>>>>> only
>>>>>>>>>>> supports key-value pairs, resulting in poor readability.
>>>>> Secondly,
>>>>>>> if
>>>>>>>>> the
>>>>>>>>>>> value is a collection type, such as a List or Map, users
>>> are
>>>>>>> required
>>>>>>>>> to
>>>>>>>>>>> write the value in a FLINK-specific pattern, which is
>>>>> inconvenient
>>>>>>> to
>>>>>>>>>> use.
>>>>>>>>>>> Additionally, the parser of FLINK has some differences in
>>>>> syntax
>>>>>>>>> compared
>>>>>>>>>>> to the standard YAML parser, such as the syntax for
>> parsing
>>>>>>> comments
>>>>>>>>> and
>>>>>>>>>>> null values. These inconsistencies can cause confusion
>> for
>>>>> users,
>>>>>>> as
>>>>>>>>> seen
>>>>>>>>>>> in FLINK-15358 and FLINK-32740.
>>>>>>>>>>>
>>>>>>>>>>> By supporting standard YAML, these issues can be
>> resolved,
>>>> and
>>>>>>> users
>>>>>>>>> can
>>>>>>>>>>> create a Flink configuration file using third-party tools
>>> and
>>>>>>>> leverage
>>>>>>>>>>> some advanced YAML features. Therefore, we propose to
>>> support
>>>>>>>> standard
>>>>>>>>>>> YAML for FLINK configuration.
>>>>>>>>>>>
>>>>>>>>>>> You can find more details in the FLIP-366[1]. Looking
>>> forward
>>>>> to
>>>>>>> your
>>>>>>>>>>> feedback.
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>>
>>>>>>>>>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration
>>>>>>>>>>> Best,
>>>>>>>>>>> Junrui
>>>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best
>>>>>>>
>>>>>>> ConradJam
>>>>>>>
Unless otherwise stated above:
IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
