Re: [DISCUSS] Should Configuration support getting value based on String key?

[Gyula Fóra]

Hi Xintong,

I don’t really see the actual practical benefit from removing the getstring
and setstring low level methods.

I understand that ConfigOptions are nicer for 95% of the cases but from a
technical point of view there will always be cases where a more flexible
dynamic handling is necessary without the added overhead of the toMap logic.

I think it’s the most natural thing for any config abstraction to expose
basic get set methods with a simple key.

What do you think?

Cheers
Gyula

On Thu, 14 Dec 2023 at 08:00, Xintong Song  wrote:

> >
> > IIUC, you both prefer using ConfigOption instead of string keys for
> > all use cases, even internal ones. We can even forcefully delete
> > these @Depreated methods in Flink-2.0 to guide users or
> > developers to use ConfigOption.
> >
>
> Yes, at least from my side.
>
>
> I noticed that Configuration is used in
> > DistributedCache#writeFileInfoToConfig and readFileInfoFromConfig
> > to store some cacheFile meta-information. Their keys are
> > temporary(key name with number) and it is not convenient
> > to predefine ConfigOption.
> >
>
> True, this one requires a bit more effort to migrate from string-key to
> ConfigOption, but still should be doable. Looking at how the two mentioned
> methods are implemented and used, it seems what is really needed is
> serialization and deserialization of `DistributedCacheEntry`-s. And all the
> entries are always written / read at once. So I think we can serialize the
> whole set of entries into a JSON string (or something similar), and use one
> ConfigOption with a deterministic key for it, rather than having one
> ConfigOption for each field in each entry. WDYT?
>
>
> If everyone agrees with this direction, we can start to refactor all
> > code that uses getXxx(String key, String defaultValue) into
> > getXxx(ConfigOption configOption), and completely
> > delete all getXxx(String key, String defaultValue) in 2.0.
> > And I'm willing to pick it up~
> >
>
> Exactly. Actually, Xuannan and a few other colleagues are working on
> reviewing all the existing configuration options. We identified some common
> issues that can potentially be improved, and not using string-key is one of
> them. We are still summarizing the findings, and will bring them to the
> community for discussion once ready. Helping hands is definitely welcomed.
> :)
>
>
> Yeah, `toMap` can solve the problem, and I also mentioned it in the
> > initial mail.
> >
>
> True. Sorry I overlooked it.
>
>
> Best,
>
> Xintong
>
>
>
> On Thu, Dec 14, 2023 at 2:14 PM Rui Fan <1996fan...@gmail.com> wrote:
>
> > Thanks Xintong and Xuannan for the feedback!
> >
> > IIUC, you both prefer using ConfigOption instead of string keys for
> > all use cases, even internal ones. We can even forcefully delete
> > these @Depreated methods in Flink-2.0 to guide users or
> > developers to use ConfigOption.
> >
> > Using ConfigOption is feasible in all scenarios because ConfigOption
> > can be easily created via
> > `ConfigOptions.key("xxx").stringType().noDefaultValue()` even if
> > there is no predefined ConfigOption.
> >
> > I noticed that Configuration is used in
> > DistributedCache#writeFileInfoToConfig and readFileInfoFromConfig
> > to store some cacheFile meta-information. Their keys are
> > temporary(key name with number) and it is not convenient
> > to predefine ConfigOption.
> >
> > If everyone agrees with this direction, we can start to refactor all
> > code that uses getXxx(String key, String defaultValue) into
> > getXxx(ConfigOption configOption), and completely
> > delete all getXxx(String key, String defaultValue) in 2.0.
> > And I'm willing to pick it up~
> >
> > To Xintong:
> >
> > > I think a `toMap` as suggested by Zhu and Xuannan should solve the
> > > problem. Alternatively, we may also consider providing an iterator for
> > the
> > > Configuration.
> >
> > Yeah, `toMap` can solve the problem, and I also mentioned it in the
> > initial mail.
> >
> > Also Providing an iterator is fine, but we don't have a strong
> > requirement for now. Simple is better, how about considering it
> > if we have other strong requirements in the future?
> >
> > Looking forwarding to your feedback, thanks~
> >
> > Best,
> > Rui
> >
> > On Thu, Dec 14, 2023 at 11:36 AM Xintong Song 
> > wrote:
> >
> >> I'm leaning towards not allowing string-key based configuration access
> in
> >> the long term.
> >>
> >> I see the Configuration being used in two different ways:
> >>
> >> 1. Writing / reading a specific option. In such cases, ConfigOption has
> >> many advantages compared to string-key, such as explicit value type,
> >> descriptions, default values, deprecated / fallback keys. I think we
> >> should
> >> encourage, and maybe even enforce, choosing ConfigOption over
> string-keys
> >> in such specific option access scenarios. That also applies to internal
> >> usages, for which the description may not be necessary because we won't
> >> generate documentation from it but we can still 

Re: [VOTE] FLIP-383: Support Job Recovery for Batch Jobs

[Xintong Song]

+1 (binding)

Best,

Xintong



On Thu, Dec 14, 2023 at 3:15 PM Lijie Wang  wrote:

> Hi devs, Thanks for all feedback about the FLIP-383: Support Job Recovery
> for Batch Jobs[1]. This FLIP was discussed in [2].
>
> I'd like to start a vote for it. The vote will be open for at least 72
> hours (until December 19th 12:00 GMT) unless there is an objection or
> insufficient votes. [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-383%3A+Support+Job+Recovery+for+Batch+Jobs
>
> [2] https://lists.apache.org/thread/074z237c07vtj74685nxo6bttkq3kshz
>   Best, Lijie
>

Re:Re: [DISCUSS] FLIP-392: Deprecate the Legacy Group Window Aggregation

[Xuyang]

Hi, Timo, Sergey and Lincoln Lee. Thanks for your feedback.


> In my opinion the FLIP touches too many
> topics at the same time and should be split into multiple FLIPs. We > should 
> stay focused on what is needed for Flink 2.0.
The main goal and topic of this Flip is to align the abilities between the 
legacy group window agg syntax and the new window tvf syntax, 
and then we can say that the legacy window syntax will be deprecated. IMO, 
although there are many misalignments about these two 
syntaxes, such as session window tvf, cdc support and so on,  they are all the 
subtasks we need to do in this flip. Another reason I'm not 
spliting this flip is that all of these subtasks like session window tvf and 
cdc support do not change the public interface and the public 
syntax, the implements of them will only be in modules table-planner and 
table-runtime.


> Can we postpone this discussion? Currently we should focus on user
> switching to Window TVFs before Flink 2.0. Early fire, late fire and > allow 
> lateness have not exposed through public configuration. It can be > 
> introduced after Flink 2.0 released.


Agree with you. This flip will not and should not expose these experimental 
flink conf to users. I list them in this flip just aims to show the 
misalignments about these two window syntaxes.


Look for your thought.




--

Best！
Xuyang





At 2023-12-13 15:40:16, "Lincoln Lee"  wrote:
>Thanks Xuyang driving this work! It's great that everyone agrees with the
>work itself in this flip[1]!
>
>Regarding whether to split the flip or adjust the scope of this flip, I'd
>like to share some thoughts:
>
>1. About the title of this flip, what I want to say is that flip-145[2] had
>marked the legacy group window deprecated in the documentation but the
>functionality of the new syntax is not aligned with the legacy one.
>This is not a user-friendly deprecation, so the initiation of this flip, as
>I understand it, is for the formal deprecation of the legacy window, which
>requires us to complete the functionality alignment.
>
>2. Agree with Timo that we can process the late-fire/early-fire features
>separately. These experimental parameters have not been officially opened
>to users.
>Considering the workload, we can focus more on this version.
>
>3. I have no objection to splitting this flip if everyone feels that the
>work included is too much.
>Regarding the support of session tvf, it seems that the main problem is
>that this part of the description occupies a large part of the flip,
>causing some misunderstandings.
>This is indeed a predetermined task in FLIP-145, just adding more
>explanation about semantics. In addition, I saw the discussion history in
>FLINK-24024[3], thanks Sergey for being willing to help driving this work
>together.
>
>[1]
>https://cwiki.apache.org/confluence/display/FLINK/FLIP-392%3A+Deprecate+the+Legacy+Group+Window+Aggregation
>[2]
>https://cwiki.apache.org/confluence/display/FLINK/FLIP-145%3A+Support+SQL+windowing+table-valued+function
>[3] https://issues.apache.org/jira/browse/FLINK-24024
>
>Best,
>Lincoln Lee
>
>
>Sergey Nuyanzin  于2023年12月13日周三 08:02写道：
>
>> thanks for summarising Timo
>>
>> +1 for splitting it in different FLIPs
>> and agree about having "SESSION Window TVF Aggregation" under FLIP-145
>> Moreover the task is already there, so no need to move it from one FLIP to
>> another
>>
>> >And actually Sergey Nuyanzin wanted to work
>> >in this if I remember correctly. Not sure if this is still the case.
>>
>> Yes, however it seems there is already an existing PR for that.
>> Anyway I'm happy to help here with the review as well and will reserve some
>> time for it in coming days.
>>
>>
>>
>> On Tue, Dec 12, 2023 at 12:18 PM Timo Walther  wrote:
>>
>> > Hi Xuyang,
>> >
>> > thanks for proposing this FLIP. In my opinion the FLIP touches too many
>> > topics at the same time and should be split into multiple FLIPs. We
>> > should stay focused on what is needed for Flink 2.0.
>> >
>> > Let me summarizing the topics:
>> >
>> > 1) SESSION Window TVF Aggregation
>> >
>> > This has been agreed in FLIP-145 already. We don't need another FLIP but
>> > someone that finally implements this after we have performed the Calcite
>> > upgrade a couple of months ago. The Calcite upgrade was important
>> > exactly for SESSION windows. And actually Sergey Nuyanzin wanted to work
>> > in this if I remember correctly. Not sure if this is still the case.
>> >
>> > 2) CDC support of Window TVFs
>> >
>> > This can be a FLIP on its own.
>> >
>> > 3) HOP window size with non-integer step length
>> >
>> > This can be a FLIP on its own.
>> >
>> > 4) Configurations such as early fire, late fire and allow lateness
>> >
>> > Can we postpone this discussion? Currently we should focus on user
>> > switching to Window TVFs before Flink 2.0. Early fire, late fire and
>> > allow lateness have not exposed through public configuration. It can be
>> > introduced after Flink 2.0 released.
>> 

[VOTE] FLIP-383: Support Job Recovery for Batch Jobs

[Lijie Wang]

Hi devs, Thanks for all feedback about the FLIP-383: Support Job Recovery
for Batch Jobs[1]. This FLIP was discussed in [2].

I'd like to start a vote for it. The vote will be open for at least 72
hours (until December 19th 12:00 GMT) unless there is an objection or
insufficient votes. [1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-383%3A+Support+Job+Recovery+for+Batch+Jobs

[2] https://lists.apache.org/thread/074z237c07vtj74685nxo6bttkq3kshz
  Best, Lijie

Re: [DISCUSS] Should Configuration support getting value based on String key?

[Xintong Song]

>
> IIUC, you both prefer using ConfigOption instead of string keys for
> all use cases, even internal ones. We can even forcefully delete
> these @Depreated methods in Flink-2.0 to guide users or
> developers to use ConfigOption.
>

Yes, at least from my side.


I noticed that Configuration is used in
> DistributedCache#writeFileInfoToConfig and readFileInfoFromConfig
> to store some cacheFile meta-information. Their keys are
> temporary(key name with number) and it is not convenient
> to predefine ConfigOption.
>

True, this one requires a bit more effort to migrate from string-key to
ConfigOption, but still should be doable. Looking at how the two mentioned
methods are implemented and used, it seems what is really needed is
serialization and deserialization of `DistributedCacheEntry`-s. And all the
entries are always written / read at once. So I think we can serialize the
whole set of entries into a JSON string (or something similar), and use one
ConfigOption with a deterministic key for it, rather than having one
ConfigOption for each field in each entry. WDYT?


If everyone agrees with this direction, we can start to refactor all
> code that uses getXxx(String key, String defaultValue) into
> getXxx(ConfigOption configOption), and completely
> delete all getXxx(String key, String defaultValue) in 2.0.
> And I'm willing to pick it up~
>

Exactly. Actually, Xuannan and a few other colleagues are working on
reviewing all the existing configuration options. We identified some common
issues that can potentially be improved, and not using string-key is one of
them. We are still summarizing the findings, and will bring them to the
community for discussion once ready. Helping hands is definitely welcomed.
:)


Yeah, `toMap` can solve the problem, and I also mentioned it in the
> initial mail.
>

True. Sorry I overlooked it.


Best,

Xintong



On Thu, Dec 14, 2023 at 2:14 PM Rui Fan <1996fan...@gmail.com> wrote:

> Thanks Xintong and Xuannan for the feedback!
>
> IIUC, you both prefer using ConfigOption instead of string keys for
> all use cases, even internal ones. We can even forcefully delete
> these @Depreated methods in Flink-2.0 to guide users or
> developers to use ConfigOption.
>
> Using ConfigOption is feasible in all scenarios because ConfigOption
> can be easily created via
> `ConfigOptions.key("xxx").stringType().noDefaultValue()` even if
> there is no predefined ConfigOption.
>
> I noticed that Configuration is used in
> DistributedCache#writeFileInfoToConfig and readFileInfoFromConfig
> to store some cacheFile meta-information. Their keys are
> temporary(key name with number) and it is not convenient
> to predefine ConfigOption.
>
> If everyone agrees with this direction, we can start to refactor all
> code that uses getXxx(String key, String defaultValue) into
> getXxx(ConfigOption configOption), and completely
> delete all getXxx(String key, String defaultValue) in 2.0.
> And I'm willing to pick it up~
>
> To Xintong:
>
> > I think a `toMap` as suggested by Zhu and Xuannan should solve the
> > problem. Alternatively, we may also consider providing an iterator for
> the
> > Configuration.
>
> Yeah, `toMap` can solve the problem, and I also mentioned it in the
> initial mail.
>
> Also Providing an iterator is fine, but we don't have a strong
> requirement for now. Simple is better, how about considering it
> if we have other strong requirements in the future?
>
> Looking forwarding to your feedback, thanks~
>
> Best,
> Rui
>
> On Thu, Dec 14, 2023 at 11:36 AM Xintong Song 
> wrote:
>
>> I'm leaning towards not allowing string-key based configuration access in
>> the long term.
>>
>> I see the Configuration being used in two different ways:
>>
>> 1. Writing / reading a specific option. In such cases, ConfigOption has
>> many advantages compared to string-key, such as explicit value type,
>> descriptions, default values, deprecated / fallback keys. I think we
>> should
>> encourage, and maybe even enforce, choosing ConfigOption over string-keys
>> in such specific option access scenarios. That also applies to internal
>> usages, for which the description may not be necessary because we won't
>> generate documentation from it but we can still benefit from other
>> advantages.
>>
>> 2. Iterating over all the configured options. In such cases, it is
>> currently impractical to find the proper ConfigOption for each configured
>> option. I think a `toMap` as suggested by Zhu and Xuannan should solve the
>> problem. Alternatively, we may also consider providing an iterator for the
>> Configuration.
>>
>> I think if we want to encourage using ConfigOption in case-1, the most
>> effective way is to not allow accessing a specific option with a
>> string-key, so that developers not awaring of this discussion won't
>> accidentally use it. On the other hand, case-2 is a much rarer use case
>> compared to case-1, and given the fact that there are alternative
>> approaches, I think we should not compromise 

Re: [DISCUSS] Should Configuration support getting value based on String key?

[Rui Fan]

Thanks Xintong and Xuannan for the feedback!

IIUC, you both prefer using ConfigOption instead of string keys for
all use cases, even internal ones. We can even forcefully delete
these @Depreated methods in Flink-2.0 to guide users or
developers to use ConfigOption.

Using ConfigOption is feasible in all scenarios because ConfigOption
can be easily created via
`ConfigOptions.key("xxx").stringType().noDefaultValue()` even if
there is no predefined ConfigOption.

I noticed that Configuration is used in
DistributedCache#writeFileInfoToConfig and readFileInfoFromConfig
to store some cacheFile meta-information. Their keys are
temporary(key name with number) and it is not convenient
to predefine ConfigOption.

If everyone agrees with this direction, we can start to refactor all
code that uses getXxx(String key, String defaultValue) into
getXxx(ConfigOption configOption), and completely
delete all getXxx(String key, String defaultValue) in 2.0.
And I'm willing to pick it up~

To Xintong:

> I think a `toMap` as suggested by Zhu and Xuannan should solve the
> problem. Alternatively, we may also consider providing an iterator for the
> Configuration.

Yeah, `toMap` can solve the problem, and I also mentioned it in the
initial mail.

Also Providing an iterator is fine, but we don't have a strong
requirement for now. Simple is better, how about considering it
if we have other strong requirements in the future?

Looking forwarding to your feedback, thanks~

Best,
Rui

On Thu, Dec 14, 2023 at 11:36 AM Xintong Song  wrote:

> I'm leaning towards not allowing string-key based configuration access in
> the long term.
>
> I see the Configuration being used in two different ways:
>
> 1. Writing / reading a specific option. In such cases, ConfigOption has
> many advantages compared to string-key, such as explicit value type,
> descriptions, default values, deprecated / fallback keys. I think we should
> encourage, and maybe even enforce, choosing ConfigOption over string-keys
> in such specific option access scenarios. That also applies to internal
> usages, for which the description may not be necessary because we won't
> generate documentation from it but we can still benefit from other
> advantages.
>
> 2. Iterating over all the configured options. In such cases, it is
> currently impractical to find the proper ConfigOption for each configured
> option. I think a `toMap` as suggested by Zhu and Xuannan should solve the
> problem. Alternatively, we may also consider providing an iterator for the
> Configuration.
>
> I think if we want to encourage using ConfigOption in case-1, the most
> effective way is to not allow accessing a specific option with a
> string-key, so that developers not awaring of this discussion won't
> accidentally use it. On the other hand, case-2 is a much rarer use case
> compared to case-1, and given the fact that there are alternative
> approaches, I think we should not compromise case-1 for it.
>
> WDYT?
>
> Best,
>
> Xintong
>
>
>
> On Thu, Dec 14, 2023 at 10:25 AM Xuannan Su  wrote:
>
> > Hi Rui,
> >
> > We are currently revisiting all the configurations for Flink 2.0, and
> > it turns out that many string-based configurations in
> > `ConfigConstants` are deprecated and have been replaced by
> > `ConfigOptions`. Since `ConfigOptions` offers many advantages over
> > string-based configurations for the end user, I believe we should
> > encourage users to set and get the Flink configuration exclusively
> > with `ConfigOption`. And we are going to eventually replace all the
> > string-based configurations with `ConfigOptions` for this use case.
> >
> > For the first use case you mentioned, I think they are all internal
> usage,
> > and we should aim to replace them with ConfigOptions gradually.
> > Meanwhile, we may consider making those getters/setters for internal
> > use only while the replacement is in progress.
> >
> > For the second use case, IIUC, you need to iterate over all the
> > configurations to replace some old configuration keys with new ones. I
> > believe  `toMap` is suitable for this scenario.
> >
> > Best,
> > Xuannan
> >
> >
> >
> > On Wed, Dec 13, 2023 at 9:04 PM Rui Fan <1996fan...@gmail.com> wrote:
> > >
> > > Thanks Zhu for the quick response!
> > >
> > > > It is not a blocker of the deprecation, epsecially given that they
> are
> > > not standard
> > > > configuration and are just using Configuration class for convenience.
> > >
> > > Yes, it's not a blocker of deprecation.
> > >
> > > > These are internal usages and we can have an internal getter method
> for
> > > them.
> > >
> > > For case1, do you mean we reuse the old getString method as the
> internal
> > > getter method or add a new getter method?
> > >
> > > Anyway, it's fine for me if we have an internal getter method. As I
> > > understand,
> > > the public method without any annotation will be the internal method,
> > right?
> > >
> > > > Not sure why it's required to convert all keys or values. If it is
> used
> 

Re: [VOTE] Release flink-connector-pulsar 4.1.0, release candidate #1

[tison]

Thanks Leonard for driving this release!

+1 (non-binding)

* Download link valid
* Maven staging artifacts look good.
* Checksum and gpg matches
* LICENSE and NOTICE exist
* Can build from source.

Best,
tison.

Rui Fan <1996fan...@gmail.com> 于2023年12月14日周四 09:23写道：
>
> Thanks Leonard for driving this release!
>
> +1 (non-binding)
>
> - Validated checksum hash
> - Verified signature
> - Verified that no binaries exist in the source archive
> - Build the source with Maven and jdk8
> - Verified licenses
> - Verified web PRs, left a minor comment
>
> Best,
> Rui
>
> On Wed, Dec 13, 2023 at 7:15 PM Leonard Xu  wrote:
>>
>> Hey all,
>>
>> Please review and vote on the release candidate #1 for the version 4.1.0 of 
>> the Apache Flink Pulsar Connector as follows:
>>
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>> The complete staging area is available for your review, which includes:
>> * JIRA release notes [1],
>> * The official Apache source release to be deployed to dist.apache.org [2], 
>> which are signed with the key with fingerprint
>> 5B2F6608732389AEB67331F5B197E1F1108998AD [3],
>> * All artifacts to be deployed to the Maven Central Repository [4],
>> * Source code tag v4.1.0-rc1 [5],
>> * Website pull request listing the new release [6].
>>
>> The vote will be open for at least 72 hours. It is adopted by majority 
>> approval, with at least 3 PMC affirmative votes.
>>
>>
>> Best,
>> Leonard
>>
>> [1] 
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353431
>> [2] 
>> https://dist.apache.org/repos/dist/dev/flink/flink-connector-pulsar-4.1.0-rc1/
>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>> [4] https://repository.apache.org/content/repositories/orgapacheflink-1688/
>> [5] https://github.com/apache/flink-connector-pulsar/tree/v4.1.0-rc1
>> [6] https://github.com/apache/flink-web/pull/703

Re: [VOTE] FLIP-396: Trial to test GitHub Actions as an alternative for Flink's current Azure CI infrastructure

[Xuannan Su]

Thanks for driving this!

+1(non-binding)

Best,
Xuannan

On Thu, Dec 14, 2023 at 11:04 AM Xintong Song  wrote:
>
> +1 (binding)
>
> Best,
>
> Xintong
>
>
>
> On Thu, Dec 14, 2023 at 10:05 AM Jiabao Sun 
> wrote:
>
> > Thanks Matthias for this hard work!
> >
> > +1(non-binding)
> >
> > Best,
> > Jiabao
> >
> >
> > > 2023年12月14日 09:57，Leonard Xu  写道：
> > >
> > > +1(binding)
> > >
> > >
> > > Best,
> > > Leonard
> > >
> > >> 2023年12月13日 下午10:59，Benchao Li  写道：
> > >>
> > >> +1 (binding)
> > >>
> > >> Thanks Matthias for driving it!
> > >>
> > >> Etienne Chauchot  于2023年12月13日周三 21:35写道：
> > >>>
> > >>> Thanks Matthias for your hard work !
> > >>>
> > >>> +1 (binding)
> > >>>
> > >>> Best
> > >>>
> > >>> Etienne
> > >>>
> > >>> Le 12/12/2023 à 11:23, Lincoln Lee a écrit :
> >  +1 (binding)
> > 
> >  Thanks for driving this!
> > 
> >  Best,
> >  Lincoln Lee
> > 
> > 
> >  Yun Tang  于2023年12月12日周二 17:52写道：
> > 
> > > Thanks for Matthias driving this work!
> > >
> > > +1 (binding)
> > >
> > > Best
> > > Yun Tang
> > > 
> > > From: Yangze Guo
> > > Sent: Tuesday, December 12, 2023 16:12
> > > To:dev@flink.apache.org  
> > > Subject: Re: [VOTE] FLIP-396: Trial to test GitHub Actions as an
> > > alternative for Flink's current Azure CI infrastructure
> > >
> > > +1 (binding)
> > >
> > > Best,
> > > Yangze Guo
> > >
> > > On Tue, Dec 12, 2023 at 3:51 PM Yuxin Tan
> > wrote:
> > >> +1 (non binding)
> > >> Thanks for the effort.
> > >>
> > >> Best,
> > >> Yuxin
> > >>
> > >>
> > >> Samrat Deb  于2023年12月12日周二 15:25写道：
> > >>
> > >>> +1 (non binding)
> > >>> Thanks for driving
> > >>>
> > >>> On Tue, 12 Dec 2023 at 11:59 AM, Sergey Nuyanzin<
> > snuyan...@gmail.com>
> > >>> wrote:
> > >>>
> >  +1 (binding)
> > 
> >  Thanks for driving this
> > 
> >  On Tue, Dec 12, 2023, 07:22 Rui Fan<1996fan...@gmail.com>  wrote:
> > 
> > > +1(binding)
> > >
> > > Best,
> > > Rui
> > >
> > > On Tue, Dec 12, 2023 at 11:58 AM weijie guo <
> > > guoweijieres...@gmail.com
> > > wrote:
> > >
> > >> Thanks Matthias for this efforts.
> > >>
> > >> +1(binding)
> > >>
> > >>
> > >> Best regards,
> > >>
> > >> Weijie
> > >>
> > >>
> > >> Matthias Pohl  于2023年12月11日周一
> > >>> 21:51写道：
> > >>> Hi everyone,
> > >>> I'd like to start a vote on FLIP-396 [1]. It covers enabling
> > > GitHub
> > >> Actions
> > >>> (GHA) in Apache Flink. This means that GHA workflows will run
> > > aside
> > > from
> > >>> the usual Azure CI workflows in a trial phase (which ends
> > > earliest
> >  with
> > >> the
> > >>> release of Flink 1.19). Azure CI will still serve as the
> > > project's
> > > ground
> > >>> of truth until the community decides in a final vote to switch
> > > to
> > >>> GHA
> > > or
> > >>> stick to Azure CI.
> > >>>
> > >>> The related discussion thread can be found in [2].
> > >>>
> > >>> The vote will remain open for at least 72 hours and only
> > > concluded
> > >>> if
> > >> there
> > >>> are no objections and enough (i.e. at least 3) binding votes.
> > >>>
> > >>> Matthias
> > >>>
> > >>> [1]
> > >>>
> > >>>
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-396%3A+Trial+to+test+GitHub+Actions+as+an+alternative+for+Flink%27s+current+Azure+CI+infrastructure
> > >>> [2]
> > >>> https://lists.apache.org/thread/h4cmv7l3y8mxx2t435dmq4ltco4sbrgb
> > >>> --
> > >>>
> > >>> [image: Aiven]
> > >>>
> > >>> *Matthias Pohl*
> > >>> Opensource Software Engineer, *Aiven*
> > >>> matthias.p...@aiven.io  |  +49 170 9869525
> > >>> aiven.io|   <
> > >> https://www.facebook.com/aivencloud
> > >>>  <
> > https://twitter.com/aiven_io>
> > >>> *Aiven Deutschland GmbH*
> > >>> Alexanderufer 3-7, 10117 Berlin
> > >>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >>> Amtsgericht Charlottenburg, HRB 209739 B
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >>
> > >> Best,
> > >> Benchao Li
> >
> >

Re: [DISCUSS] Should Configuration support getting value based on String key?

[Xintong Song]

I'm leaning towards not allowing string-key based configuration access in
the long term.

I see the Configuration being used in two different ways:

1. Writing / reading a specific option. In such cases, ConfigOption has
many advantages compared to string-key, such as explicit value type,
descriptions, default values, deprecated / fallback keys. I think we should
encourage, and maybe even enforce, choosing ConfigOption over string-keys
in such specific option access scenarios. That also applies to internal
usages, for which the description may not be necessary because we won't
generate documentation from it but we can still benefit from other
advantages.

2. Iterating over all the configured options. In such cases, it is
currently impractical to find the proper ConfigOption for each configured
option. I think a `toMap` as suggested by Zhu and Xuannan should solve the
problem. Alternatively, we may also consider providing an iterator for the
Configuration.

I think if we want to encourage using ConfigOption in case-1, the most
effective way is to not allow accessing a specific option with a
string-key, so that developers not awaring of this discussion won't
accidentally use it. On the other hand, case-2 is a much rarer use case
compared to case-1, and given the fact that there are alternative
approaches, I think we should not compromise case-1 for it.

WDYT?

Best,

Xintong



On Thu, Dec 14, 2023 at 10:25 AM Xuannan Su  wrote:

> Hi Rui,
>
> We are currently revisiting all the configurations for Flink 2.0, and
> it turns out that many string-based configurations in
> `ConfigConstants` are deprecated and have been replaced by
> `ConfigOptions`. Since `ConfigOptions` offers many advantages over
> string-based configurations for the end user, I believe we should
> encourage users to set and get the Flink configuration exclusively
> with `ConfigOption`. And we are going to eventually replace all the
> string-based configurations with `ConfigOptions` for this use case.
>
> For the first use case you mentioned, I think they are all internal usage,
> and we should aim to replace them with ConfigOptions gradually.
> Meanwhile, we may consider making those getters/setters for internal
> use only while the replacement is in progress.
>
> For the second use case, IIUC, you need to iterate over all the
> configurations to replace some old configuration keys with new ones. I
> believe  `toMap` is suitable for this scenario.
>
> Best,
> Xuannan
>
>
>
> On Wed, Dec 13, 2023 at 9:04 PM Rui Fan <1996fan...@gmail.com> wrote:
> >
> > Thanks Zhu for the quick response!
> >
> > > It is not a blocker of the deprecation, epsecially given that they are
> > not standard
> > > configuration and are just using Configuration class for convenience.
> >
> > Yes, it's not a blocker of deprecation.
> >
> > > These are internal usages and we can have an internal getter method for
> > them.
> >
> > For case1, do you mean we reuse the old getString method as the internal
> > getter method or add a new getter method?
> >
> > Anyway, it's fine for me if we have an internal getter method. As I
> > understand,
> > the public method without any annotation will be the internal method,
> right?
> >
> > > Not sure why it's required to convert all keys or values. If it is used
> > > to create strings for dynamic properties or config files to deploy
> jobs,
> > > toMap()/toFileWritableMap() may be a better choice. They are already
> > > used in this kind of scenarios.
> >
> > For case2, it's really a special scenario, and toMap() is fine for case2.
> > Case2 uses the getString instead of toMap due to getString is easier.
> > Also, kubernetes-operator is also a internal usage, if case1 is solved,
> > case2 also can use the internal getter method.So we can focus on case1.
> >
> > Thank you
> >
> > Best,
> > Rui
> >
> > On Wed, Dec 13, 2023 at 8:01 PM Zhu Zhu  wrote:
> >
> > > Hi Rui,
> > >
> > > I'd like to understand why there is a strong requirement for these
> > > deprecated
> > > methods. The ConfigOption param methods help to do the type conversion
> so
> > > that users do not need to do it by themselves.
> > >
> > > For the 2 reasons to keep these methods mentioned above:
> > >
> > > > 1. A lot of scenarios don't define the ConfigOption, they using
> > > String as the key and value directly, such as: StreamConfig,
> > > TaskConfig, DistributedCache, etc.
> > >
> > > These are internal usages and we can have an internal getter method for
> > > them.
> > > It is not a blocker of the deprecation, epsecially given that they are
> not
> > > standard
> > > configuration and are just using Configuration class for convenience.
> > >
> > > > 2. Some code wanna convert all keys or values, this converting
> > > is generic, so the getString(String key, String defaultValue) is
> needed.
> > > Such as: kubernetes-operator [3].
> > >
> > > Not sure why it's required to convert all keys or values. If it is used
> > > to create strings for dynamic properties or 

[jira] [Created] (FLINK-33819) Suppor setting CompressType in RocksDBStateBackend

[Yue Ma (Jira)]

Yue Ma created FLINK-33819:
--

 Summary: Suppor setting CompressType in RocksDBStateBackend
 Key: FLINK-33819
 URL: https://issues.apache.org/jira/browse/FLINK-33819
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / State Backends
Affects Versions: 1.18.0
Reporter: Yue Ma
 Fix For: 1.19.0
 Attachments: image-2023-12-14-11-32-32-968.png

Currently, RocksDBStateBackend does not support setting the compression level, 
and Snappy is used for compression by default. But we have some scenarios where 
compression will use a lot of CPU resources. Turning off compression can 
significantly reduce CPU overhead. So we may need to support a parameter for 
users to set the CompressType of Rocksdb.

  
!https://internal-api-drive-stream.larkoffice.com/space/api/box/stream/download/preview/ALADbWTMGoD6WexSFGecz2Olnrb/?preview_type=16!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-396: Trial to test GitHub Actions as an alternative for Flink's current Azure CI infrastructure

[Xintong Song]

+1 (binding)

Best,

Xintong



On Thu, Dec 14, 2023 at 10:05 AM Jiabao Sun 
wrote:

> Thanks Matthias for this hard work!
>
> +1(non-binding)
>
> Best,
> Jiabao
>
>
> > 2023年12月14日 09:57，Leonard Xu  写道：
> >
> > +1(binding)
> >
> >
> > Best,
> > Leonard
> >
> >> 2023年12月13日 下午10:59，Benchao Li  写道：
> >>
> >> +1 (binding)
> >>
> >> Thanks Matthias for driving it!
> >>
> >> Etienne Chauchot  于2023年12月13日周三 21:35写道：
> >>>
> >>> Thanks Matthias for your hard work !
> >>>
> >>> +1 (binding)
> >>>
> >>> Best
> >>>
> >>> Etienne
> >>>
> >>> Le 12/12/2023 à 11:23, Lincoln Lee a écrit :
>  +1 (binding)
> 
>  Thanks for driving this!
> 
>  Best,
>  Lincoln Lee
> 
> 
>  Yun Tang  于2023年12月12日周二 17:52写道：
> 
> > Thanks for Matthias driving this work!
> >
> > +1 (binding)
> >
> > Best
> > Yun Tang
> > 
> > From: Yangze Guo
> > Sent: Tuesday, December 12, 2023 16:12
> > To:dev@flink.apache.org  
> > Subject: Re: [VOTE] FLIP-396: Trial to test GitHub Actions as an
> > alternative for Flink's current Azure CI infrastructure
> >
> > +1 (binding)
> >
> > Best,
> > Yangze Guo
> >
> > On Tue, Dec 12, 2023 at 3:51 PM Yuxin Tan
> wrote:
> >> +1 (non binding)
> >> Thanks for the effort.
> >>
> >> Best,
> >> Yuxin
> >>
> >>
> >> Samrat Deb  于2023年12月12日周二 15:25写道：
> >>
> >>> +1 (non binding)
> >>> Thanks for driving
> >>>
> >>> On Tue, 12 Dec 2023 at 11:59 AM, Sergey Nuyanzin<
> snuyan...@gmail.com>
> >>> wrote:
> >>>
>  +1 (binding)
> 
>  Thanks for driving this
> 
>  On Tue, Dec 12, 2023, 07:22 Rui Fan<1996fan...@gmail.com>  wrote:
> 
> > +1(binding)
> >
> > Best,
> > Rui
> >
> > On Tue, Dec 12, 2023 at 11:58 AM weijie guo <
> > guoweijieres...@gmail.com
> > wrote:
> >
> >> Thanks Matthias for this efforts.
> >>
> >> +1(binding)
> >>
> >>
> >> Best regards,
> >>
> >> Weijie
> >>
> >>
> >> Matthias Pohl  于2023年12月11日周一
> >>> 21:51写道：
> >>> Hi everyone,
> >>> I'd like to start a vote on FLIP-396 [1]. It covers enabling
> > GitHub
> >> Actions
> >>> (GHA) in Apache Flink. This means that GHA workflows will run
> > aside
> > from
> >>> the usual Azure CI workflows in a trial phase (which ends
> > earliest
>  with
> >> the
> >>> release of Flink 1.19). Azure CI will still serve as the
> > project's
> > ground
> >>> of truth until the community decides in a final vote to switch
> > to
> >>> GHA
> > or
> >>> stick to Azure CI.
> >>>
> >>> The related discussion thread can be found in [2].
> >>>
> >>> The vote will remain open for at least 72 hours and only
> > concluded
> >>> if
> >> there
> >>> are no objections and enough (i.e. at least 3) binding votes.
> >>>
> >>> Matthias
> >>>
> >>> [1]
> >>>
> >>>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-396%3A+Trial+to+test+GitHub+Actions+as+an+alternative+for+Flink%27s+current+Azure+CI+infrastructure
> >>> [2]
> >>> https://lists.apache.org/thread/h4cmv7l3y8mxx2t435dmq4ltco4sbrgb
> >>> --
> >>>
> >>> [image: Aiven]
> >>>
> >>> *Matthias Pohl*
> >>> Opensource Software Engineer, *Aiven*
> >>> matthias.p...@aiven.io  |  +49 170 9869525
> >>> aiven.io|   <
> >> https://www.facebook.com/aivencloud
> >>>  <
> https://twitter.com/aiven_io>
> >>> *Aiven Deutschland GmbH*
> >>> Alexanderufer 3-7, 10117 Berlin
> >>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >>> Amtsgericht Charlottenburg, HRB 209739 B
> >>>
> >>
> >>
> >>
> >> --
> >>
> >> Best,
> >> Benchao Li
>
>

Re:Re: [DISCUSS][2.0] FLIP-382: Unify the Provision of Diverse Metadata for Context-like APIs

[Wencong Liu]

Hi Lijie,

Thank you for the response and for supporting the proposal.

Regarding your suggestions:

1. While the getTaskNameWithIndexAndAttemptNumber may seem
redundant initially, this concatenated string, for example
"MyTask (3/6)#1", serves as an identifier used to denote
a specific Task across multiple components including
TaskExecutor, Task, ChangelogStateBackend, StreamTask, etc.
Hence, offering a dedicated method for this representation
is quite practical. WDYT?

2. Good point. I will indicate the replacement methods in the
note of deprecated methods to provide clearer guidance.

Please let me know if you have any further questions.

Best regards,
Wencong

















At 2023-12-12 14:57:53, "Lijie Wang"  wrote:
>Hi Wencong
>
>Thanks for driving the discussion, +1 for the proposal. I left two minor
>questions/suggestions:
>
>1. Is the getTaskNameWithIndexAndAttemptNumber method a bit redundant? It
>can be replaced by getTaskName + getTaskIndex + getAttemptNumber.
>2. I think it would be better if we can explicitly specify the alternative
>(based on TaskInfo/JobInfo) for each deprecated method
>
>Best,
>Lijie
>
>Wencong Liu  于2023年11月30日周四 14:50写道：
>
>> Hi devs,
>>
>> I would like to start a discussion on FLIP-382: Unify the Provision
>> of Diverse Metadata for Context-like APIs [1].
>>
>> In the Flink project, the context-like APIs are interfaces annotated by
>> @Public and supply runtime metadata and functionalities to its modules and
>> components. RuntimeContext is such an interface with 27 methods for
>> accessing metadata and framework functionalities. Currently, any
>> addition of metadata requires updating the RuntimeContext interface
>> and all 12 of its implementation classes, leading to high code
>> maintenance costs. To improve this, we propose to a categorize all
>> metadata into some metadata classes and provide it through dedicated
>> methods in context-like APIs. The newly provided metadata in context-like
>> API will only require to update the metadata classes, not context-like API
>> itself or it's implementations.
>>
>> Looking forward to your feedback.
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-382%3A+Unify+the+Provision+of+Diverse+Metadata+for+Context-like+APIs
>>
>> Best regards,
>> Wencong Liu

Re: [DISCUSS] Should Configuration support getting value based on String key?

[Xuannan Su]

Hi Rui,

We are currently revisiting all the configurations for Flink 2.0, and
it turns out that many string-based configurations in
`ConfigConstants` are deprecated and have been replaced by
`ConfigOptions`. Since `ConfigOptions` offers many advantages over
string-based configurations for the end user, I believe we should
encourage users to set and get the Flink configuration exclusively
with `ConfigOption`. And we are going to eventually replace all the
string-based configurations with `ConfigOptions` for this use case.

For the first use case you mentioned, I think they are all internal usage,
and we should aim to replace them with ConfigOptions gradually.
Meanwhile, we may consider making those getters/setters for internal
use only while the replacement is in progress.

For the second use case, IIUC, you need to iterate over all the
configurations to replace some old configuration keys with new ones. I
believe  `toMap` is suitable for this scenario.

Best,
Xuannan



On Wed, Dec 13, 2023 at 9:04 PM Rui Fan <1996fan...@gmail.com> wrote:
>
> Thanks Zhu for the quick response!
>
> > It is not a blocker of the deprecation, epsecially given that they are
> not standard
> > configuration and are just using Configuration class for convenience.
>
> Yes, it's not a blocker of deprecation.
>
> > These are internal usages and we can have an internal getter method for
> them.
>
> For case1, do you mean we reuse the old getString method as the internal
> getter method or add a new getter method?
>
> Anyway, it's fine for me if we have an internal getter method. As I
> understand,
> the public method without any annotation will be the internal method, right?
>
> > Not sure why it's required to convert all keys or values. If it is used
> > to create strings for dynamic properties or config files to deploy jobs,
> > toMap()/toFileWritableMap() may be a better choice. They are already
> > used in this kind of scenarios.
>
> For case2, it's really a special scenario, and toMap() is fine for case2.
> Case2 uses the getString instead of toMap due to getString is easier.
> Also, kubernetes-operator is also a internal usage, if case1 is solved,
> case2 also can use the internal getter method.So we can focus on case1.
>
> Thank you
>
> Best,
> Rui
>
> On Wed, Dec 13, 2023 at 8:01 PM Zhu Zhu  wrote:
>
> > Hi Rui,
> >
> > I'd like to understand why there is a strong requirement for these
> > deprecated
> > methods. The ConfigOption param methods help to do the type conversion so
> > that users do not need to do it by themselves.
> >
> > For the 2 reasons to keep these methods mentioned above:
> >
> > > 1. A lot of scenarios don't define the ConfigOption, they using
> > String as the key and value directly, such as: StreamConfig,
> > TaskConfig, DistributedCache, etc.
> >
> > These are internal usages and we can have an internal getter method for
> > them.
> > It is not a blocker of the deprecation, epsecially given that they are not
> > standard
> > configuration and are just using Configuration class for convenience.
> >
> > > 2. Some code wanna convert all keys or values, this converting
> > is generic, so the getString(String key, String defaultValue) is needed.
> > Such as: kubernetes-operator [3].
> >
> > Not sure why it's required to convert all keys or values. If it is used
> > to create strings for dynamic properties or config files to deploy jobs,
> > toMap()/toFileWritableMap() may be a better choice. They are already
> > used in this kind of scenarios.
> > If it just needs to read some of the config, why not using the proposed
> > way to read and parse the config? Pre-defined ConfigOptions are better
> > for configuration maintenance, compared to arbitrary strings
> >
> > Thanks,
> > Zhu
> >
> > Rui Fan <1996fan...@gmail.com> 于2023年12月13日周三 19:27写道：
> >
> >> Thanks Martijn for the quick clarification!
> >>
> >> I see Zhu Zhu and Junrui Lee are working on configuration related
> >> work of Flink-2.0. I would cc them, and hear some thoughts from them.
> >>
> >> Best,
> >> Rui
> >>
> >> On Wed, Dec 13, 2023 at 7:17 PM Martijn Visser 
> >> wrote:
> >>
> >>> Hi Rui,
> >>>
> >>> I'm more wondering if part of the configuration layer changes would
> >>> mean that these APIs would be removed in 2.0. Because if so, then I
> >>> don't think we should remove the Deprecate annotation. But I have very
> >>> little visibility on the plans for the configuration layer.
> >>>
> >>> Thanks,
> >>>
> >>> Martijn
> >>>
> >>> On Wed, Dec 13, 2023 at 12:15 PM Rui Fan <1996fan...@gmail.com> wrote:
> >>> >
> >>> > Hi Martijn,
> >>> >
> >>> > Thanks for your reply!
> >>> >
> >>> > I noticed the 2.0 is doing some work related to clean configuration.
> >>> > But this discussion is different from other work. Most of other work
> >>> > are deprecate some Apis in Flink-1.19 and remove them in Flink-2.0.
> >>> >
> >>> > This discussion is a series of methods have been marked to @Deprecate,
> >>> > but they are still used so far. I propose remove 

Re: [VOTE] FLIP-396: Trial to test GitHub Actions as an alternative for Flink's current Azure CI infrastructure

[Jiabao Sun]

Thanks Matthias for this hard work!

+1(non-binding)

Best,
Jiabao


> 2023年12月14日 09:57，Leonard Xu  写道：
> 
> +1(binding)
> 
> 
> Best,
> Leonard
> 
>> 2023年12月13日 下午10:59，Benchao Li  写道：
>> 
>> +1 (binding)
>> 
>> Thanks Matthias for driving it!
>> 
>> Etienne Chauchot  于2023年12月13日周三 21:35写道：
>>> 
>>> Thanks Matthias for your hard work !
>>> 
>>> +1 (binding)
>>> 
>>> Best
>>> 
>>> Etienne
>>> 
>>> Le 12/12/2023 à 11:23, Lincoln Lee a écrit :
 +1 (binding)
 
 Thanks for driving this!
 
 Best,
 Lincoln Lee
 
 
 Yun Tang  于2023年12月12日周二 17:52写道：
 
> Thanks for Matthias driving this work!
> 
> +1 (binding)
> 
> Best
> Yun Tang
> 
> From: Yangze Guo
> Sent: Tuesday, December 12, 2023 16:12
> To:dev@flink.apache.org  
> Subject: Re: [VOTE] FLIP-396: Trial to test GitHub Actions as an
> alternative for Flink's current Azure CI infrastructure
> 
> +1 (binding)
> 
> Best,
> Yangze Guo
> 
> On Tue, Dec 12, 2023 at 3:51 PM Yuxin Tan  wrote:
>> +1 (non binding)
>> Thanks for the effort.
>> 
>> Best,
>> Yuxin
>> 
>> 
>> Samrat Deb  于2023年12月12日周二 15:25写道：
>> 
>>> +1 (non binding)
>>> Thanks for driving
>>> 
>>> On Tue, 12 Dec 2023 at 11:59 AM, Sergey Nuyanzin
>>> wrote:
>>> 
 +1 (binding)
 
 Thanks for driving this
 
 On Tue, Dec 12, 2023, 07:22 Rui Fan<1996fan...@gmail.com>  wrote:
 
> +1(binding)
> 
> Best,
> Rui
> 
> On Tue, Dec 12, 2023 at 11:58 AM weijie guo <
> guoweijieres...@gmail.com
> wrote:
> 
>> Thanks Matthias for this efforts.
>> 
>> +1(binding)
>> 
>> 
>> Best regards,
>> 
>> Weijie
>> 
>> 
>> Matthias Pohl  于2023年12月11日周一
>>> 21:51写道：
>>> Hi everyone,
>>> I'd like to start a vote on FLIP-396 [1]. It covers enabling
> GitHub
>> Actions
>>> (GHA) in Apache Flink. This means that GHA workflows will run
> aside
> from
>>> the usual Azure CI workflows in a trial phase (which ends
> earliest
 with
>> the
>>> release of Flink 1.19). Azure CI will still serve as the
> project's
> ground
>>> of truth until the community decides in a final vote to switch
> to
>>> GHA
> or
>>> stick to Azure CI.
>>> 
>>> The related discussion thread can be found in [2].
>>> 
>>> The vote will remain open for at least 72 hours and only
> concluded
>>> if
>> there
>>> are no objections and enough (i.e. at least 3) binding votes.
>>> 
>>> Matthias
>>> 
>>> [1]
>>> 
>>> 
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-396%3A+Trial+to+test+GitHub+Actions+as+an+alternative+for+Flink%27s+current+Azure+CI+infrastructure
>>> [2]
>>> https://lists.apache.org/thread/h4cmv7l3y8mxx2t435dmq4ltco4sbrgb
>>> --
>>> 
>>> [image: Aiven]
>>> 
>>> *Matthias Pohl*
>>> Opensource Software Engineer, *Aiven*
>>> matthias.p...@aiven.io  |  +49 170 9869525
>>> aiven.io|   <
>> https://www.facebook.com/aivencloud
>>>  < 
>>> https://twitter.com/aiven_io>
>>> *Aiven Deutschland GmbH*
>>> Alexanderufer 3-7, 10117 Berlin
>>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>>> Amtsgericht Charlottenburg, HRB 209739 B
>>> 
>> 
>> 
>> 
>> -- 
>> 
>> Best,
>> Benchao Li

Re: [VOTE] FLIP-396: Trial to test GitHub Actions as an alternative for Flink's current Azure CI infrastructure

[Leonard Xu]

+1(binding)


Best,
Leonard

> 2023年12月13日 下午10:59，Benchao Li  写道：
> 
> +1 (binding)
> 
> Thanks Matthias for driving it!
> 
> Etienne Chauchot  于2023年12月13日周三 21:35写道：
>> 
>> Thanks Matthias for your hard work !
>> 
>> +1 (binding)
>> 
>> Best
>> 
>> Etienne
>> 
>> Le 12/12/2023 à 11:23, Lincoln Lee a écrit :
>>> +1 (binding)
>>> 
>>> Thanks for driving this!
>>> 
>>> Best,
>>> Lincoln Lee
>>> 
>>> 
>>> Yun Tang  于2023年12月12日周二 17:52写道：
>>> 
 Thanks for Matthias driving this work!
 
 +1 (binding)
 
 Best
 Yun Tang
 
 From: Yangze Guo
 Sent: Tuesday, December 12, 2023 16:12
 To:dev@flink.apache.org  
 Subject: Re: [VOTE] FLIP-396: Trial to test GitHub Actions as an
 alternative for Flink's current Azure CI infrastructure
 
 +1 (binding)
 
 Best,
 Yangze Guo
 
 On Tue, Dec 12, 2023 at 3:51 PM Yuxin Tan  wrote:
> +1 (non binding)
> Thanks for the effort.
> 
> Best,
> Yuxin
> 
> 
> Samrat Deb  于2023年12月12日周二 15:25写道：
> 
>> +1 (non binding)
>> Thanks for driving
>> 
>> On Tue, 12 Dec 2023 at 11:59 AM, Sergey Nuyanzin
>> wrote:
>> 
>>> +1 (binding)
>>> 
>>> Thanks for driving this
>>> 
>>> On Tue, Dec 12, 2023, 07:22 Rui Fan<1996fan...@gmail.com>  wrote:
>>> 
 +1(binding)
 
 Best,
 Rui
 
 On Tue, Dec 12, 2023 at 11:58 AM weijie guo <
 guoweijieres...@gmail.com
 wrote:
 
> Thanks Matthias for this efforts.
> 
> +1(binding)
> 
> 
> Best regards,
> 
> Weijie
> 
> 
> Matthias Pohl  于2023年12月11日周一
>> 21:51写道：
>> Hi everyone,
>> I'd like to start a vote on FLIP-396 [1]. It covers enabling
 GitHub
> Actions
>> (GHA) in Apache Flink. This means that GHA workflows will run
 aside
 from
>> the usual Azure CI workflows in a trial phase (which ends
 earliest
>>> with
> the
>> release of Flink 1.19). Azure CI will still serve as the
 project's
 ground
>> of truth until the community decides in a final vote to switch
 to
>> GHA
 or
>> stick to Azure CI.
>> 
>> The related discussion thread can be found in [2].
>> 
>> The vote will remain open for at least 72 hours and only
 concluded
>> if
> there
>> are no objections and enough (i.e. at least 3) binding votes.
>> 
>> Matthias
>> 
>> [1]
>> 
>> 
 https://cwiki.apache.org/confluence/display/FLINK/FLIP-396%3A+Trial+to+test+GitHub+Actions+as+an+alternative+for+Flink%27s+current+Azure+CI+infrastructure
>> [2]
>> https://lists.apache.org/thread/h4cmv7l3y8mxx2t435dmq4ltco4sbrgb
>> --
>> 
>> [image: Aiven]
>> 
>> *Matthias Pohl*
>> Opensource Software Engineer, *Aiven*
>> matthias.p...@aiven.io  |  +49 170 9869525
>> aiven.io|   <
> https://www.facebook.com/aivencloud
>>   < 
>> https://twitter.com/aiven_io>
>> *Aiven Deutschland GmbH*
>> Alexanderufer 3-7, 10117 Berlin
>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> Amtsgericht Charlottenburg, HRB 209739 B
>> 
> 
> 
> 
> -- 
> 
> Best,
> Benchao Li

Re: [VOTE] Release flink-connector-pulsar 4.1.0, release candidate #1

[Rui Fan]

Thanks Leonard for driving this release!

+1 (non-binding)

- Validated checksum hash
- Verified signature
- Verified that no binaries exist in the source archive
- Build the source with Maven and jdk8
- Verified licenses
- Verified web PRs, left a minor comment

Best,
Rui

On Wed, Dec 13, 2023 at 7:15 PM Leonard Xu  wrote:

> Hey all,
>
> Please review and vote on the release candidate #1 for the version 4.1.0
> of the Apache Flink Pulsar Connector as follows:
>
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * The official Apache source release to be deployed to dist.apache.org
> [2], which are signed with the key with fingerprint
> 5B2F6608732389AEB67331F5B197E1F1108998AD [3],
> * All artifacts to be deployed to the Maven Central Repository [4],
> * Source code tag v4.1.0-rc1 [5],
> * Website pull request listing the new release [6].
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
>
> Best,
> Leonard
>
> [1]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353431
> [2]
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-pulsar-4.1.0-rc1/
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4]
> https://repository.apache.org/content/repositories/orgapacheflink-1688/
> [5] https://github.com/apache/flink-connector-pulsar/tree/v4.1.0-rc1
> [6] https://github.com/apache/flink-web/pull/703

[jira] [Created] (FLINK-33818) Implement restore tests for WindowDeduplicate node

[Bonnie Varghese (Jira)]

Bonnie Varghese created FLINK-33818:
---

 Summary: Implement restore tests for WindowDeduplicate node
 Key: FLINK-33818
 URL: https://issues.apache.org/jira/browse/FLINK-33818
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Bonnie Varghese
Assignee: Bonnie Varghese






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33817) Allow ReadDefaultValues = False for non primitive types on Proto3

[Sai Sharath Dandi (Jira)]

Sai Sharath Dandi created FLINK-33817:
-

 Summary: Allow ReadDefaultValues = False for non primitive types 
on Proto3
 Key: FLINK-33817
 URL: https://issues.apache.org/jira/browse/FLINK-33817
 Project: Flink
  Issue Type: Improvement
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.18.0
Reporter: Sai Sharath Dandi


*Background*

 

The current Protobuf format 
[implementation|https://github.com/apache/flink/blob/c3e2d163a637dca5f49522721109161bd7ebb723/flink-formats/flink-protobuf/src/main/java/org/apache/flink/formats/protobuf/deserialize/ProtoToRowConverter.java]
 always sets ReadDefaultValues=False when using Proto3 version. This can cause 
severe performance degradation for large Protobuf schemas with OneOf fields as 
the entire generated code needs to be executed during deserialization even when 
certain fields are not present in the data to be deserialized and all the 
subsequent nested Fields can be skipped. Proto3 supports hasXXX() methods for 
checking field presence for non primitive types since Proto version 
[3.15|https://github.com/protocolbuffers/protobuf/releases/tag/v3.15.0]. In 
internal benchmarks in our company, we've seen almost 10x difference in 
performance when allowing to set ReadDefaultValues=False with proto3 version

 

*Solution*

 

Support using ReadDefaultValues=False when using Proto3 version. We need to be 
careful to check for field presence only on non-primitive types if 
ReadDefaultValues is false and version used is Proto3



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Edit permissions request

[Alex Nitavsky]

Hello,

I would love to initiate FLIP in order to extend Apache Curator
configuration: https://issues.apache.org/jira/browse/FLINK-33376.

Would it be please possible to get edit permissions for Apache Flink
confluence space please?

Thanks

Re: [DISCUSS] Allow TwoPhaseCommittingSink WithPreCommitTopology to alter the type of the Committable

[Gyula Fóra]

Thank you Peter Vary for updating the FLIP based on the discussions.
I really like the improvements introduced by the mixin interfaces which now
aligns much better with the source and table connectors.

While this introduces some breaking changes to the existing connectors,
this is a technical debt that we need to resolve as soon as possible and
fully before 2.0.

+1 from my side.

I am cc'ing some folks participating in the other threads, sorry about that
:)

Cheers,
Gyula

On Wed, Dec 13, 2023 at 4:14 PM Péter Váry 
wrote:

> I have updated the FLIP-372 [1] based on the comments from multiple
> sources. Moved to the mixin approach as this seems to be the consensus
> based on this thread [2]
> Also created a draft implementation [3] PR, so I can test the changes and
> default implementations (no new tests yet)
> Please provide your feedback, so I can address your questions, comments and
> then we can move forward to voting.
>
> Thanks,
> Peter
>
> [1] -
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-372%3A+Allow+TwoPhaseCommittingSink+WithPreCommitTopology+to+alter+the+type+of+the+Committable
> [2] - https://lists.apache.org/thread/h6nkgth838dlh5s90sd95zd6hlsxwx57
> [3] - https://github.com/apache/flink/pull/23912
>
> Péter Váry  ezt írta (időpont: 2023. dec.
> 11.,
> H, 14:28):
>
> > We identified another issue with the current Sink API in a thread [1]
> > related to FLIP-371 [2]. Currently it is not possible to evolve the
> > Sink.createWriter method with deprecation, because StatefulSink and
> > TwoPhaseCommittingSink has methods with the same name and parameters, but
> > narrowed down return type (StatefulSinkWriter, PrecommittingSinkWriter).
> >
> > To make the Sink API evolvable, we minimally have to remove these.
> >
> > The participants there also pointed out, that the Source API also uses
> > mixin interfaces (SupportsHandleExecutionAttemptSourceEvent,
> > SupportsIntermediateNoMoreSplits) in some cases. My observation is that
> it
> > has inheritance as well (ReaderOutput, ExternallyInducedSourceReader)
> >
> > I have created a draft API along these lines in a branch [3] where only
> > the last commit is relevant [4]. This implementation would follow the
> same
> > patterns as the current Source API.
> >
> > I see two different general approaches here, and I would like to hear
> your
> > preferences:
> > - Keep the changes minimal, stick to the current Sink API design. We
> > introduce the required new combination of interfaces
> > (TwoPhaseCommttingSinkWithPreCommitTopology,
> > WithPostCommitTopologyWithPreCommitTopology), and do not change the API
> > structure.
> >  - Pros:
> >   - Minimal change - smaller rewrite on the connector side
> >   - Type checks happen on compile time
> >  - Cons:
> >   - Harder to evolve
> >   - The number of interfaces increases with the possible
> > combinations
> >   - Inconsistent API patterns wrt Source API - harder for
> > developers to understand
> > - Migrate to a model similar to the Source API. We create mixin
> interfaces
> > for SupportsCommitter, SupportsWriterState, SupportsPreCommitTopology,
> > SupportsPostCommitTopology, SupportsPreWriteTopology.
> > - Pros:
> > - Better evolvability
> > - Consistent with the Source API
> > - Cons:
> > - The connectors need to change their inheritance patterns (after
> > the deprecation period) if they are using any of the more complicated
> Sinks.
> > - Type checks need to use `instanceof`, which could happen on DAG
> > generation time. Also, if the developer fails to correctly match the
> > generic types on the mixin interfaces, the error will only surface during
> > execution time - when the job tries to process the first record
> >
> > I personally prefer the Mixin approach for easier evolvability and better
> > consistency, but I would like to hear your thoughts, so I can flash out
> the
> > chosen approach in FLIP-372
> >
> > Thanks,
> > Peter
> >
> > [1] - https://lists.apache.org/thread/h6nkgth838dlh5s90sd95zd6hlsxwx57
> > [2] -
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink
> > [3] - https://github.com/pvary/flink/tree/mixin_demo
> > [4] -
> >
> https://github.com/pvary/flink/commit/acfc09f4c846f983f633bbf0c902aea49aa6b156
> >
> >
> > On Fri, Nov 24, 2023, 11:38 Gyula Fóra  wrote:
> >
> >> Hi Peter!
> >>
> >> Thank you for the analysis of the options.
> >>
> >> I don't really have a strong opinion, but in general I am in favor of
> the
> >> mix in style interfaces.
> >> We follow the same approach for table sources / sinks as well.
> >>
> >> Some other benefits I see in addition to what you mentioned:
> >>  - Easier to introduce new experimental / public-evolving interfaces in
> >> the
> >> future
> >>  - Easier to declare parts of the api stable going forward as it's not
> all
> >> or nothing
> >>
> >> The 

Re: [DISCUSS] Allow TwoPhaseCommittingSink WithPreCommitTopology to alter the type of the Committable

[Péter Váry]

I have updated the FLIP-372 [1] based on the comments from multiple
sources. Moved to the mixin approach as this seems to be the consensus
based on this thread [2]
Also created a draft implementation [3] PR, so I can test the changes and
default implementations (no new tests yet)
Please provide your feedback, so I can address your questions, comments and
then we can move forward to voting.

Thanks,
Peter

[1] -
https://cwiki.apache.org/confluence/display/FLINK/FLIP-372%3A+Allow+TwoPhaseCommittingSink+WithPreCommitTopology+to+alter+the+type+of+the+Committable
[2] - https://lists.apache.org/thread/h6nkgth838dlh5s90sd95zd6hlsxwx57
[3] - https://github.com/apache/flink/pull/23912

Péter Váry  ezt írta (időpont: 2023. dec. 11.,
H, 14:28):

> We identified another issue with the current Sink API in a thread [1]
> related to FLIP-371 [2]. Currently it is not possible to evolve the
> Sink.createWriter method with deprecation, because StatefulSink and
> TwoPhaseCommittingSink has methods with the same name and parameters, but
> narrowed down return type (StatefulSinkWriter, PrecommittingSinkWriter).
>
> To make the Sink API evolvable, we minimally have to remove these.
>
> The participants there also pointed out, that the Source API also uses
> mixin interfaces (SupportsHandleExecutionAttemptSourceEvent,
> SupportsIntermediateNoMoreSplits) in some cases. My observation is that it
> has inheritance as well (ReaderOutput, ExternallyInducedSourceReader)
>
> I have created a draft API along these lines in a branch [3] where only
> the last commit is relevant [4]. This implementation would follow the same
> patterns as the current Source API.
>
> I see two different general approaches here, and I would like to hear your
> preferences:
> - Keep the changes minimal, stick to the current Sink API design. We
> introduce the required new combination of interfaces
> (TwoPhaseCommttingSinkWithPreCommitTopology,
> WithPostCommitTopologyWithPreCommitTopology), and do not change the API
> structure.
>  - Pros:
>   - Minimal change - smaller rewrite on the connector side
>   - Type checks happen on compile time
>  - Cons:
>   - Harder to evolve
>   - The number of interfaces increases with the possible
> combinations
>   - Inconsistent API patterns wrt Source API - harder for
> developers to understand
> - Migrate to a model similar to the Source API. We create mixin interfaces
> for SupportsCommitter, SupportsWriterState, SupportsPreCommitTopology,
> SupportsPostCommitTopology, SupportsPreWriteTopology.
> - Pros:
> - Better evolvability
> - Consistent with the Source API
> - Cons:
> - The connectors need to change their inheritance patterns (after
> the deprecation period) if they are using any of the more complicated Sinks.
> - Type checks need to use `instanceof`, which could happen on DAG
> generation time. Also, if the developer fails to correctly match the
> generic types on the mixin interfaces, the error will only surface during
> execution time - when the job tries to process the first record
>
> I personally prefer the Mixin approach for easier evolvability and better
> consistency, but I would like to hear your thoughts, so I can flash out the
> chosen approach in FLIP-372
>
> Thanks,
> Peter
>
> [1] - https://lists.apache.org/thread/h6nkgth838dlh5s90sd95zd6hlsxwx57
> [2] -
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink
> [3] - https://github.com/pvary/flink/tree/mixin_demo
> [4] -
> https://github.com/pvary/flink/commit/acfc09f4c846f983f633bbf0c902aea49aa6b156
>
>
> On Fri, Nov 24, 2023, 11:38 Gyula Fóra  wrote:
>
>> Hi Peter!
>>
>> Thank you for the analysis of the options.
>>
>> I don't really have a strong opinion, but in general I am in favor of the
>> mix in style interfaces.
>> We follow the same approach for table sources / sinks as well.
>>
>> Some other benefits I see in addition to what you mentioned:
>>  - Easier to introduce new experimental / public-evolving interfaces in
>> the
>> future
>>  - Easier to declare parts of the api stable going forward as it's not all
>> or nothing
>>
>> The ability to do proper compile time validation is definitely a downside
>> but this should mostly make initial development a little harder I believe.
>>
>> Cheers,
>> Gyula
>>
>> On Thu, Nov 23, 2023 at 1:25 PM Péter Váry 
>> wrote:
>>
>> > We had a longer discussion with Gordon yesterday.
>> > The main conclusion was that moving to a completely new interface is not
>> > justified, and we try to improve the current one.
>> >
>> > Another ask from Gordon was to check when the user will be notified if
>> the
>> > parameter types are incorrect using the mixin approach.
>> > Imagine the type definition below:
>> >
>> > private static class
>> > TestTwoPhaseCommittingSinkWithPreCommitTopologyWrongMixin
>> > implements
>> > 

Re: [DISCUSS] Resolve diamond inheritance of Sink.createWriter

[Péter Váry]

The PR is merged, so everything should be back to normal.
Thanks everyone involved!

Also based on the discussion here, I have updated the FLIP-372 [1]. I tried
to incorporate every comment, but in some cases they were contradictory, so
please share your opinion on the corresponding thread [2].

Thanks again,
Peter

[1] -
https://cwiki.apache.org/confluence/display/FLINK/FLIP-372%3A+Allow+TwoPhaseCommittingSink+WithPreCommitTopology+to+alter+the+type+of+the+Committable
[2] - https://lists.apache.org/thread/344pzbrqbbb4w0sfj67km25msp7hxlyd

Leonard Xu  ezt írta (időpont: 2023. dec. 12., K, 7:33):

> Thanks Peter and Marton for the quick response and fix, I’ve left +1 for
> this PR.
>
> I’d like to take a look at followup FLIP-372 as well.
>
> Best,
> Leonard
>
>
> > 2023年12月12日 上午7:39，Becket Qin  写道：
> >
> > Hi Peter,
> >
> > Thanks for updating the patch. The latest patch looks good to me. I've
> +1ed
> > on the PR.
> >
> > Cheers,
> >
> > Jiangjie (Becket) Qin
> >
> > On Mon, Dec 11, 2023 at 9:21 PM Péter Váry 
> > wrote:
> >
> >> Thanks everyone for the lively discussion!
> >>
> >> The PR is available which strictly adheres the accepted changes from
> >> FLIP-371. Thanks Gyula and Marton for the review. Becket, if you have
> any
> >> questions left, please let me know, so I can fix and we can merge the
> >> changes.
> >>
> >> I would like to invite everyone involved here to take a look at FLIP-372
> >> [1], and the related mailing thread [2]. The discussion there is also at
> >> the stage where we are debating the merits of migrating to a mixin based
> >> Sink API. So if you are interested, please join us there.
> >>
> >> Thanks,
> >> Peter
> >>
> >> [1] -
> >>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-372%3A+Allow+TwoPhaseCommittingSink+WithPreCommitTopology+to+alter+the+type+of+the+Committable
> >> [2] - https://lists.apache.org/thread/344pzbrqbbb4w0sfj67km25msp7hxlyd
> >>
> >>
> >> On Tue, Dec 5, 2023, 18:05 Márton Balassi 
> >> wrote:
> >>
> >>> Thanks, Peter. Given the discussion I also agree that the consensus is
> to
> >>> move towards the mixin interface approach (and accept its disadvantages
> >>> given its advantages).
> >>>
> >>> +1 for the general direction of your proposed code change in
> >>> https://github.com/apache/flink/pull/23876.
> >>>
> >>> On Tue, Dec 5, 2023 at 3:44 PM Péter Váry  >
> >>> wrote:
> >>>
>  It seems to me we have a consensus to move forward with the mixin
> >>> approach.
>  I hope that everyone is aware that with the mixin interfaces we lose
> >> the
>  opportunity of the strong type checks. This will be especially painful
> >>> for
>  generic types, where we will not have a way to ensure that the generic
>  types are correctly synchronized between the different interfaces,
> even
> >>> on
>  DAG creation time.
> 
>  Even with this drawback, I like this approach too, so +1 from my side.
> 
>  As a first step in the direction of the mixin approach, we can remove
> >> the
>  specific implementations of the `createWriter` methods from the
>  `StatefulSink` and the `TwoPhaseCommitingSink` interfaces (and replace
> >>> them
>  with an instanceof check where needed).
>  - This would remove the diamond inheritance - enable us to create
> >> default
>  methods for backward compatibility.
>  - This would not break the API, as the same method with wider return
> >>> value
>  will be inherited from the `Sink` interface.
> 
>  Since, it might be easier to understand the proposed changes, I have
>  created a new PR: https://github.com/apache/flink/pull/23876
>  The PR has 2 commits:
>  - Reverting the previous change - non-clean, since there were some
>  additional fixes on the tests -
> 
> 
> >>>
> >>
> https://github.com/apache/flink/pull/23876/commits/c7625d5fa62a6e9a182f39f53fb7e5626105f3b0
>  - The new change with mixin approach, and deprecation -
> 
> 
> >>>
> >>
> https://github.com/apache/flink/pull/23876/commits/99ec936966af527598ca49712c1263bc4aa03c15
> 
>  Thanks,
>  Peter
> 
>  weijie guo  ezt írta (időpont: 2023. dec.
> >> 5.,
>  K,
>  8:01):
> 
> > Thanks Martijn for driving this!
> >
> > I'm +1  to reverting the breaking change.
> >
> >> For new functionality or changes we can make easily, we should
> >> switch
>  to
> > the decorative/mixin interface approach used successfully in the
> >> source
>  and
> > table interfaces.
> >
> > I like the way of switching to mixin interface.
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > Becket Qin  于2023年12月5日周二 14:50写道：
> >
> >> I am with Gyula about fixing the current SinkV2 API.
> >>
> >> A SinkV3 seems not necessary because we are not changing the
>  fundamental
> >> design of the API. Hopefully we can modify the interface structure
> >> a
> > little
> >> 

Re: [VOTE] FLIP-396: Trial to test GitHub Actions as an alternative for Flink's current Azure CI infrastructure

[Benchao Li]

+1 (binding)

Thanks Matthias for driving it!

Etienne Chauchot  于2023年12月13日周三 21:35写道：
>
> Thanks Matthias for your hard work !
>
> +1 (binding)
>
> Best
>
> Etienne
>
> Le 12/12/2023 à 11:23, Lincoln Lee a écrit :
> > +1 (binding)
> >
> > Thanks for driving this!
> >
> > Best,
> > Lincoln Lee
> >
> >
> > Yun Tang  于2023年12月12日周二 17:52写道：
> >
> >> Thanks for Matthias driving this work!
> >>
> >> +1 (binding)
> >>
> >> Best
> >> Yun Tang
> >> 
> >> From: Yangze Guo
> >> Sent: Tuesday, December 12, 2023 16:12
> >> To:dev@flink.apache.org  
> >> Subject: Re: [VOTE] FLIP-396: Trial to test GitHub Actions as an
> >> alternative for Flink's current Azure CI infrastructure
> >>
> >> +1 (binding)
> >>
> >> Best,
> >> Yangze Guo
> >>
> >> On Tue, Dec 12, 2023 at 3:51 PM Yuxin Tan  wrote:
> >>> +1 (non binding)
> >>> Thanks for the effort.
> >>>
> >>> Best,
> >>> Yuxin
> >>>
> >>>
> >>> Samrat Deb  于2023年12月12日周二 15:25写道：
> >>>
>  +1 (non binding)
>  Thanks for driving
> 
>  On Tue, 12 Dec 2023 at 11:59 AM, Sergey Nuyanzin
>  wrote:
> 
> > +1 (binding)
> >
> > Thanks for driving this
> >
> > On Tue, Dec 12, 2023, 07:22 Rui Fan<1996fan...@gmail.com>  wrote:
> >
> >> +1(binding)
> >>
> >> Best,
> >> Rui
> >>
> >> On Tue, Dec 12, 2023 at 11:58 AM weijie guo <
> >> guoweijieres...@gmail.com
> >> wrote:
> >>
> >>> Thanks Matthias for this efforts.
> >>>
> >>> +1(binding)
> >>>
> >>>
> >>> Best regards,
> >>>
> >>> Weijie
> >>>
> >>>
> >>> Matthias Pohl  于2023年12月11日周一
>  21:51写道：
>  Hi everyone,
>  I'd like to start a vote on FLIP-396 [1]. It covers enabling
> >> GitHub
> >>> Actions
>  (GHA) in Apache Flink. This means that GHA workflows will run
> >> aside
> >> from
>  the usual Azure CI workflows in a trial phase (which ends
> >> earliest
> > with
> >>> the
>  release of Flink 1.19). Azure CI will still serve as the
> >> project's
> >> ground
>  of truth until the community decides in a final vote to switch
> >> to
>  GHA
> >> or
>  stick to Azure CI.
> 
>  The related discussion thread can be found in [2].
> 
>  The vote will remain open for at least 72 hours and only
> >> concluded
>  if
> >>> there
>  are no objections and enough (i.e. at least 3) binding votes.
> 
>  Matthias
> 
>  [1]
> 
> 
> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-396%3A+Trial+to+test+GitHub+Actions+as+an+alternative+for+Flink%27s+current+Azure+CI+infrastructure
>  [2]
>  https://lists.apache.org/thread/h4cmv7l3y8mxx2t435dmq4ltco4sbrgb
>  --
> 
>  [image: Aiven]
> 
>  *Matthias Pohl*
>  Opensource Software Engineer, *Aiven*
>  matthias.p...@aiven.io  |  +49 170 9869525
>  aiven.io|   <
> >>> https://www.facebook.com/aivencloud
> < 
>  https://twitter.com/aiven_io>
>  *Aiven Deutschland GmbH*
>  Alexanderufer 3-7, 10117 Berlin
>  Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>  Amtsgericht Charlottenburg, HRB 209739 B
> 



-- 

Best,
Benchao Li

Re: [VOTE] FLIP-401: REST API JSON response deserialization unknown field tolerance

[Márton Balassi]

+1 (binding)

On Tue, Dec 12, 2023 at 4:16 PM Rodrigo Meneses  wrote:

> +1
>
> On Tue, Dec 12, 2023 at 6:58 AM Maximilian Michels  wrote:
>
> > +1 (binding)
> >
> > On Tue, Dec 12, 2023 at 2:23 PM Peter Huang 
> > wrote:
> > >
> > > +1 Non-binding
> > >
> > >
> > > Peter Huang
> > >
> > > Őrhidi Mátyás 于2023年12月12日 周二下午9:14写道：
> > >
> > > > +1
> > > > Matyas
> > > >
> > > > On Mon, Dec 11, 2023 at 10:26 PM Gyula Fóra 
> > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > Gyula
> > > > >
> > > > > On Mon, Dec 11, 2023 at 1:26 PM Gabor Somogyi <
> > gabor.g.somo...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > I'd like to start a vote on FLIP-401: REST API JSON response
> > > > > > deserialization unknown field tolerance [1] which has been
> > discussed in
> > > > > > this thread [2].
> > > > > >
> > > > > > The vote will be open for at least 72 hours unless there is an
> > > > objection
> > > > > or
> > > > > > not enough votes.
> > > > > >
> > > > > > BR,
> > > > > > G
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-401%3A+REST+API+JSON+response+deserialization+unknown+field+tolerance
> > > > > > [2]
> > https://lists.apache.org/thread/s52w9cf60d6s10bpzv9qjczpl6m394rz
> > > > > >
> > > > >
> > > >
> >
>

Re: [VOTE] FLIP-396: Trial to test GitHub Actions as an alternative for Flink's current Azure CI infrastructure

[Etienne Chauchot]

Thanks Matthias for your hard work !

+1 (binding)

Best

Etienne

Le 12/12/2023 à 11:23, Lincoln Lee a écrit :

+1 (binding)

Thanks for driving this!

Best,
Lincoln Lee


Yun Tang  于2023年12月12日周二 17:52写道：


Thanks for Matthias driving this work!

+1 (binding)

Best
Yun Tang

From: Yangze Guo
Sent: Tuesday, December 12, 2023 16:12
To:dev@flink.apache.org  
Subject: Re: [VOTE] FLIP-396: Trial to test GitHub Actions as an
alternative for Flink's current Azure CI infrastructure

+1 (binding)

Best,
Yangze Guo

On Tue, Dec 12, 2023 at 3:51 PM Yuxin Tan  wrote:

+1 (non binding)
Thanks for the effort.

Best,
Yuxin


Samrat Deb  于2023年12月12日周二 15:25写道：


+1 (non binding)
Thanks for driving

On Tue, 12 Dec 2023 at 11:59 AM, Sergey Nuyanzin
wrote:


+1 (binding)

Thanks for driving this

On Tue, Dec 12, 2023, 07:22 Rui Fan<1996fan...@gmail.com>  wrote:


+1(binding)

Best,
Rui

On Tue, Dec 12, 2023 at 11:58 AM weijie guo <

guoweijieres...@gmail.com

wrote:


Thanks Matthias for this efforts.

+1(binding)


Best regards,

Weijie


Matthias Pohl  于2023年12月11日周一

21:51写道：

Hi everyone,
I'd like to start a vote on FLIP-396 [1]. It covers enabling

GitHub

Actions

(GHA) in Apache Flink. This means that GHA workflows will run

aside

from

the usual Azure CI workflows in a trial phase (which ends

earliest

with

the

release of Flink 1.19). Azure CI will still serve as the

project's

ground

of truth until the community decides in a final vote to switch

to

GHA

or

stick to Azure CI.

The related discussion thread can be found in [2].

The vote will remain open for at least 72 hours and only

concluded

if

there

are no objections and enough (i.e. at least 3) binding votes.

Matthias

[1]



https://cwiki.apache.org/confluence/display/FLINK/FLIP-396%3A+Trial+to+test+GitHub+Actions+as+an+alternative+for+Flink%27s+current+Azure+CI+infrastructure

[2]

https://lists.apache.org/thread/h4cmv7l3y8mxx2t435dmq4ltco4sbrgb

--

[image: Aiven]

*Matthias Pohl*
Opensource Software Engineer, *Aiven*
matthias.p...@aiven.io  |  +49 170 9869525
aiven.io|   <

https://www.facebook.com/aivencloud

   < https://twitter.com/aiven_io>
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

Re: [DISCUSS] Should Configuration support getting value based on String key?

[Rui Fan]

Thanks Zhu for the quick response!

> It is not a blocker of the deprecation, epsecially given that they are
not standard
> configuration and are just using Configuration class for convenience.

Yes, it's not a blocker of deprecation.

> These are internal usages and we can have an internal getter method for
them.

For case1, do you mean we reuse the old getString method as the internal
getter method or add a new getter method?

Anyway, it's fine for me if we have an internal getter method. As I
understand,
the public method without any annotation will be the internal method, right?

> Not sure why it's required to convert all keys or values. If it is used
> to create strings for dynamic properties or config files to deploy jobs,
> toMap()/toFileWritableMap() may be a better choice. They are already
> used in this kind of scenarios.

For case2, it's really a special scenario, and toMap() is fine for case2.
Case2 uses the getString instead of toMap due to getString is easier.
Also, kubernetes-operator is also a internal usage, if case1 is solved,
case2 also can use the internal getter method.So we can focus on case1.

Thank you

Best,
Rui

On Wed, Dec 13, 2023 at 8:01 PM Zhu Zhu  wrote:

> Hi Rui,
>
> I'd like to understand why there is a strong requirement for these
> deprecated
> methods. The ConfigOption param methods help to do the type conversion so
> that users do not need to do it by themselves.
>
> For the 2 reasons to keep these methods mentioned above:
>
> > 1. A lot of scenarios don't define the ConfigOption, they using
> String as the key and value directly, such as: StreamConfig,
> TaskConfig, DistributedCache, etc.
>
> These are internal usages and we can have an internal getter method for
> them.
> It is not a blocker of the deprecation, epsecially given that they are not
> standard
> configuration and are just using Configuration class for convenience.
>
> > 2. Some code wanna convert all keys or values, this converting
> is generic, so the getString(String key, String defaultValue) is needed.
> Such as: kubernetes-operator [3].
>
> Not sure why it's required to convert all keys or values. If it is used
> to create strings for dynamic properties or config files to deploy jobs,
> toMap()/toFileWritableMap() may be a better choice. They are already
> used in this kind of scenarios.
> If it just needs to read some of the config, why not using the proposed
> way to read and parse the config? Pre-defined ConfigOptions are better
> for configuration maintenance, compared to arbitrary strings
>
> Thanks,
> Zhu
>
> Rui Fan <1996fan...@gmail.com> 于2023年12月13日周三 19:27写道：
>
>> Thanks Martijn for the quick clarification!
>>
>> I see Zhu Zhu and Junrui Lee are working on configuration related
>> work of Flink-2.0. I would cc them, and hear some thoughts from them.
>>
>> Best,
>> Rui
>>
>> On Wed, Dec 13, 2023 at 7:17 PM Martijn Visser 
>> wrote:
>>
>>> Hi Rui,
>>>
>>> I'm more wondering if part of the configuration layer changes would
>>> mean that these APIs would be removed in 2.0. Because if so, then I
>>> don't think we should remove the Deprecate annotation. But I have very
>>> little visibility on the plans for the configuration layer.
>>>
>>> Thanks,
>>>
>>> Martijn
>>>
>>> On Wed, Dec 13, 2023 at 12:15 PM Rui Fan <1996fan...@gmail.com> wrote:
>>> >
>>> > Hi Martijn,
>>> >
>>> > Thanks for your reply!
>>> >
>>> > I noticed the 2.0 is doing some work related to clean configuration.
>>> > But this discussion is different from other work. Most of other work
>>> > are deprecate some Apis in Flink-1.19 and remove them in Flink-2.0.
>>> >
>>> > This discussion is a series of methods have been marked to @Deprecate,
>>> > but they are still used so far. I propose remove the @Deprecate
>>> annotation
>>> > and keep these methods.
>>> >
>>> > In brief, I think some deprecated methods shouldn't be deprecated.
>>> > WDYT?
>>> >
>>> > Please correct me if I'm wrong, thanks~
>>> >
>>> > Best,
>>> > Rui
>>> >
>>> > On Wed, Dec 13, 2023 at 7:07 PM Martijn Visser <
>>> martijnvis...@apache.org>
>>> > wrote:
>>> >
>>> > > Hi Rui,
>>> > >
>>> > > Are you thinking about this as part of Flink 2.0, since that has the
>>> > > goal to introduce a completely clean configuration layer? [1]
>>> > >
>>> > > Best regards,
>>> > >
>>> > > Martijn
>>> > >
>>> > > [1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release
>>> > >
>>> > > On Wed, Dec 13, 2023 at 11:28 AM Maximilian Michels 
>>> > > wrote:
>>> > > >
>>> > > > Hi Rui,
>>> > > >
>>> > > > +1 for removing the @Deprecated annotation from `getString(String
>>> key,
>>> > > > String defaultValue)`. I would remove the other typed variants with
>>> > > > default values but I'm ok with keeping them if they are still used.
>>> > > >
>>> > > > -Max
>>> > > >
>>> > > > On Wed, Dec 13, 2023 at 4:59 AM Rui Fan <1996fan...@gmail.com>
>>> wrote:
>>> > > > >
>>> > > > > Hi devs,
>>> > > > >
>>> > > > > I'd like to start a discussion to discuss whether Configuration
>>> > > 

[SUMMARY] Flink 1.19 Release Sync 12/12/2023

[Lincoln Lee]

Hi devs,

This is the summary of the 3rd release sync of Flink 1.19:

- *Features & issues tracking*
  There're 7 weeks until the feature freeze date, so far we've had 23
flips/features and the status looks good. It is encouraged to continuously
updating the 1.19 wiki page[1] for contributors.
  - FLINK-33528 Externalize Python connector code[2], volunteers on the
related connectors are welcome!
  - Voting for flink-shaded[3] version 1.18.0 is still open, binding votes
are needed!

- *Blockers*
  - [closed] external connectors compilation failure cause by FLINK-25857,
discussion thread[3]
  - [potential blocker] the new JUnit5 fails to delete a directory on AZP
for various table-planner tests FLINK-33641[4]

- *Sync meeting* (https://meet.google.com/vcx-arzs-trv)
  The next scheduled release time sync is *December 26, 2023*, but given
that it falls during the Christmas holidays, we'd like to hear your
suggestions on whether to skip this sync?
  If we don't receive feedback by *December 25*, the sync will go on as
usual.

Lastly, we encourage attendees to fill out the topics to be discussed at
the bottom of 1.19 wiki page[1] a day in advance, make it easier for
everyone to understand the background of the topics.

[1] https://cwiki.apache.org/confluence/display/FLINK/1.19+Release
[2] https://issues.apache.org/jira/browse/FLINK-33528
[3] https://lists.apache.org/thread/h6nkgth838dlh5s90sd95zd6hlsxwx57
[4] https://issues.apache.org/jira/browse/FLINK-33641

Best,
Yun, Jing, Martijn and Lincoln

Re: join flink slack

[Romit Mahanta]

Second that

On Wed, 13 Dec, 2023, 5:58 pm yd c,  wrote:

> can u invite flink slack
>

join flink slack

[yd c]

can u invite flink slack

Re: [DISCUSS] Should Configuration support getting value based on String key?

[Zhu Zhu]

Hi Rui,

I'd like to understand why there is a strong requirement for these
deprecated
methods. The ConfigOption param methods help to do the type conversion so
that users do not need to do it by themselves.

For the 2 reasons to keep these methods mentioned above:

> 1. A lot of scenarios don't define the ConfigOption, they using
String as the key and value directly, such as: StreamConfig,
TaskConfig, DistributedCache, etc.

These are internal usages and we can have an internal getter method for
them.
It is not a blocker of the deprecation, epsecially given that they are not
standard
configuration and are just using Configuration class for convenience.

> 2. Some code wanna convert all keys or values, this converting
is generic, so the getString(String key, String defaultValue) is needed.
Such as: kubernetes-operator [3].

Not sure why it's required to convert all keys or values. If it is used
to create strings for dynamic properties or config files to deploy jobs,
toMap()/toFileWritableMap() may be a better choice. They are already
used in this kind of scenarios.
If it just needs to read some of the config, why not using the proposed
way to read and parse the config? Pre-defined ConfigOptions are better
for configuration maintenance, compared to arbitrary strings

Thanks,
Zhu

Rui Fan <1996fan...@gmail.com> 于2023年12月13日周三 19:27写道：

> Thanks Martijn for the quick clarification!
>
> I see Zhu Zhu and Junrui Lee are working on configuration related
> work of Flink-2.0. I would cc them, and hear some thoughts from them.
>
> Best,
> Rui
>
> On Wed, Dec 13, 2023 at 7:17 PM Martijn Visser 
> wrote:
>
>> Hi Rui,
>>
>> I'm more wondering if part of the configuration layer changes would
>> mean that these APIs would be removed in 2.0. Because if so, then I
>> don't think we should remove the Deprecate annotation. But I have very
>> little visibility on the plans for the configuration layer.
>>
>> Thanks,
>>
>> Martijn
>>
>> On Wed, Dec 13, 2023 at 12:15 PM Rui Fan <1996fan...@gmail.com> wrote:
>> >
>> > Hi Martijn,
>> >
>> > Thanks for your reply!
>> >
>> > I noticed the 2.0 is doing some work related to clean configuration.
>> > But this discussion is different from other work. Most of other work
>> > are deprecate some Apis in Flink-1.19 and remove them in Flink-2.0.
>> >
>> > This discussion is a series of methods have been marked to @Deprecate,
>> > but they are still used so far. I propose remove the @Deprecate
>> annotation
>> > and keep these methods.
>> >
>> > In brief, I think some deprecated methods shouldn't be deprecated.
>> > WDYT?
>> >
>> > Please correct me if I'm wrong, thanks~
>> >
>> > Best,
>> > Rui
>> >
>> > On Wed, Dec 13, 2023 at 7:07 PM Martijn Visser <
>> martijnvis...@apache.org>
>> > wrote:
>> >
>> > > Hi Rui,
>> > >
>> > > Are you thinking about this as part of Flink 2.0, since that has the
>> > > goal to introduce a completely clean configuration layer? [1]
>> > >
>> > > Best regards,
>> > >
>> > > Martijn
>> > >
>> > > [1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release
>> > >
>> > > On Wed, Dec 13, 2023 at 11:28 AM Maximilian Michels 
>> > > wrote:
>> > > >
>> > > > Hi Rui,
>> > > >
>> > > > +1 for removing the @Deprecated annotation from `getString(String
>> key,
>> > > > String defaultValue)`. I would remove the other typed variants with
>> > > > default values but I'm ok with keeping them if they are still used.
>> > > >
>> > > > -Max
>> > > >
>> > > > On Wed, Dec 13, 2023 at 4:59 AM Rui Fan <1996fan...@gmail.com>
>> wrote:
>> > > > >
>> > > > > Hi devs,
>> > > > >
>> > > > > I'd like to start a discussion to discuss whether Configuration
>> > > supports
>> > > > > getting value based on the String key.
>> > > > >
>> > > > > In the FLIP-77[1] and FLINK-14493[2], a series of methods of
>> > > Configuration
>> > > > > are marked as @Deprecated, for example:
>> > > > > - public String getString(String key, String defaultValue)
>> > > > > - public long getLong(String key, long defaultValue)
>> > > > > - public boolean getBoolean(String key, boolean defaultValue)
>> > > > > - public int getInteger(String key, int defaultValue)
>> > > > >
>> > > > > The java doc suggests using getString(ConfigOption, String) or
>> > > > > getOptional(ConfigOption), it means using ConfigOption as key
>> > > > > instead of String.
>> > > > >
>> > > > > They are depreated since Flink-1.10, but these methods still
>> > > > > be used in a lot of code. I think getString(String key, String
>> > > > > defaultValue)
>> > > > > shouldn't be deprecated with 2 reasons:
>> > > > >
>> > > > > 1. A lot of scenarios don't define the ConfigOption, they using
>> > > > > String as the key and value directly, such as: StreamConfig,
>> > > > > TaskConfig, DistributedCache, etc.
>> > > > >
>> > > > > 2. Some code wanna convert all keys or values, this converting
>> > > > > is generic, so the getString(String key, String defaultValue) is
>> > > needed.
>> > > > > Such as: kubernetes-operator [3].
>> > > > >
>> > 

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

[Leonard Xu]

Hey, Zarzycki

Thanks for your reply and recognization ! 

> As I understand you're thinking with Flink CDC 3.0 to make the project
> broader than CDC, as more "Streaming Integration" framework. This will
> resemble Kafka Connect, what about renaming the project to something like
> "Flink Connect"? :-)
> It would be amazing to have such a tool, "swiss army knife" for data
> integration, based on Flink.

Flink Community has a lot of connectors thus the Flink Connect name may not 
suitable.
Although we offers end-to-end streaming integration capability , we still want 
to use the current name 
Flink CDC as we want to do best in database CDC area and users has 
accepted/known the name well. 
While the project name holds some significance, I believe its true importance 
lies in its ability to effectively
 address problems and deliver substantial value to the community users.

Best,
Leonard


> 
> 
> On Tue, Dec 12, 2023 at 3:01 PM Lincoln Lee  wrote:
> 
>> +1
>> Thanks Leonard and all CDC contributors for the great work!
>> 
>> Best,
>> Lincoln Lee
>> 
>> 
>> Dian Fu  于2023年12月12日周二 19:11写道：
>> 
>>> Big +1 for this! Thanks for driving this effort!
>>> 
>>> Regards,
>>> Dian
>>> 
>>> On Mon, Dec 11, 2023 at 4:48 PM Leonard Xu  wrote:
 
 Hey all,
 
 I want to extend my gratitude for the positive feedback we've received,
>>> particularly from voices in other communities. It is an honor for the
>> Flink
>>> CDC community to have received your recognition.
 
 Moving forward, I will collaborate with the Flink CDC community
>>> maintainers to prepare a FLIP (Flink Improvement Proposal) document that
>>> will outline specific plans and implementation details.
 
 Thank you for your support, and we look forward to contributing to
>>> Apache Flink community.
 
 Best,
 Leonard
 
> 2023年12月11日 下午4:12，yh z  写道：
> 
> Excited to hear the news,
> +1
> 
> Best,
> Yunhong (Swuferhong)
> 
> liu ron  于2023年12月11日周一 14:05写道：
> 
>> +1
>> 
>> Best,
>> Ron
>> 
>> Yunqing Mo  于2023年12月11日周一 12:01写道：
>> 
>>> So cool, Big +1 for this exciting work.
>>> 
>>> On 2023/12/07 03:24:59 Leonard Xu wrote:
 Dear Flink devs,
 
 As you may have heard, we at Alibaba (Ververica) are planning to
>>> donate
>>> CDC Connectors for the Apache Flink project[1] to the Apache Flink
>>> community.
 
 CDC Connectors for Apache Flink comprise a collection of source
>>> connectors designed specifically for Apache Flink. These
>>> connectors[2]
>>> enable the ingestion of changes from various databases using Change
>>> Data
>>> Capture (CDC), most of these CDC connectors are powered by
>>> Debezium[3].
>>> They support both the DataStream API and the Table/SQL API,
>>> facilitating
>>> the reading of database snapshots and continuous reading of
>>> transaction
>>> logs with exactly-once processing, even in the event of failures.
 
 
 Additionally, in the latest version 3.0, we have introduced many
>>> long-awaited features. Starting from CDC version 3.0, we've built a
>>> Streaming ELT Framework available for streaming data integration.
>>> This
>>> framework allows users to write their data synchronization logic
>> in a
>>> simple YAML file, which will automatically be translated into a
>> Flink
>>> DataStreaming job. It emphasizes optimizing the task submission
>>> process
>> and
>>> offers advanced functionalities such as whole database
>>> synchronization,
>>> merging sharded tables, and schema evolution[4].
 
 
 I believe this initiative is a perfect match for both sides. For
>> the
>>> Flink community, it presents an opportunity to enhance Flink's
>> competitive
>>> advantage in streaming data integration, promoting the healthy
>>> growth and
>>> prosperity of the Apache Flink ecosystem. For the CDC Connectors
>>> project,
>>> becoming a sub-project of Apache Flink means being part of a
>> neutral
>>> open-source community, which can attract a more diverse pool of
>>> contributors.
 
 Please note that the aforementioned points represent only some of
>>> our
>>> motivations and vision for this donation. Specific future
>> operations
>>> need
>>> to be further discussed in this thread. For example, the
>> sub-project
>>> name
>>> after the donation; we hope to name it Flink-CDC aiming to
>> streaming
>>> data
>>> intergration through Apache Flink, following the naming convention
>> of
>>> Flink-ML; And this project is managed by a total of 8 maintainers,
>>> including 3 Flink PMC members and 1 Flink Committer. The remaining
>> 4
>>> maintainers are also highly active contributors to the Flink
>>> community,
>>> donating this project to the Flink community implies that their
>> permissions
>>> might 

Re: [DISCUSS] Should Configuration support getting value based on String key?

[Rui Fan]

Thanks Martijn for the quick clarification!

I see Zhu Zhu and Junrui Lee are working on configuration related
work of Flink-2.0. I would cc them, and hear some thoughts from them.

Best,
Rui

On Wed, Dec 13, 2023 at 7:17 PM Martijn Visser 
wrote:

> Hi Rui,
>
> I'm more wondering if part of the configuration layer changes would
> mean that these APIs would be removed in 2.0. Because if so, then I
> don't think we should remove the Deprecate annotation. But I have very
> little visibility on the plans for the configuration layer.
>
> Thanks,
>
> Martijn
>
> On Wed, Dec 13, 2023 at 12:15 PM Rui Fan <1996fan...@gmail.com> wrote:
> >
> > Hi Martijn,
> >
> > Thanks for your reply!
> >
> > I noticed the 2.0 is doing some work related to clean configuration.
> > But this discussion is different from other work. Most of other work
> > are deprecate some Apis in Flink-1.19 and remove them in Flink-2.0.
> >
> > This discussion is a series of methods have been marked to @Deprecate,
> > but they are still used so far. I propose remove the @Deprecate
> annotation
> > and keep these methods.
> >
> > In brief, I think some deprecated methods shouldn't be deprecated.
> > WDYT?
> >
> > Please correct me if I'm wrong, thanks~
> >
> > Best,
> > Rui
> >
> > On Wed, Dec 13, 2023 at 7:07 PM Martijn Visser  >
> > wrote:
> >
> > > Hi Rui,
> > >
> > > Are you thinking about this as part of Flink 2.0, since that has the
> > > goal to introduce a completely clean configuration layer? [1]
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > [1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release
> > >
> > > On Wed, Dec 13, 2023 at 11:28 AM Maximilian Michels 
> > > wrote:
> > > >
> > > > Hi Rui,
> > > >
> > > > +1 for removing the @Deprecated annotation from `getString(String
> key,
> > > > String defaultValue)`. I would remove the other typed variants with
> > > > default values but I'm ok with keeping them if they are still used.
> > > >
> > > > -Max
> > > >
> > > > On Wed, Dec 13, 2023 at 4:59 AM Rui Fan <1996fan...@gmail.com>
> wrote:
> > > > >
> > > > > Hi devs,
> > > > >
> > > > > I'd like to start a discussion to discuss whether Configuration
> > > supports
> > > > > getting value based on the String key.
> > > > >
> > > > > In the FLIP-77[1] and FLINK-14493[2], a series of methods of
> > > Configuration
> > > > > are marked as @Deprecated, for example:
> > > > > - public String getString(String key, String defaultValue)
> > > > > - public long getLong(String key, long defaultValue)
> > > > > - public boolean getBoolean(String key, boolean defaultValue)
> > > > > - public int getInteger(String key, int defaultValue)
> > > > >
> > > > > The java doc suggests using getString(ConfigOption, String) or
> > > > > getOptional(ConfigOption), it means using ConfigOption as key
> > > > > instead of String.
> > > > >
> > > > > They are depreated since Flink-1.10, but these methods still
> > > > > be used in a lot of code. I think getString(String key, String
> > > > > defaultValue)
> > > > > shouldn't be deprecated with 2 reasons:
> > > > >
> > > > > 1. A lot of scenarios don't define the ConfigOption, they using
> > > > > String as the key and value directly, such as: StreamConfig,
> > > > > TaskConfig, DistributedCache, etc.
> > > > >
> > > > > 2. Some code wanna convert all keys or values, this converting
> > > > > is generic, so the getString(String key, String defaultValue) is
> > > needed.
> > > > > Such as: kubernetes-operator [3].
> > > > >
> > > > > Based on it, I have 2 solutions:
> > > > >
> > > > > 1. Removing the @Deprecated for these methods.
> > > > >
> > > > > 2. Only removing the @Deprecated for `public String
> getString(String
> > > key,
> > > > > String defaultValue)`
> > > > > and delete other getXxx(String key, Xxx defaultValue) directly.
> > > > > They have been depreated 8 minor versions ago. In general, the
> > > > > getString can replace getInteger, getBoolean, etc.
> > > > >
> > > > > I prefer solution1, because these getXxx methods are used for now,
> > > > > they are easy to use and don't bring large maintenance costs.
> > > > >
> > > > > Note: The alternative to public String getString(String key, String
> > > > > defaultValue)
> > > > > is Configuration.toMap. But the ease of use is not very convenient.
> > > > >
> > > > > Looking forward to hear more thoughts about it! Thank you~
> > > > > Also, very much looking forward to feedback from Dawid, the author
> of
> > > > > FLIP-77.
> > > > >
> > > > > [1] https://cwiki.apache.org/confluence/x/_RPABw
> > > > > [2] https://issues.apache.org/jira/browse/FLINK-14493
> > > > > [3]
> > > > >
> > >
> https://github.com/apache/flink-kubernetes-operator/pull/729/files#r1424811105
> > > > >
> > > > > Best,
> > > > > Rui
> > >
>

Re: [DISCUSS] Should Configuration support getting value based on String key?

[Martijn Visser]

Hi Rui,

I'm more wondering if part of the configuration layer changes would
mean that these APIs would be removed in 2.0. Because if so, then I
don't think we should remove the Deprecate annotation. But I have very
little visibility on the plans for the configuration layer.

Thanks,

Martijn

On Wed, Dec 13, 2023 at 12:15 PM Rui Fan <1996fan...@gmail.com> wrote:
>
> Hi Martijn,
>
> Thanks for your reply!
>
> I noticed the 2.0 is doing some work related to clean configuration.
> But this discussion is different from other work. Most of other work
> are deprecate some Apis in Flink-1.19 and remove them in Flink-2.0.
>
> This discussion is a series of methods have been marked to @Deprecate,
> but they are still used so far. I propose remove the @Deprecate annotation
> and keep these methods.
>
> In brief, I think some deprecated methods shouldn't be deprecated.
> WDYT?
>
> Please correct me if I'm wrong, thanks~
>
> Best,
> Rui
>
> On Wed, Dec 13, 2023 at 7:07 PM Martijn Visser 
> wrote:
>
> > Hi Rui,
> >
> > Are you thinking about this as part of Flink 2.0, since that has the
> > goal to introduce a completely clean configuration layer? [1]
> >
> > Best regards,
> >
> > Martijn
> >
> > [1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release
> >
> > On Wed, Dec 13, 2023 at 11:28 AM Maximilian Michels 
> > wrote:
> > >
> > > Hi Rui,
> > >
> > > +1 for removing the @Deprecated annotation from `getString(String key,
> > > String defaultValue)`. I would remove the other typed variants with
> > > default values but I'm ok with keeping them if they are still used.
> > >
> > > -Max
> > >
> > > On Wed, Dec 13, 2023 at 4:59 AM Rui Fan <1996fan...@gmail.com> wrote:
> > > >
> > > > Hi devs,
> > > >
> > > > I'd like to start a discussion to discuss whether Configuration
> > supports
> > > > getting value based on the String key.
> > > >
> > > > In the FLIP-77[1] and FLINK-14493[2], a series of methods of
> > Configuration
> > > > are marked as @Deprecated, for example:
> > > > - public String getString(String key, String defaultValue)
> > > > - public long getLong(String key, long defaultValue)
> > > > - public boolean getBoolean(String key, boolean defaultValue)
> > > > - public int getInteger(String key, int defaultValue)
> > > >
> > > > The java doc suggests using getString(ConfigOption, String) or
> > > > getOptional(ConfigOption), it means using ConfigOption as key
> > > > instead of String.
> > > >
> > > > They are depreated since Flink-1.10, but these methods still
> > > > be used in a lot of code. I think getString(String key, String
> > > > defaultValue)
> > > > shouldn't be deprecated with 2 reasons:
> > > >
> > > > 1. A lot of scenarios don't define the ConfigOption, they using
> > > > String as the key and value directly, such as: StreamConfig,
> > > > TaskConfig, DistributedCache, etc.
> > > >
> > > > 2. Some code wanna convert all keys or values, this converting
> > > > is generic, so the getString(String key, String defaultValue) is
> > needed.
> > > > Such as: kubernetes-operator [3].
> > > >
> > > > Based on it, I have 2 solutions:
> > > >
> > > > 1. Removing the @Deprecated for these methods.
> > > >
> > > > 2. Only removing the @Deprecated for `public String getString(String
> > key,
> > > > String defaultValue)`
> > > > and delete other getXxx(String key, Xxx defaultValue) directly.
> > > > They have been depreated 8 minor versions ago. In general, the
> > > > getString can replace getInteger, getBoolean, etc.
> > > >
> > > > I prefer solution1, because these getXxx methods are used for now,
> > > > they are easy to use and don't bring large maintenance costs.
> > > >
> > > > Note: The alternative to public String getString(String key, String
> > > > defaultValue)
> > > > is Configuration.toMap. But the ease of use is not very convenient.
> > > >
> > > > Looking forward to hear more thoughts about it! Thank you~
> > > > Also, very much looking forward to feedback from Dawid, the author of
> > > > FLIP-77.
> > > >
> > > > [1] https://cwiki.apache.org/confluence/x/_RPABw
> > > > [2] https://issues.apache.org/jira/browse/FLINK-14493
> > > > [3]
> > > >
> > https://github.com/apache/flink-kubernetes-operator/pull/729/files#r1424811105
> > > >
> > > > Best,
> > > > Rui
> >

[VOTE] Release flink-connector-pulsar 4.1.0, release candidate #1

[Leonard Xu]

Hey all,

Please review and vote on the release candidate #1 for the version 4.1.0 of the 
Apache Flink Pulsar Connector as follows:

[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* The official Apache source release to be deployed to dist.apache.org [2], 
which are signed with the key with fingerprint
5B2F6608732389AEB67331F5B197E1F1108998AD [3],
* All artifacts to be deployed to the Maven Central Repository [4],
* Source code tag v4.1.0-rc1 [5],
* Website pull request listing the new release [6].

The vote will be open for at least 72 hours. It is adopted by majority 
approval, with at least 3 PMC affirmative votes.


Best,
Leonard

[1] 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353431
[2] 
https://dist.apache.org/repos/dist/dev/flink/flink-connector-pulsar-4.1.0-rc1/
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] https://repository.apache.org/content/repositories/orgapacheflink-1688/
[5] https://github.com/apache/flink-connector-pulsar/tree/v4.1.0-rc1
[6] https://github.com/apache/flink-web/pull/703

Re: [DISCUSS] Should Configuration support getting value based on String key?

[Rui Fan]

Hi Martijn,

Thanks for your reply!

I noticed the 2.0 is doing some work related to clean configuration.
But this discussion is different from other work. Most of other work
are deprecate some Apis in Flink-1.19 and remove them in Flink-2.0.

This discussion is a series of methods have been marked to @Deprecate,
but they are still used so far. I propose remove the @Deprecate annotation
and keep these methods.

In brief, I think some deprecated methods shouldn't be deprecated.
WDYT?

Please correct me if I'm wrong, thanks~

Best,
Rui

On Wed, Dec 13, 2023 at 7:07 PM Martijn Visser 
wrote:

> Hi Rui,
>
> Are you thinking about this as part of Flink 2.0, since that has the
> goal to introduce a completely clean configuration layer? [1]
>
> Best regards,
>
> Martijn
>
> [1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release
>
> On Wed, Dec 13, 2023 at 11:28 AM Maximilian Michels 
> wrote:
> >
> > Hi Rui,
> >
> > +1 for removing the @Deprecated annotation from `getString(String key,
> > String defaultValue)`. I would remove the other typed variants with
> > default values but I'm ok with keeping them if they are still used.
> >
> > -Max
> >
> > On Wed, Dec 13, 2023 at 4:59 AM Rui Fan <1996fan...@gmail.com> wrote:
> > >
> > > Hi devs,
> > >
> > > I'd like to start a discussion to discuss whether Configuration
> supports
> > > getting value based on the String key.
> > >
> > > In the FLIP-77[1] and FLINK-14493[2], a series of methods of
> Configuration
> > > are marked as @Deprecated, for example:
> > > - public String getString(String key, String defaultValue)
> > > - public long getLong(String key, long defaultValue)
> > > - public boolean getBoolean(String key, boolean defaultValue)
> > > - public int getInteger(String key, int defaultValue)
> > >
> > > The java doc suggests using getString(ConfigOption, String) or
> > > getOptional(ConfigOption), it means using ConfigOption as key
> > > instead of String.
> > >
> > > They are depreated since Flink-1.10, but these methods still
> > > be used in a lot of code. I think getString(String key, String
> > > defaultValue)
> > > shouldn't be deprecated with 2 reasons:
> > >
> > > 1. A lot of scenarios don't define the ConfigOption, they using
> > > String as the key and value directly, such as: StreamConfig,
> > > TaskConfig, DistributedCache, etc.
> > >
> > > 2. Some code wanna convert all keys or values, this converting
> > > is generic, so the getString(String key, String defaultValue) is
> needed.
> > > Such as: kubernetes-operator [3].
> > >
> > > Based on it, I have 2 solutions:
> > >
> > > 1. Removing the @Deprecated for these methods.
> > >
> > > 2. Only removing the @Deprecated for `public String getString(String
> key,
> > > String defaultValue)`
> > > and delete other getXxx(String key, Xxx defaultValue) directly.
> > > They have been depreated 8 minor versions ago. In general, the
> > > getString can replace getInteger, getBoolean, etc.
> > >
> > > I prefer solution1, because these getXxx methods are used for now,
> > > they are easy to use and don't bring large maintenance costs.
> > >
> > > Note: The alternative to public String getString(String key, String
> > > defaultValue)
> > > is Configuration.toMap. But the ease of use is not very convenient.
> > >
> > > Looking forward to hear more thoughts about it! Thank you~
> > > Also, very much looking forward to feedback from Dawid, the author of
> > > FLIP-77.
> > >
> > > [1] https://cwiki.apache.org/confluence/x/_RPABw
> > > [2] https://issues.apache.org/jira/browse/FLINK-14493
> > > [3]
> > >
> https://github.com/apache/flink-kubernetes-operator/pull/729/files#r1424811105
> > >
> > > Best,
> > > Rui
>

Re: [DISCUSS] Should Configuration support getting value based on String key?

[Martijn Visser]

Hi Rui,

Are you thinking about this as part of Flink 2.0, since that has the
goal to introduce a completely clean configuration layer? [1]

Best regards,

Martijn

[1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release

On Wed, Dec 13, 2023 at 11:28 AM Maximilian Michels  wrote:
>
> Hi Rui,
>
> +1 for removing the @Deprecated annotation from `getString(String key,
> String defaultValue)`. I would remove the other typed variants with
> default values but I'm ok with keeping them if they are still used.
>
> -Max
>
> On Wed, Dec 13, 2023 at 4:59 AM Rui Fan <1996fan...@gmail.com> wrote:
> >
> > Hi devs,
> >
> > I'd like to start a discussion to discuss whether Configuration supports
> > getting value based on the String key.
> >
> > In the FLIP-77[1] and FLINK-14493[2], a series of methods of Configuration
> > are marked as @Deprecated, for example:
> > - public String getString(String key, String defaultValue)
> > - public long getLong(String key, long defaultValue)
> > - public boolean getBoolean(String key, boolean defaultValue)
> > - public int getInteger(String key, int defaultValue)
> >
> > The java doc suggests using getString(ConfigOption, String) or
> > getOptional(ConfigOption), it means using ConfigOption as key
> > instead of String.
> >
> > They are depreated since Flink-1.10, but these methods still
> > be used in a lot of code. I think getString(String key, String
> > defaultValue)
> > shouldn't be deprecated with 2 reasons:
> >
> > 1. A lot of scenarios don't define the ConfigOption, they using
> > String as the key and value directly, such as: StreamConfig,
> > TaskConfig, DistributedCache, etc.
> >
> > 2. Some code wanna convert all keys or values, this converting
> > is generic, so the getString(String key, String defaultValue) is needed.
> > Such as: kubernetes-operator [3].
> >
> > Based on it, I have 2 solutions:
> >
> > 1. Removing the @Deprecated for these methods.
> >
> > 2. Only removing the @Deprecated for `public String getString(String key,
> > String defaultValue)`
> > and delete other getXxx(String key, Xxx defaultValue) directly.
> > They have been depreated 8 minor versions ago. In general, the
> > getString can replace getInteger, getBoolean, etc.
> >
> > I prefer solution1, because these getXxx methods are used for now,
> > they are easy to use and don't bring large maintenance costs.
> >
> > Note: The alternative to public String getString(String key, String
> > defaultValue)
> > is Configuration.toMap. But the ease of use is not very convenient.
> >
> > Looking forward to hear more thoughts about it! Thank you~
> > Also, very much looking forward to feedback from Dawid, the author of
> > FLIP-77.
> >
> > [1] https://cwiki.apache.org/confluence/x/_RPABw
> > [2] https://issues.apache.org/jira/browse/FLINK-14493
> > [3]
> > https://github.com/apache/flink-kubernetes-operator/pull/729/files#r1424811105
> >
> > Best,
> > Rui

Re: [DISCUSS] Should Configuration support getting value based on String key?

[Maximilian Michels]

Hi Rui,

+1 for removing the @Deprecated annotation from `getString(String key,
String defaultValue)`. I would remove the other typed variants with
default values but I'm ok with keeping them if they are still used.

-Max

On Wed, Dec 13, 2023 at 4:59 AM Rui Fan <1996fan...@gmail.com> wrote:
>
> Hi devs,
>
> I'd like to start a discussion to discuss whether Configuration supports
> getting value based on the String key.
>
> In the FLIP-77[1] and FLINK-14493[2], a series of methods of Configuration
> are marked as @Deprecated, for example:
> - public String getString(String key, String defaultValue)
> - public long getLong(String key, long defaultValue)
> - public boolean getBoolean(String key, boolean defaultValue)
> - public int getInteger(String key, int defaultValue)
>
> The java doc suggests using getString(ConfigOption, String) or
> getOptional(ConfigOption), it means using ConfigOption as key
> instead of String.
>
> They are depreated since Flink-1.10, but these methods still
> be used in a lot of code. I think getString(String key, String
> defaultValue)
> shouldn't be deprecated with 2 reasons:
>
> 1. A lot of scenarios don't define the ConfigOption, they using
> String as the key and value directly, such as: StreamConfig,
> TaskConfig, DistributedCache, etc.
>
> 2. Some code wanna convert all keys or values, this converting
> is generic, so the getString(String key, String defaultValue) is needed.
> Such as: kubernetes-operator [3].
>
> Based on it, I have 2 solutions:
>
> 1. Removing the @Deprecated for these methods.
>
> 2. Only removing the @Deprecated for `public String getString(String key,
> String defaultValue)`
> and delete other getXxx(String key, Xxx defaultValue) directly.
> They have been depreated 8 minor versions ago. In general, the
> getString can replace getInteger, getBoolean, etc.
>
> I prefer solution1, because these getXxx methods are used for now,
> they are easy to use and don't bring large maintenance costs.
>
> Note: The alternative to public String getString(String key, String
> defaultValue)
> is Configuration.toMap. But the ease of use is not very convenient.
>
> Looking forward to hear more thoughts about it! Thank you~
> Also, very much looking forward to feedback from Dawid, the author of
> FLIP-77.
>
> [1] https://cwiki.apache.org/confluence/x/_RPABw
> [2] https://issues.apache.org/jira/browse/FLINK-14493
> [3]
> https://github.com/apache/flink-kubernetes-operator/pull/729/files#r1424811105
>
> Best,
> Rui

[jira] [Created] (FLINK-33816) SourceStreamTaskTest.testTriggeringStopWithSavepointWithDrain failed due async checkpoint triggering not being completed

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-33816:
-

 Summary: 
SourceStreamTaskTest.testTriggeringStopWithSavepointWithDrain failed due async 
checkpoint triggering not being completed 
 Key: FLINK-33816
 URL: https://issues.apache.org/jira/browse/FLINK-33816
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Checkpointing, Runtime / Coordination
Affects Versions: 1.19.0
Reporter: Matthias Pohl


[https://github.com/XComp/flink/actions/runs/7182604625/job/19559947894#step:12:9430]
{code:java}
rror: 14:39:01 14:39:01.930 [ERROR] Tests run: 16, Failures: 1, Errors: 0, 
Skipped: 0, Time elapsed: 1.878 s <<< FAILURE! - in 
org.apache.flink.streaming.runtime.tasks.SourceStreamTaskTest
9426Error: 14:39:01 14:39:01.930 [ERROR] 
org.apache.flink.streaming.runtime.tasks.SourceStreamTaskTest.testTriggeringStopWithSavepointWithDrain
  Time elapsed: 0.034 s  <<< FAILURE!
9427Dec 12 14:39:01 org.opentest4j.AssertionFailedError: 
9428Dec 12 14:39:01 
9429Dec 12 14:39:01 Expecting value to be true but was false
9430Dec 12 14:39:01 at 
java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
9431Dec 12 14:39:01 at 
java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
9432Dec 12 14:39:01 at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTaskTest.testTriggeringStopWithSavepointWithDrain(SourceStreamTaskTest.java:710)
9433Dec 12 14:39:01 at 
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
9434Dec 12 14:39:01 at 
java.base/java.lang.reflect.Method.invoke(Method.java:580)
[...] {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33815) Add tests against jdk17 for pulsar connector

[Sergey Nuyanzin (Jira)]

Sergey Nuyanzin created FLINK-33815:
---

 Summary: Add tests against jdk17 for pulsar connector
 Key: FLINK-33815
 URL: https://issues.apache.org/jira/browse/FLINK-33815
 Project: Flink
  Issue Type: Technical Debt
  Components: Connectors / Pulsar
Reporter: Sergey Nuyanzin
Assignee: Sergey Nuyanzin






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Release Flink 1.18.1

[Jing Ge]

Hi Martijn,

Thanks for the heads-up! I already started 1.18.1-rc1[1]. But please feel
free to ping me. I will either redo rc1 or start rc2. Thanks!

Best regards,
Jing

[1] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.1-rc1/

On Tue, Dec 12, 2023 at 9:49 PM Martijn Visser 
wrote:

> Hi Jing,
>
> The only thing that was brought to my attention today was
> https://issues.apache.org/jira/browse/FLINK-33793. I've marked it as a
> Critical (I think we should have working checkpointing with GCS), but
> could also be considered a Blocker. There is a PR open for it, but I
> don't think it's the right fix so I'm testing
>
> https://github.com/apache/flink/commit/2ebe7f6e090690486c1954099a2b57283c578192
> at this moment. If you haven't started the 1.18.1 release yet, perhaps
> we could include it (hopefully we can merge it tomorrow), else it
> would have to wait for the next release.
>
> Thanks,
>
> Martijn
>
> On Tue, Dec 12, 2023 at 9:30 AM Jing Ge 
> wrote:
> >
> > Hi All,
> >
> > Thank you for your feedback!
> >
> > Since FLINK-33523[1] is done(thanks Martijn for driving it) and there are
> > no other concerns or objections. Currently I am not aware of any
> unresolved
> > blockers. There is one critical task[2] whose PR still has some
> checkstyle
> > issues. I will try to include it into 1.18.1 with best effort, since it
> > depends on how quickly the author can finish it and cp to 1.18. If anyone
> > considers FLINK-33588 as a must-have fix in 1.18.1, please let me know,
> > thanks!
> >
> > I will start working on the RC1 release today.
> >
> > @benchao: gotcha wrt FLINK-33313
> >
> > Best regards,
> > Jing
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-33523
> > [2] https://issues.apache.org/jira/browse/FLINK-33588
> >
> > On Tue, Dec 12, 2023 at 7:03 AM weijie guo 
> > wrote:
> >
> > > Thanks Jing for driving this bug-fix release.
> > >
> > > +1 from my side.
> > >
> > > Best regards,
> > >
> > > Weijie
> > >
> > >
> > > Jark Wu  于2023年12月12日周二 12:17写道：
> > >
> > > > Thanks Jing for driving 1.18.1.
> > > > +1 for this.
> > > >
> > > > Best,
> > > > Jark
> > > >
> > > > On Mon, 11 Dec 2023 at 16:59, Hong Liang  wrote:
> > > >
> > > > > +1. Thanks Jing for driving this.
> > > > >
> > > > > Hong
> > > > >
> > > > > On Mon, Dec 11, 2023 at 2:27 AM Yun Tang  wrote:
> > > > >
> > > > > > Thanks Jing for driving 1.18.1 release, +1 for this.
> > > > > >
> > > > > >
> > > > > > Best
> > > > > > Yun Tang
> > > > > > 
> > > > > > From: Rui Fan <1996fan...@gmail.com>
> > > > > > Sent: Saturday, December 9, 2023 21:46
> > > > > > To: dev@flink.apache.org 
> > > > > > Subject: Re: [DISCUSS] Release Flink 1.18.1
> > > > > >
> > > > > > Thanks Jing for driving this release, +1
> > > > > >
> > > > > > Best,
> > > > > > Rui
> > > > > >
> > > > > > On Sat, Dec 9, 2023 at 7:33 AM Leonard Xu 
> wrote:
> > > > > >
> > > > > > > Thanks Jing for driving this release, +1
> > > > > > >
> > > > > > > Best,
> > > > > > > Leonard
> > > > > > >
> > > > > > > > 2023年12月9日 上午1:23，Danny Cranmer 
> 写道：
> > > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > > Thanks for driving this
> > > > > > > >
> > > > > > > > On Fri, 8 Dec 2023, 12:05 Timo Walther, 
> > > > wrote:
> > > > > > > >
> > > > > > > >> Thanks for taking care of this Jing.
> > > > > > > >>
> > > > > > > >> +1 to release 1.18.1 for this.
> > > > > > > >>
> > > > > > > >> Cheers,
> > > > > > > >> Timo
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On 08.12.23 10:00, Benchao Li wrote:
> > > > > > > >>> I've merged FLINK-33313 to release-1.18 branch.
> > > > > > > >>>
> > > > > > > >>> Péter Váry  于2023年12月8日周五
> > > 16:56写道：
> > > > > > > 
> > > > > > >  Hi Jing,
> > > > > > >  Thanks for taking care of this!
> > > > > > >  +1 (non-binding)
> > > > > > >  Peter
> > > > > > > 
> > > > > > >  Sergey Nuyanzin  ezt írta (időpont:
> > > 2023.
> > > > > dec.
> > > > > > > >> 8., P,
> > > > > > >  9:36):
> > > > > > > 
> > > > > > > > Thanks Jing driving it
> > > > > > > > +1
> > > > > > > >
> > > > > > > > also +1 to include FLINK-33313 mentioned by Benchao Li
> > > > > > > >
> > > > > > > > On Fri, Dec 8, 2023 at 9:17 AM Benchao Li <
> > > > libenc...@apache.org>
> > > > > > > >> wrote:
> > > > > > > >
> > > > > > > >> Thanks Jing for driving 1.18.1 releasing.
> > > > > > > >>
> > > > > > > >> I would like to include FLINK-33313[1] in 1.18.1, it's
> just
> > > a
> > > > > > > bugfix,
> > > > > > > >> not a blocker, but it's already merged into master, I
> plan
> > > to
> > > > > > merge
> > > > > > > it
> > > > > > > >> to 1.8/1.7 branches today after the CI passes.
> > > > > > > >>
> > > > > > > >> [1] https://issues.apache.org/jira/browse/FLINK-33313
> > > > > > > >>
> > > > > > > >> Jing Ge  于2023年12月8日周五
> 16:06写道：
> > > > > > > >>>
> > > > > > > >>> Hi all,
> > > > > > > >>>
> > > > > > > >>> I 

[jira] [Created] (FLINK-33814) Autoscaler Standalone control loop supports multiple thread

[Rui Fan (Jira)]

Rui Fan created FLINK-33814:
---

 Summary: Autoscaler Standalone control loop supports multiple 
thread
 Key: FLINK-33814
 URL: https://issues.apache.org/jira/browse/FLINK-33814
 Project: Flink
  Issue Type: Sub-task
  Components: Autoscaler
Reporter: Rui Fan
Assignee: Rui Fan
 Fix For: kubernetes-operator-1.8.0


When the job list has a lot of jobs, single thread isn't enough.

So Autoscaler Standalone control loop supports multiple thread is very useful 
for massive production, it's similar to 
kubernetes.operator.reconcile.parallelism.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)