Re: [ANNOUNCE] New PMC member: Yuan Mei

[Yuxin Tan]

Congratulations, Yuan!

Best,
Yuxin

Yang Wang  于2022年3月15日周二 15:00写道：

> Congratulations, Yuan!
>
> Best,
> Yang
>
> Lincoln Lee  于2022年3月15日周二 13:32写道：
>
> > Congratulations, Yuan!
> >
> > Best,
> > Lincoln Lee
> >
> >
> > godfrey he  于2022年3月15日周二 09:53写道：
> >
> > > Congratulations, Yuan!
> > >
> > > Best,
> > > Godfrey
> > >
> > > Lijie Wang  于2022年3月15日周二 09:18写道：
> > > >
> > > > Congratulations, Yuan!
> > > >
> > > > Best,
> > > > Lijie
> > > >
> > > > Benchao Li  于2022年3月15日周二 08:18写道：
> > > >
> > > > > Congratulations, Yuan!
> > > > >
> > > > > Yun Gao  于2022年3月15日周二 01:37写道：
> > > > >
> > > > > > Congratulations, Yuan!
> > > > > >
> > > > > > Best,
> > > > > > Yun Gao
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> --
> > > > > > From:Francesco Guardiani 
> > > > > > Send Time:2022 Mar. 15 (Tue.) 00:21
> > > > > > To:dev 
> > > > > > Subject:Re: [ANNOUNCE] New PMC member: Yuan Mei
> > > > > >
> > > > > > Congratulations, Yuan!
> > > > > >
> > > > > > On Mon, Mar 14, 2022 at 3:51 PM yanfei lei 
> > > wrote:
> > > > > >
> > > > > > > Congratulations, Yuan!
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Zhilong Hong  于2022年3月14日周一 19:31写道：
> > > > > > >
> > > > > > > > Congratulations, Yuan!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Zhilong
> > > > > > > >
> > > > > > > > On Mon, Mar 14, 2022 at 7:22 PM Konstantin Knauf <
> > > kna...@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Congratulations, Yuan!
> > > > > > > > >
> > > > > > > > > On Mon, Mar 14, 2022 at 11:29 AM Jing Zhang <
> > > beyond1...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Congratulations, Yuan!
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Jing Zhang
> > > > > > > > > >
> > > > > > > > > > Jing Ge  于2022年3月14日周一 18:15写道：
> > > > > > > > > >
> > > > > > > > > > > Congrats! Very well deserved!
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Jing
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Mar 14, 2022 at 10:34 AM Piotr Nowojski <
> > > > > > > > pnowoj...@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Congratulations :)
> > > > > > > > > > > >
> > > > > > > > > > > > pon., 14 mar 2022 o 09:59 Yun Tang  >
> > > > > > > napisał(a):
> > > > > > > > > > > >
> > > > > > > > > > > > > Congratulations, Yuan!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Yun Tang
> > > > > > > > > > > > > 
> > > > > > > > > > > > > From: Zakelly Lan 
> > > > > > > > > > > > > Sent: Monday, March 14, 2022 16:55
> > > > > > > > > > > > > To: dev@flink.apache.org 
> > > > > > > > > > > > > Subject: Re: [ANNOUNCE] New PMC member: Yuan Mei
> > > > > > > > > > > > >
> > > > > > > > > > > > > Congratulations, Yuan!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Zakelly
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Mar 14, 2022 at 4:49 PM Johannes Moser <
> > > > > > > > j...@ververica.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Congrats Yuan.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On 14.03.2022, at 09:45, Arvid Heise <
> > > ar...@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Congratulations and well deserved!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, Mar 14, 2022 at 9:30 AM Matthias Pohl <
> > > > > > > > > map...@apache.org
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >> Congratulations, Yuan.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> On Mon, Mar 14, 2022 at 9:25 AM Shuo Cheng <
> > > > > > > > > njucs...@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >>> Congratulations, Yuan!
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> On Mon, Mar 14, 2022 at 4:22 PM Anton
> > > Kalashnikov <
> > > > > > > > > > > > > kaa@yandex.com>
> > > > > > > > > > > > > > >>> wrote:
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >  Congratulations, Yuan!
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > >  --
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > >  Best regards,
> > > > > > > > > > > > > >  Anton Kalashnikov
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > >  14.03.2022 09:13, Leonard Xu пишет:
> > > > > > > > > > > > > > > Congratulations Yuan!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > Leonard
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >> 2022年3月14日 下午4:09，Yangze Guo <
> > > karma...@gmail.com>
> > > > > > 写道：
> > > > > > > > > > > > > > >>
> > > > > > > > > > > 

Re: [ANNOUNCE] New Apache Flink Committer - Lijie Wang

[Yuxin Tan]

Congratulations, Lijie!

Best,
Yuxin


Guowei Ma  于2022年8月17日周三 18:42写道：

> Congratulations, Lijie. Welcome on board~!
> Best,
> Guowei
>
>
> On Wed, Aug 17, 2022 at 6:25 PM Zhu Zhu  wrote:
>
> > Hi everyone,
> >
> > On behalf of the PMC, I'm very happy to announce Lijie Wang as
> > a new Flink committer.
> >
> > Lijie has been contributing to Flink project for more than 2 years.
> > He mainly works on the runtime/coordination part, doing feature
> > development, problem debugging and code reviews. He has also
> > driven the work of FLIP-187(Adaptive Batch Scheduler) and
> > FLIP-224(Blocklist for Speculative Execution), which are important
> > to run batch jobs.
> >
> > Please join me in congratulating Lijie for becoming a Flink committer!
> >
> > Cheers,
> > Zhu
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Caizhi Weng

[Yuxin Tan]

Caizhi, Congratulations!

Best,
Yuxin


Jane Chan  于2022年9月9日周五 10:09写道：

> Congrats Caizhi
>
> Best,
> Jane
>
> On Fri, Sep 9, 2022 at 9:58 AM Xingbo Huang  wrote:
>
> > Congratulations Caizhi
> >
> > Best,
> > Xingbo
> >
> > Jingsong Lee  于2022年9月9日周五 09:41写道：
> >
> > > Hi everyone,
> > >
> > > On behalf of the PMC, I'm very happy to announce Caizhi Weng as a new
> > > Flink committer.
> > >
> > > Caizhi has been contributing to the Flink project for more than 3
> > > years, and has authored 150+ PRs. He is one of the key driving members
> > > of the Flink Table Store subproject. He is responsible for the core
> > > design of transaction committing. Expanded the Hive ecosystem of Flink
> > > Table Store. He also works in Flink SQL, helps solve the problems of
> > > ease of use and performance.
> > >
> > > Please join me in congratulating Caizhi for becoming a Flink committer!
> > >
> > > Best,
> > > Jingsong
> > >
> >
>

Re: [ANNOUNCE] Apache Flink 1.16.0 released

[Yuxin Tan]

Congrats! Glad to hear that.

Best,
Yuxin


weijie guo  于2022年10月31日周一 11:51写道：

> Congratulations, this is a version with many new features and thanks to
> everyone involved!
>
> Best regards,
>
> Weijie
>
>
> Yun Tang  于2022年10月31日周一 11:32写道：
>
> > Congratulations, and thanks to everyone involved!
> >
> >
> > Best
> > Yun Tang
> > 
> > From: Zakelly Lan 
> > Sent: Monday, October 31, 2022 10:44
> > To: dev@flink.apache.org 
> > Subject: Re: [ANNOUNCE] Apache Flink 1.16.0 released
> >
> > Good to know & Congrats!
> >
> > Thanks everyone involved!
> >
> > Best,
> > Zakelly
> >
> > On Mon, Oct 31, 2022 at 10:40 AM Becket Qin 
> wrote:
> > >
> > > Hooray!! Congratulations to the team!
> > >
> > > Cheers,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Mon, Oct 31, 2022 at 9:57 AM Hang Ruan 
> > wrote:
> > >
> > > > Congratulations!
> > > >
> > > > Best,
> > > > Hang
> > > >
> > > > Shengkai Fang  于2022年10月31日周一 09:40写道：
> > > >
> > > > > Congratulations!
> > > > >
> > > > > Best,
> > > > > Shengkai
> > > > >
> > > > > Hangxiang Yu  于2022年10月31日周一 09:38写道：
> > > > >
> > > > > > Congratulations!
> > > > > > Thanks Chesnay, Martijn, Godfrey & Xingbo for managing the
> release.
> > > > > >
> > > > > > On Fri, Oct 28, 2022 at 7:35 PM Jing Ge 
> > wrote:
> > > > > >
> > > > > > > Congrats!
> > > > > > >
> > > > > > > On Fri, Oct 28, 2022 at 1:22 PM 任庆盛 
> wrote:
> > > > > > >
> > > > > > >> Congratulations and a big thanks to Chesnay, Martijn, Godfrey
> > and
> > > > > Xingbo
> > > > > > >> for the awesome work for 1.16!
> > > > > > >>
> > > > > > >> Best regards,
> > > > > > >> Qingsheng Ren
> > > > > > >>
> > > > > > >> > On Oct 28, 2022, at 14:46, Xingbo Huang 
> > wrote:
> > > > > > >> >
> > > > > > >> > The Apache Flink community is very happy to announce the
> > release
> > > > of
> > > > > > >> Apache
> > > > > > >> > Flink 1.16.0, which is the first release for the Apache
> Flink
> > 1.16
> > > > > > >> series.
> > > > > > >> >
> > > > > > >> > Apache Flink® is an open-source stream processing framework
> > for
> > > > > > >> > distributed, high-performing, always-available, and accurate
> > data
> > > > > > >> streaming
> > > > > > >> > applications.
> > > > > > >> >
> > > > > > >> > The release is available for download at:
> > > > > > >> > https://flink.apache.org/downloads.html
> > > > > > >> >
> > > > > > >> > Please check out the release blog post for an overview of
> the
> > > > > > >> > improvements for this release:
> > > > > > >> >
> > https://flink.apache.org/news/2022/10/28/1.16-announcement.html
> > > > > > >> >
> > > > > > >> > The full release notes are available in Jira:
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351275
> > > > > > >> >
> > > > > > >> > We would like to thank all contributors of the Apache Flink
> > > > > community
> > > > > > >> > who made this release possible!
> > > > > > >> >
> > > > > > >> > Regards,
> > > > > > >> > Chesnay, Martijn, Godfrey & Xingbo
> > > > > > >>
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > Best,
> > > > > > Hangxiang.
> > > > > >
> > > > >
> > > >
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer

[Yuxin Tan]

Congrats, Danny!

Best,
Yuxin


Guowei Ma  于2022年11月1日周二 16:36写道：

> Congratulations Danny!
> Best,
> Guowei
>
>
> On Tue, Nov 1, 2022 at 2:20 PM weijie guo 
> wrote:
>
> > Congratulations Danny!
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > Maximilian Michels  于2022年10月13日周四 21:41写道：
> >
> > > Congratulations Danny! Well deserved :)
> > >
> > > -Max
> > >
> > > On Thu, Oct 13, 2022 at 2:40 PM Yang Wang 
> wrote:
> > >
> > > > Congratulations Danny!
> > > >
> > > > Best,
> > > > Yang
> > > >
> > > > Hang Ruan  于2022年10月13日周四 10:58写道：
> > > >
> > > > > Congratulations Danny!
> > > > >
> > > > > Best,
> > > > > Hang
> > > > >
> > > > > Yun Gao  于2022年10月13日周四 10:56写道：
> > > > >
> > > > > > Congratulations Danny!
> > > > > > Best,
> > > > > > Yun Gao
> > > > > >
> --
> > > > > > From:yuxia 
> > > > > > Send Time:2022 Oct. 12 (Wed.) 09:49
> > > > > > To:dev 
> > > > > > Subject:Re: [ANNOUNCE] New Apache Flink PMC Member - Danny
> Cranmer
> > > > > > Congratulations Danny!
> > > > > > Best regards,
> > > > > > Yuxia
> > > > > > - 原始邮件 -
> > > > > > 发件人: "Xingbo Huang" 
> > > > > > 收件人: "dev" 
> > > > > > 发送时间: 星期三, 2022年 10 月 12日 上午 9:44:22
> > > > > > 主题: Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer
> > > > > > Congratulations Danny!
> > > > > > Best,
> > > > > > Xingbo
> > > > > > Sergey Nuyanzin  于2022年10月12日周三 01:26写道：
> > > > > > > Congratulations, Danny
> > > > > > >
> > > > > > > On Tue, Oct 11, 2022, 15:18 Lincoln Lee <
> lincoln.8...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Congratulations Danny!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Lincoln Lee
> > > > > > > >
> > > > > > > >
> > > > > > > > Congxian Qiu  于2022年10月11日周二
> 19:42写道：
> > > > > > > >
> > > > > > > > > Congratulations Danny!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Congxian
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Leonard Xu  于2022年10月11日周二 18:03写道：
> > > > > > > > >
> > > > > > > > > > Congratulations Danny!
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Leonard
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Need right

[Yuxin Tan]

Hi, Stan

You could get more contribution details here. [1]
Thanks and looking forward to your contribution!

[1] https://flink.apache.org/contributing/how-to-contribute.html

Best,
Yuxin


Martijn Visser  于2022年12月5日周一 14:21写道：

> Hi,
>
> There is no need for additional permissions: you can start working on a
> Jira issue of your liking (feel free to ping me to get it assigned to you)
> and open up a PR.
>
> Thanks and looking forward to your contribution!
>
> Best regards,
>
> Martijn
>
> On Mon, Dec 5, 2022 at 6:59 AM Stan1005 <532338...@qq.com.invalid> wrote:
>
> > Hi, I want to contribute to Apache Flink. Would you please give me the
> > contributor permission? My JIRA ID is StarBoy1005.
>

[DISCUSS] FLIP-266: Simplify network memory configurations for TaskManager

[Yuxin Tan]

Hi, devs,

I'd like to start a discussion about FLIP-266: Simplify network memory
configurations for TaskManager[1].

When using Flink, users may encounter the following issues that affect
usability.
1. The job may fail with an "Insufficient number of network buffers"
exception.
2. Flink network memory size adjustment is complex.
When encountering these issues, users can solve some problems by adding or
adjusting parameters. However, multiple memory config options should be
changed. The config option adjustment requires understanding the detailed
internal implementation, which is impractical for most users.

To simplify network memory configurations for TaskManager and improve Flink
usability, this FLIP proposed some optimization solutions for the issues.

Looking forward to your feedback.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-266%3A+Simplify+network+memory+configurations+for+TaskManager

Best regards,
Yuxin

Re: [VOTE] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

[Yuxin Tan]

+1 (non-binding)

Best,
Yuxin


Yangze Guo  于2023年6月25日周日 12:21写道：

> +1 (binding)
>
> Best,
> Yangze Guo
>
> On Sun, Jun 25, 2023 at 11:41 AM Jark Wu  wrote:
> >
> > +1 (binding)
> >
> > Best,
> > Jark
> >
> > > 2023年6月25日 10:04，Xia Sun  写道：
> > >
> > > +1 (non-binding)
> > >
> > > Best Regards,
> > >
> > > Xia
> > >
> > > yuxia  于2023年6月25日周日 09:23写道：
> > >
> > >> +1 (binding)
> > >> Thanks Lijie driving it.
> > >>
> > >> Best regards,
> > >> Yuxia
> > >>
> > >> - 原始邮件 -
> > >> 发件人: "Yuepeng Pan" 
> > >> 收件人: "dev" 
> > >> 发送时间: 星期六, 2023年 6 月 24日 下午 9:06:53
> > >> 主题: Re:[VOTE] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs
> > >>
> > >> +1 (non-binding)
> > >>
> > >> Thanks,
> > >> Yuepeng Pan
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> At 2023-06-23 23:49:53, "Lijie Wang" 
> wrote:
> > >>> Hi all,
> > >>>
> > >>> Thanks for all the feedback about the FLIP-324: Introduce Runtime
> Filter
> > >>> for Flink Batch Jobs[1]. This FLIP was discussed in [2].
> > >>>
> > >>> I'd like to start a vote for it. The vote will be open for at least
> 72
> > >>> hours (until June 29th 12:00 GMT) unless there is an objection or
> > >>> insufficient votes.
> > >>>
> > >>> [1]
> > >>>
> > >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-324%3A+Introduce+Runtime+Filter+for+Flink+Batch+Jobs
> > >>> [2] https://lists.apache.org/thread/mm0o8fv7x7k13z11htt88zhy7lo8npmg
> > >>>
> > >>> Best,
> > >>> Lijie
> > >>
> >
>

Re: Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

[Yuxin Tan]

Congratulations!

Best,
Yuxin


Dunn Bangui  于2023年7月4日周二 16:04写道：

> Congratulations!
>
> Best,
> Bangui Dunn
>
> Yangze Guo  于2023年7月4日周二 15:59写道：
>
> > Congrats everyone!
> >
> > Best,
> > Yangze Guo
> >
> > On Tue, Jul 4, 2023 at 3:53 PM Rui Fan <1996fan...@gmail.com> wrote:
> > >
> > > Congratulations!
> > >
> > > Best,
> > > Rui Fan
> > >
> > > On Tue, Jul 4, 2023 at 2:08 PM Zhu Zhu  wrote:
> > >
> > > > Congratulations everyone!
> > > >
> > > > Thanks,
> > > > Zhu
> > > >
> > > > Hang Ruan  于2023年7月4日周二 14:06写道：
> > > > >
> > > > > Congratulations!
> > > > >
> > > > > Best,
> > > > > Hang
> > > > >
> > > > > Jingsong Li  于2023年7月4日周二 13:47写道：
> > > > >
> > > > > > Congratulations!
> > > > > >
> > > > > > Thank you! All of the Flink community!
> > > > > >
> > > > > > Best,
> > > > > > Jingsong
> > > > > >
> > > > > > On Tue, Jul 4, 2023 at 1:24 PM tison 
> wrote:
> > > > > > >
> > > > > > > Congrats and with honor :D
> > > > > > >
> > > > > > > Best,
> > > > > > > tison.
> > > > > > >
> > > > > > >
> > > > > > > Mang Zhang  于2023年7月4日周二 11:08写道：
> > > > > > >
> > > > > > > > Congratulations!--
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Mang Zhang
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > 在 2023-07-04 01:53:46，"liu ron"  写道：
> > > > > > > > >Congrats everyone
> > > > > > > > >
> > > > > > > > >Best,
> > > > > > > > >Ron
> > > > > > > > >
> > > > > > > > >Jark Wu  于2023年7月3日周一 22:48写道：
> > > > > > > > >
> > > > > > > > >> Congrats everyone!
> > > > > > > > >>
> > > > > > > > >> Best,
> > > > > > > > >> Jark
> > > > > > > > >>
> > > > > > > > >> > 2023年7月3日 22:37，Yuval Itzchakov  写道：
> > > > > > > > >> >
> > > > > > > > >> > Congrats team!
> > > > > > > > >> >
> > > > > > > > >> > On Mon, Jul 3, 2023, 17:28 Jing Ge via user <
> > > > > > u...@flink.apache.org
> > > > > > > > >> > wrote:
> > > > > > > > >> >> Congratulations!
> > > > > > > > >> >>
> > > > > > > > >> >> Best regards,
> > > > > > > > >> >> Jing
> > > > > > > > >> >>
> > > > > > > > >> >>
> > > > > > > > >> >> On Mon, Jul 3, 2023 at 3:21 PM yuxia <
> > > > > > luoyu...@alumni.sjtu.edu.cn
> > > > > > > > >> > wrote:
> > > > > > > > >> >>> Congratulations!
> > > > > > > > >> >>>
> > > > > > > > >> >>> Best regards,
> > > > > > > > >> >>> Yuxia
> > > > > > > > >> >>>
> > > > > > > > >> >>> 发件人: "Pushpa Ramakrishnan" <
> > pushpa.ramakrish...@icloud.com
> > > > > >  > > > > > > > >> pushpa.ramakrish...@icloud.com>>
> > > > > > > > >> >>> 收件人: "Xintong Song"  > > > > > > > >> tonysong...@gmail.com>>
> > > > > > > > >> >>> 抄送: "dev"  > > > dev@flink.apache.org>>,
> > > > > > > > >> "User"  u...@flink.apache.org
> > >>
> > > > > > > > >> >>> 发送时间: 星期一, 2023年 7 月 03日 下午 8:36:30
> > > > > > > > >> >>> 主题: Re: [ANNOUNCE] Apache Flink has won the 2023
> SIGMOD
> > > > Systems
> > > > > > > > Award
> > > > > > > > >> >>>
> > > > > > > > >> >>> Congratulations \uD83E\uDD73
> > > > > > > > >> >>>
> > > > > > > > >> >>> On 03-Jul-2023, at 3:30 PM, Xintong Song <
> > > > tonysong...@gmail.com
> > > > > > > > >> > wrote:
> > > > > > > > >> >>>
> > > > > > > > >> >>> 
> > > > > > > > >> >>> Dear Community,
> > > > > > > > >> >>>
> > > > > > > > >> >>> I'm pleased to share this good news with everyone. As
> > some
> > > > of
> > > > > > you
> > > > > > > > may
> > > > > > > > >> have already heard, Apache Flink has won the 2023 SIGMOD
> > Systems
> > > > > > Award
> > > > > > > > [1].
> > > > > > > > >> >>>
> > > > > > > > >> >>> "Apache Flink greatly expanded the use of stream
> > > > > > data-processing."
> > > > > > > > --
> > > > > > > > >> SIGMOD Awards Committee
> > > > > > > > >> >>>
> > > > > > > > >> >>> SIGMOD is one of the most influential data management
> > > > research
> > > > > > > > >> conferences in the world. The Systems Award is awarded to
> an
> > > > > > individual
> > > > > > > > or
> > > > > > > > >> set of individuals to recognize the development of a
> > software or
> > > > > > > > hardware
> > > > > > > > >> system whose technical contributions have had significant
> > > > impact on
> > > > > > the
> > > > > > > > >> theory or practice of large-scale data management systems.
> > > > Winning
> > > > > > of
> > > > > > > > the
> > > > > > > > >> award indicates the high recognition of Flink's
> > technological
> > > > > > > > advancement
> > > > > > > > >> and industry influence from academia.
> > > > > > > > >> >>>
> > > > > > > > >> >>> As an open-source project, Flink wouldn't have come
> > this far
> > > > > > without
> > > > > > > > >> the wide, active and supportive community behind it. Kudos
> > to
> > > > all
> > > > > > of us
> > > > > > > > who
> > > > > > > > >> helped make this happen, including the over 1,400
> > contributors
> > > > and
> > > > > > many
> > > > > > > > >> others who contributed in ways beyond code.
> > > > > > > > >> >>>
> > > > > > > > >

Re: [DISCUSS][2.0] FLIP-343: Remove parameter in WindowAssigner#getDefaultTrigger()

[Yuxin Tan]

+1

Best,
Yuxin


Jing Ge  于2023年7月21日周五 15:41写道：

> +1
>
> NIT: the release in the FLIP is still empty, it should be 2.0
>
> Best regards,
> Jing
>
> On Fri, Jul 21, 2023 at 6:03 AM Xintong Song 
> wrote:
>
> > +1
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Fri, Jul 21, 2023 at 10:53 AM Wencong Liu 
> wrote:
> >
> > > Hi devs,
> > >
> > > I would like to start a discussion on FLIP-343: Remove parameter in
> > > WindowAssigner#getDefaultTrigger() [1].
> > >
> > >
> > > The method getDefaultTrigger() in WindowAssigner takes a
> > > StreamExecutionEnvironment
> > > parameter, but this parameter is not actually used for any subclasses
> of
> > > WindowAssigner.
> > > Therefore, it is unnecessary to include this parameter.
> > > As such I propose to remove the StreamExecutionEnvironment field from
> > > WindowAssigner#getDefaultTrigger(StreamExecutionEnvironment env).
> > > Looking forward to your feedback.
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263425229
> > > Best regards,
> > >
> > >
> > > Wencong Liu
> >
>

Re: [DISCUSS][2.0] FLIP-347: Remove IOReadableWritable serialization in Path

[Yuxin Tan]

+1

Best,
Yuxin


Xintong Song  于2023年7月21日周五 12:04写道：

> +1
>
> Best,
>
> Xintong
>
>
>
> On Fri, Jul 21, 2023 at 10:54 AM Wencong Liu  wrote:
>
> > Hi devs,
> >
> > I would like to start a discussion on FLIP-347: Remove IOReadableWritable
> > serialization in Path [1].
> >
> >
> > The Path class is currently mutable to support IOReadableWritable
> > serialization. However, many parts
> > of the code assume that the Path is immutable. By making the Path class
> > immutable, we can ensure
> > that paths are stored correctly without the possibility of mutation and
> > eliminate the occurrence of subtle errors.
> > As such I propose to modify the Path class to no longer implement the
> > IOReadableWritable interface.
> > Looking forward to your feedback.
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-347%3A+Remove+IOReadableWritable+serialization+in+Path
> > Best regards,
> >
> >
> > Wencong Liu
>

Re: [DISCUSS][2.0] FLIP-344: Remove parameter in RichFunction#open

[Yuxin Tan]

+1

Best,
Yuxin


Xintong Song  于2023年7月21日周五 12:04写道：

> +1
>
> Best,
>
> Xintong
>
>
>
> On Fri, Jul 21, 2023 at 10:52 AM Wencong Liu  wrote:
>
> > Hi devs,
> >
> > I would like to start a discussion on FLIP-344: Remove parameter in
> > RichFunction#open [1].
> >
> > The open() method in RichFunction requires a Configuration instance as an
> > argument,
> > which is always passed as a new instance without any configuration
> > parameters in
> > AbstractUdfStreamOperator#open. Thus, it is unnecessary to include this
> > parameter
> > in the open() method.
> > As such I propose to remove the Configuration field from
> > RichFunction#open(Configuration parameters).
> > Looking forward to your feedback.
> > [1]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263425231
> > Best regards,
> >
> >
> > Wencong Liu
>

Re: Re: [ANNOUNCE] New Apache Flink Committer - Yong Fang

[Yuxin Tan]

Congratulations, Yong Fang!

Best,
Yuxin


Yanfei Lei  于2023年7月26日周三 10:18写道：

> Congratulations!
>
> Best regards,
> Yanfei
>
> weijie guo  于2023年7月26日周三 10:10写道：
> >
> > Congrats, Yong Fang!
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > Danny Cranmer  于2023年7月26日周三 03:34写道：
> >
> > > Congrats and welcome!
> > >
> > > Danny.
> > >
> > > On Tue, 25 Jul 2023, 16:48 Matthias Pohl,  .invalid>
> > > wrote:
> > >
> > > > Congratulations :)
> > > >
> > > > On Tue, Jul 25, 2023 at 5:13 PM Jing Ge 
> > > > wrote:
> > > >
> > > > > Congrats, Yong Fang!
> > > > >
> > > > > Best regards,
> > > > > Jing
> > > > >
> > > > > On Tue, Jul 25, 2023 at 7:35 PM Yu Li  wrote:
> > > > >
> > > > > > Congrats, Yong!
> > > > > >
> > > > > > Best Regards,
> > > > > > Yu
> > > > > >
> > > > > >
> > > > > > On Tue, 25 Jul 2023 at 18:03, Sergey Nuyanzin <
> snuyan...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Congratulations, Yong Fang!
> > > > > > >
> > > > > > > On Tue, Jul 25, 2023 at 7:53 AM ConradJam  >
> > > > wrote:
> > > > > > >
> > > > > > > > Congratulations, Yong Fang
> > > > > > > >
> > > > > > > > Mang Zhang  于2023年7月25日周二 12:08写道：
> > > > > > > >
> > > > > > > > > Congratulations, Yong Fang!
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Mang Zhang
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 在 2023-07-25 10:30:24，"Jark Wu"  写道：
> > > > > > > > > >Congratulations, Yong Fang!
> > > > > > > > > >
> > > > > > > > > >Best,
> > > > > > > > > >Jark
> > > > > > > > > >
> > > > > > > > > >On Mon, 24 Jul 2023 at 22:11, Wencong Liu <
> > > liuwencle...@163.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Congratulations!
> > > > > > > > > >>
> > > > > > > > > >> Best,
> > > > > > > > > >> Wencong Liu
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> 在 2023-07-24 11:03:30，"Paul Lam"  >
> > > 写道：
> > > > > > > > > >> >Congrats, Shammon!
> > > > > > > > > >> >
> > > > > > > > > >> >Best,
> > > > > > > > > >> >Paul Lam
> > > > > > > > > >> >
> > > > > > > > > >> >> 2023年7月24日 10:56，Jingsong Li  >
> > > 写道：
> > > > > > > > > >> >>
> > > > > > > > > >> >> Shammon
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Best
> > > > > > > >
> > > > > > > > ConradJam
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best regards,
> > > > > > > Sergey
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

Re: [VOTE][2.0] FLIP-343: Remove parameter in WindowAssigner#getDefaultTrigger()

[Yuxin Tan]

+1 (non-binding)

Best,
Yuxin


Xintong Song  于2023年7月26日周三 16:08写道：

> +1 (binding)
>
> Best,
>
> Xintong
>
>
>
> On Wed, Jul 26, 2023 at 3:35 PM Yuepeng Pan  wrote:
>
> > +1 (non-binding)
> >
> > Thanks.
> >
> > Yuepeng Pan.
> > At 2023-07-26 14:26:04, "Wencong Liu"  wrote:
> > >Hi dev,
> > >
> > >
> > >I'd like to start a vote on FLIP-343.
> > >
> > >
> > >Discussion thread:
> > https://lists.apache.org/thread/zn11f460x70nn7f2ckqph41bvx416wxc
> > >FLIP:
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263425229
> > >
> > >
> > >Best regards,
> > >Wencong Liu
> >
>

Re: [VOTE][2.0] FLIP-344: Remove parameter in RichFunction#open

[Yuxin Tan]

+1 (non-binding)

Best,
Yuxin


Xintong Song  于2023年7月26日周三 16:09写道：

> +1 (binding)
>
> Best,
>
> Xintong
>
>
>
> On Wed, Jul 26, 2023 at 2:26 PM Wencong Liu  wrote:
>
> > Hi dev,
> >
> >
> > I'd like to start a vote on FLIP-344.
> >
> >
> > Discussion thread:
> > https://lists.apache.org/thread/5lyjrrdtwkngkol2t541r4xwoh7133km
> > FLIP:
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263425231
> >
> >
> > Best regards,
> > Wencong Liu
>

Re: [VOTE][2.0] FLIP-347: Remove IOReadableWritable serialization in Path

[Yuxin Tan]

+1 (non-binding)

Best,
Yuxin


Xintong Song  于2023年7月26日周三 16:09写道：

> +1 (binding)
>
> Best,
>
> Xintong
>
>
>
> On Wed, Jul 26, 2023 at 2:29 PM Wencong Liu  wrote:
>
> > Hi dev,
> >
> >
> > I'd like to start a vote on FLIP-347.
> >
> >
> > Discussion thread:
> > https://lists.apache.org/thread/3gcxhnqpsvb85golnlxf9tv5p43xkjgj
> > FLIP:
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-347%3A+Remove+IOReadableWritable+serialization+in+Path
> >
> >
> > Best regards,
> > Wencong Liu
>

Re: [ANNOUNCE] New Apache Flink Committer - Weihua Hu

[Yuxin Tan]

Congratulations Weihua!

Best,
Yuxin


Junrui Lee  于2023年8月4日周五 14:28写道：

> Congrats, Weihua!
> Best,
> Junrui
>
> Geng Biao  于2023年8月4日周五 14:25写道：
>
> > Congrats, Weihua!
> > Best,
> > Biao Geng
> >
> > 发送自 Outlook for iOS
> > 
> > 发件人: 周仁祥 
> > 发送时间: Friday, August 4, 2023 2:23:42 PM
> > 收件人: dev@flink.apache.org 
> > 抄送: Weihua Hu 
> > 主题: Re: [ANNOUNCE] New Apache Flink Committer - Weihua Hu
> >
> > Congratulations, Weihua~
> >
> > > 2023年8月4日 14:21，Sergey Nuyanzin  写道：
> > >
> > > Congratulations, Weihua!
> > >
> > > On Fri, Aug 4, 2023 at 8:03 AM Chen Zhanghao <
> zhanghao.c...@outlook.com>
> > > wrote:
> > >
> > >> Congratulations, Weihua!
> > >>
> > >> Best,
> > >> Zhanghao Chen
> > >> 
> > >> 发件人: Xintong Song 
> > >> 发送时间: 2023年8月4日 11:18
> > >> 收件人: dev 
> > >> 抄送: Weihua Hu 
> > >> 主题: [ANNOUNCE] New Apache Flink Committer - Weihua Hu
> > >>
> > >> Hi everyone,
> > >>
> > >> On behalf of the PMC, I'm very happy to announce Weihua Hu as a new
> > Flink
> > >> Committer!
> > >>
> > >> Weihua has been consistently contributing to the project since May
> > 2022. He
> > >> mainly works in Flink's distributed coordination areas. He is the main
> > >> contributor of FLIP-298 and many other improvements in large-scale job
> > >> scheduling and improvements. He is also quite active in mailing lists,
> > >> participating discussions and answering user questions.
> > >>
> > >> Please join me in congratulating Weihua!
> > >>
> > >> Best,
> > >>
> > >> Xintong (on behalf of the Apache Flink PMC)
> > >>
> > >
> > >
> > > --
> > > Best regards,
> > > Sergey
> >
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Matthias Pohl

[Yuxin Tan]

Congratulations, Matthias!

Best,
Yuxin


Sergey Nuyanzin  于2023年8月4日周五 14:21写道：

> Congratulations, Matthias!
> Well deserved!
>
> On Fri, Aug 4, 2023 at 7:59 AM liu ron  wrote:
>
> > Congrats, Matthias!
> >
> > Best,
> > Ron
> >
> > Shammon FY  于2023年8月4日周五 13:24写道：
> >
> > > Congratulations, Matthias!
> > >
> > > On Fri, Aug 4, 2023 at 1:13 PM Samrat Deb 
> wrote:
> > >
> > > > Congrats, Matthias!
> > > >
> > > >
> > > > On Fri, 4 Aug 2023 at 10:13 AM, Benchao Li 
> > wrote:
> > > >
> > > > > Congratulations, Matthias!
> > > > >
> > > > > Jing Ge  于2023年8月4日周五 12:35写道：
> > > > >
> > > > > > Congrats! Matthias!
> > > > > >
> > > > > > Best regards,
> > > > > > Jing
> > > > > >
> > > > > > On Fri, Aug 4, 2023 at 12:09 PM Yangze Guo 
> > > wrote:
> > > > > >
> > > > > > > Congrats, Matthias!
> > > > > > >
> > > > > > > Best,
> > > > > > > Yangze Guo
> > > > > > >
> > > > > > > On Fri, Aug 4, 2023 at 11:44 AM Qingsheng Ren <
> re...@apache.org>
> > > > > wrote:
> > > > > > > >
> > > > > > > > Congratulations, Matthias! This is absolutely well deserved.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Qingsheng
> > > > > > > >
> > > > > > > > On Fri, Aug 4, 2023 at 11:31 AM Rui Fan <
> 1996fan...@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Congratulations Matthias, well deserved!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Rui Fan
> > > > > > > > >
> > > > > > > > > On Fri, Aug 4, 2023 at 11:30 AM Leonard Xu <
> > xbjt...@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Congratulations,  Matthias.
> > > > > > > > > >
> > > > > > > > > > Well deserved ^_^
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Leonard
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > On Aug 4, 2023, at 11:18 AM, Xintong Song <
> > > > > tonysong...@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi everyone,
> > > > > > > > > > >
> > > > > > > > > > > On behalf of the PMC, I'm very happy to announce that
> > > > Matthias
> > > > > > > Pohl has
> > > > > > > > > > > joined the Flink PMC!
> > > > > > > > > > >
> > > > > > > > > > > Matthias has been consistently contributing to the
> > project
> > > > > since
> > > > > > > Sep
> > > > > > > > > > 2020,
> > > > > > > > > > > and became a committer in Dec 2021. He mainly works in
> > > > Flink's
> > > > > > > > > > distributed
> > > > > > > > > > > coordination and high availability areas. He has worked
> > on
> > > > many
> > > > > > > FLIPs
> > > > > > > > > > > including FLIP195/270/285. He helped a lot with the
> > release
> > > > > > > management,
> > > > > > > > > > > being one of the Flink 1.17 release managers and also
> > very
> > > > > active
> > > > > > > in
> > > > > > > > > > Flink
> > > > > > > > > > > 1.18 / 2.0 efforts. He also contributed a lot to
> > improving
> > > > the
> > > > > > > build
> > > > > > > > > > > stability.
> > > > > > > > > > >
> > > > > > > > > > > Please join me in congratulating Matthias!
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > >
> > > > > > > > > > > Xintong (on behalf of the Apache Flink PMC)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Best,
> > > > > Benchao Li
> > > > >
> > > >
> > >
> >
>
>
> --
> Best regards,
> Sergey
>

Re: [ANNOUNCE] New Apache Flink Committer - Hong Teoh

[Yuxin Tan]

Congratulations, Hong!

Best,
Yuxin


Sergey Nuyanzin  于2023年8月4日周五 14:24写道：

> Congratulations, Hong!
>
> On Fri, Aug 4, 2023 at 7:25 AM Shammon FY  wrote:
>
> > Congratulations, Hong!
> >
> > Best,
> > Shammon FY
> >
> > On Fri, Aug 4, 2023 at 12:33 PM Jing Ge 
> > wrote:
> >
> > > congrats! Hong!
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Fri, Aug 4, 2023 at 11:48 AM Qingsheng Ren 
> > wrote:
> > >
> > > > Congratulations and welcome aboard, Hong!
> > > >
> > > > Best,
> > > > Qingsheng
> > > >
> > > > On Fri, Aug 4, 2023 at 11:04 AM Matt Wang  wrote:
> > > >
> > > > > Congratulations, Hong!
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Best,
> > > > > Matt Wang
> > > > >
> > > > >
> > > > >  Replied Message 
> > > > > | From | Weihua Hu |
> > > > > | Date | 08/4/2023 10:55 |
> > > > > | To |  |
> > > > > | Subject | Re: [ANNOUNCE] New Apache Flink Committer - Hong Teoh |
> > > > > Congratulations, Hong!
> > > > >
> > > > > Best,
> > > > > Weihua
> > > > >
> > > > >
> > > > > On Fri, Aug 4, 2023 at 10:49 AM Samrat Deb 
> > > > wrote:
> > > > >
> > > > > Congratulations, Hong Teoh
> > > > >
> > > > > On Fri, 4 Aug 2023 at 7:52 AM, Benchao Li 
> > > wrote:
> > > > >
> > > > > Congratulations, Hong!
> > > > >
> > > > > yuxia  于2023年8月4日周五 09:23写道：
> > > > >
> > > > > Congratulations, Hong Teoh!
> > > > >
> > > > > Best regards,
> > > > > Yuxia
> > > > >
> > > > > - 原始邮件 -
> > > > > 发件人: "Matthias Pohl" 
> > > > > 收件人: "dev" 
> > > > > 发送时间: 星期四, 2023年 8 月 03日 下午 11:24:39
> > > > > 主题: Re: [ANNOUNCE] New Apache Flink Committer - Hong Teoh
> > > > >
> > > > > Congratulations, Hong! :)
> > > > >
> > > > > On Thu, Aug 3, 2023 at 3:39 PM Leonard Xu 
> wrote:
> > > > >
> > > > > Congratulations, Hong!
> > > > >
> > > > >
> > > > > Best,
> > > > > Leonard
> > > > >
> > > > > On Aug 3, 2023, at 8:45 PM, Jiabao Sun  > > > > .INVALID>
> > > > > wrote:
> > > > >
> > > > > Congratulations, Hong Teoh!
> > > > >
> > > > > Best,
> > > > > Jiabao Sun
> > > > >
> > > > > 2023年8月3日 下午7:32，Danny Cranmer  写道：
> > > > >
> > > > > On behalf of the PMC, I'm very happy to announce Hong Teoh as a
> > > > > new
> > > > > Flink
> > > > > Committer.
> > > > >
> > > > > Hong has been active in the Flink community for over 1 year and
> > > > > has
> > > > > played
> > > > > a key role in developing and maintaining AWS integrations, core
> > > > > connector
> > > > > APIs and more recently, improvements to the Flink REST API.
> > > > > Additionally,
> > > > > Hong is a very active community member, supporting users and
> > > > > participating
> > > > > in discussions on the mailing lists, Flink slack channels and
> > > > > speaking
> > > > > at
> > > > > conferences.
> > > > >
> > > > > Please join me in congratulating Hong for becoming a Flink
> > > > > Committer!
> > > > >
> > > > > Thanks,
> > > > > Danny Cranmer (on behalf of the Flink PMC)
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Best,
> > > > > Benchao Li
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
> Best regards,
> Sergey
>

Re: [ANNOUNCE] New Apache Flink Committer - Yanfei Lei

[Yuxin Tan]

Congrats, Yanfei!

Best,
Yuxin


weijie guo  于2023年8月7日周一 17:59写道：

> Congrats, Yanfei!
>
> Best regards,
>
> Weijie
>
>
> Biao Geng  于2023年8月7日周一 17:03写道：
>
> > Congrats, Yanfei!
> > Best,
> > Biao Geng
> >
> > 发送自 Outlook for iOS
> > 
> > 发件人: Qingsheng Ren 
> > 发送时间: Monday, August 7, 2023 4:23:52 PM
> > 收件人: dev@flink.apache.org 
> > 主题: Re: [ANNOUNCE] New Apache Flink Committer - Yanfei Lei
> >
> > Congratulations and welcome, Yanfei!
> >
> > Best,
> > Qingsheng
> >
> > On Mon, Aug 7, 2023 at 4:19 PM Matthias Pohl  > .invalid>
> > wrote:
> >
> > > Congratulations, Yanfei! :)
> > >
> > > On Mon, Aug 7, 2023 at 10:00 AM Junrui Lee 
> wrote:
> > >
> > > > Congratulations Yanfei!
> > > >
> > > > Best,
> > > > Junrui
> > > >
> > > > Yun Tang  于2023年8月7日周一 15:19写道：
> > > >
> > > > > Congratulations, Yanfei!
> > > > >
> > > > > Best
> > > > > Yun Tang
> > > > > 
> > > > > From: Danny Cranmer 
> > > > > Sent: Monday, August 7, 2023 15:10
> > > > > To: dev 
> > > > > Subject: Re: [ANNOUNCE] New Apache Flink Committer - Yanfei Lei
> > > > >
> > > > > Congrats Yanfei! Welcome to the team.
> > > > >
> > > > > Danny
> > > > >
> > > > > On Mon, 7 Aug 2023, 08:03 Rui Fan, <1996fan...@gmail.com> wrote:
> > > > >
> > > > > > Congratulations Yanfei!
> > > > > >
> > > > > > Best,
> > > > > > Rui
> > > > > >
> > > > > > On Mon, Aug 7, 2023 at 2:56 PM Yuan Mei 
> > > > wrote:
> > > > > >
> > > > > > > On behalf of the PMC, I'm happy to announce Yanfei Lei as a new
> > > Flink
> > > > > > > Committer.
> > > > > > >
> > > > > > > Yanfei has been active in the Flink community for almost two
> > years
> > > > and
> > > > > > has
> > > > > > > played an important role in developing and maintaining State
> and
> > > > > > Checkpoint
> > > > > > > related features/components, including RocksDB Rescaling
> > > Performance
> > > > > > > Improvement and Generic Incremental Checkpoints.
> > > > > > >
> > > > > > > Yanfei also helps improve community infrastructure in many
> ways,
> > > > > > including
> > > > > > > migrating the Flink Daily performance benchmark to the Apache
> > Flink
> > > > > slack
> > > > > > > channel. She is the maintainer of the benchmark and has
> improved
> > > its
> > > > > > > detection stability significantly. She is also one of the major
> > > > > > maintainers
> > > > > > > of the FrocksDB Repo and released FRocksDB 6.20.3 (part of
> Flink
> > > 1.17
> > > > > > > release). Yanfei is a very active community member, supporting
> > > users
> > > > > and
> > > > > > > participating
> > > > > > > in tons of discussions on the mailing lists.
> > > > > > >
> > > > > > > Please join me in congratulating Yanfei for becoming a Flink
> > > > Committer!
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Yuan Mei (on behalf of the Flink PMC)
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Hangxiang Yu

[Yuxin Tan]

Congrats, Hangxiang!

Best,
Yuxin


weijie guo  于2023年8月7日周一 17:59写道：

> Congrats, Hangxiang!
>
> Best regards,
>
> Weijie
>
>
> Biao Geng  于2023年8月7日周一 17:04写道：
>
> > Congrats, Hangxiang!
> > Best,
> > Biao Geng
> >
> >
> > 发送自 Outlook for iOS
> > 
> > 发件人: Qingsheng Ren 
> > 发送时间: Monday, August 7, 2023 4:23:11 PM
> > 收件人: dev@flink.apache.org 
> > 主题: Re: [ANNOUNCE] New Apache Flink Committer - Hangxiang Yu
> >
> > Congratulations and welcome aboard, Hangxiang!
> >
> > Best,
> > Qingsheng
> >
> > On Mon, Aug 7, 2023 at 4:19 PM Matthias Pohl  > .invalid>
> > wrote:
> >
> > > Congratulations, Hangxiang! :)
> > >
> > > On Mon, Aug 7, 2023 at 10:01 AM Junrui Lee 
> wrote:
> > >
> > > > Congratulations, Hangxiang!
> > > >
> > > > Best,
> > > > Junrui
> > > >
> > > > Yun Tang  于2023年8月7日周一 15:19写道：
> > > >
> > > > > Congratulations, Hangxiang!
> > > > >
> > > > > Best
> > > > > Yun Tang
> > > > > 
> > > > > From: Danny Cranmer 
> > > > > Sent: Monday, August 7, 2023 15:11
> > > > > To: dev 
> > > > > Subject: Re: [ANNOUNCE] New Apache Flink Committer - Hangxiang Yu
> > > > >
> > > > > Congrats Hangxiang! Welcome to the team.
> > > > >
> > > > > Danny.
> > > > >
> > > > > On Mon, 7 Aug 2023, 08:04 Rui Fan, <1996fan...@gmail.com> wrote:
> > > > >
> > > > > > Congratulations Hangxiang!
> > > > > >
> > > > > > Best,
> > > > > > Rui
> > > > > >
> > > > > > On Mon, Aug 7, 2023 at 2:58 PM Yuan Mei 
> > > > wrote:
> > > > > >
> > > > > > > On behalf of the PMC, I'm happy to announce Hangxiang Yu as a
> new
> > > > Flink
> > > > > > > Committer.
> > > > > > >
> > > > > > > Hangxiang has been active in the Flink community for more than
> > 1.5
> > > > > years
> > > > > > > and has played an important role in developing and maintaining
> > > State
> > > > > and
> > > > > > > Checkpoint related features/components, including Generic
> > > Incremental
> > > > > > > Checkpoints (take great efforts to make the feature
> prod-ready).
> > > > > > Hangxiang
> > > > > > > is also the main driver of the FLIP-263: Resolving schema
> > > > > compatibility.
> > > > > > >
> > > > > > > Hangxiang is passionate about the Flink community. Besides the
> > > > > technical
> > > > > > > contribution above, he is also actively promoting Flink: talks
> > > about
> > > > > > > Generic
> > > > > > > Incremental Checkpoints in Flink Forward and Meet-up. Hangxiang
> > > also
> > > > > > spent
> > > > > > > a good amount of time supporting users, participating in
> > > Jira/mailing
> > > > > > list
> > > > > > > discussions, and reviewing code.
> > > > > > >
> > > > > > > Please join me in congratulating Hangxiang for becoming a Flink
> > > > > > Committer!
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Yuan Mei (on behalf of the Flink PMC)
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] FLIP-357: Deprecate Iteration API of DataStream

[Yuxin Tan]

+1 (non-binding)

Best,
Yuxin


Xintong Song  于2023年9月14日周四 17:14写道：

> +1 (binding)
>
> Best,
>
> Xintong
>
>
>
> On Thu, Sep 14, 2023 at 3:48 PM Jing Ge 
> wrote:
>
> > +1(binding)
> >
> > Best regards,
> > Jing
> >
> > On Thu, Sep 14, 2023 at 7:31 AM Dong Lin  wrote:
> >
> > > Thanks Wencong for the FLIP.
> > >
> > > +1 (binding)
> > >
> > > On Thu, Sep 14, 2023 at 12:36 PM Wencong Liu 
> > wrote:
> > >
> > > > Hi dev,
> > > >
> > > >
> > > > I'd like to start a vote on FLIP-357.
> > > >
> > > >
> > > > Discussion thread:
> > > > https://lists.apache.org/thread/shf77phc0wzlbj06jsfj3nclxnm2mrv5
> > > > FLIP:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-357%3A+Deprecate+Iteration+API+of+DataStream
> > > >
> > > >
> > > > Best regards,
> > > > Wencong Liu
> > >
> >
>

Re: Re: [DISCUSS] FLIP-331: Support EndOfStreamWindows and isOutputOnEOF operator attribute to optimize task deployment

[Yuxin Tan]

Hi, Dong,
Thanks for your efforts.

+1 to this proposal,
I believe this will improve the performance in some mixture circumstances
of bounded and unbounded workloads.

Best,
Yuxin


Xintong Song  于2023年9月18日周一 10:56写道：

> Thanks for addressing my comments, Dong.
>
> LGTM.
>
> Best,
>
> Xintong
>
>
>
> On Sat, Sep 16, 2023 at 3:34 PM Wencong Liu  wrote:
>
> > Hi Dong & Jinhao,
> >
> > Thanks for your clarification! +1
> >
> > Best regards,
> > Wencong
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > At 2023-09-15 11:26:16, "Dong Lin"  wrote:
> > >Hi Wencong,
> > >
> > >Thanks for your comments! Please see my reply inline.
> > >
> > >On Thu, Sep 14, 2023 at 12:30 PM Wencong Liu 
> > wrote:
> > >
> > >> Dear Dong,
> > >>
> > >> I have thoroughly reviewed the proposal for FLIP-331 and believe it
> > would
> > >> be
> > >> a valuable addition to Flink. However, I do have a few questions that
> I
> > >> would
> > >> like to discuss:
> > >>
> > >>
> > >> 1. The FLIP-331 proposed the EndOfStreamWindows that is implemented by
> > >> TimeWindow with maxTimestamp = (Long.MAX_VALUE - 1), which naturally
> > >> supports WindowedStream and AllWindowedStream to process all records
> > >> belonging to a key in a 'global' window under both STREAMING and BATCH
> > >> runtime execution mode.
> > >>
> > >>
> > >> However, besides coGroup and keyBy().aggregate(), other operators on
> > >> WindowedStream and AllWindowedStream, such as join/reduce, etc,
> > currently
> > >> are still implemented based on WindowOperator.
> > >>
> > >>
> > >> In fact, these operators can also be implemented without using
> > >> WindowOperator
> > >> to prevent additional WindowAssigner#assignWindows or
> > >> triggerContext#onElement
> > >> invocation cost. Will there be plans to support these operators in the
> > >> future?
> > >>
> > >
> > >You are right. The EndOfStreamWindows proposed in this FLIP can
> > potentially
> > >benefit any DataStream API that takes WindowAssigner as parameters. This
> > >can involve more operations than aggregate and co-group.
> > >
> > >And yes, we have plans to take advantage of this API to optimize these
> > >operators in the future. This FLIP focuses on the introduction of the
> > >public APIs and uses aggregate/co-group as the first two examples to
> > >show-case the performance benefits.
> > >
> > >I have added a "Analysis of APIs affected by this FLIP" to list the
> > >DataStream APIs that can benefit from this FLIP. Would this answer your
> > >question?
> > >
> > >
> > >>
> > >> 2. When using EndOfStreamWindows, upstream operators no longer support
> > >> checkpointing. This limit may be too strict, especially when dealing
> > with
> > >> bounded data in streaming runtime execution mode, where checkpointing
> > >> can still be useful.
> > >>
> > >
> > >I am not sure we have a good way to support checkpoint while still
> > >achieving the performance improves targeted by this FLIP.
> > >
> > >The issue here is that if we support checkpoint, then we can not take
> > >advantage of algorithms (e.g. sorting inputs using ExternalSorter) that
> > are
> > >not compatible with checkpoints. These algorithms (which do not support
> > >checkpoint) are the main reasons why batch mode currently significantly
> > >outperforms stream mode in doing aggregation/cogroup etc.
> > >
> > >In most cases where the user does not care about processing latency, it
> is
> > >generally preferred to use batch mode to perform aggregation operations
> > >(which should be 10X faster than the existing stream mode performance)
> > >instead of doing checkpoint.
> > >
> > >Also note that we can still let operators perform failover in the same
> as
> > >the existing batch mode execution, where the intermediate results
> > (produced
> > >by one operator) can be persisted in shuffle service and downstream
> > >operators can re-read those data from shuffle service after failover.
> > >
> > >
> > >>
> > >> 3. The proposal mentions that if a transformation has isOutputOnEOF ==
> > >> true, the
> > >> operator as well as its upstream operators will be executed in 'batch
> > >> mode' with
> > >> checkpointing disabled. I would like to understand the specific
> > >> implications of this
> > >> 'batch mode' and if there are any other changes associated with it?
> > >
> > >
> > >Good point. We should explicitly mention the changes. I have updated the
> > >FLIP to clarify this.
> > >
> > >More specifically, the checkpoint is disabled when these operators are
> > >running, such that these operators can do operations not compatible with
> > >checkpoints (e.g. sorting inputs). And operators should re-read the data
> > >from the upstream blocking edge or sources after failover.
> > >
> > >Would this answer your question?
> > >
> > >
> > >>
> > >> Additionally, I am curious to know if this 'batch mode' conflicts with
> > the
> > >> 'mix mode'
> > >>
> > >> described in FLIP-327. While the coGroup and keyBy().aggregate()
> > operators
> > >

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

[Yuxin Tan]

Hi, dear community and release managers,

Thanks for bringing this up.

When testing the release candidate #0 for the batch scenario, I found an
issue of frequent flushing in Hybrid shuffle. It is a new bug introduced by
1.18 and may significantly impact the performance of shuffle writing.

The fix ticket FLINK-33044[1] has been reviewed and approved. To prevent
any performance issues, I would like to backport the fix to version 1.18
unless there are any objections.

Glad to hear your suggestions.

[1] https://issues.apache.org/jira/browse/FLINK-33044

Best,
Yuxin

Zakelly Lan  于2023年9月19日周二 17:26写道：

> Hi Yuan and Jing,
>
> Thank you for sharing your thoughts. I completely agree that it is our
> top priority to ensure that there are no regressions from the last
> commit the previous benchmark pipeline covered to the final commit of
> this release. I will try to get this result first.
>
>
> Best,
> Zakelly
>
> On Tue, Sep 19, 2023 at 4:55 PM Jing Ge 
> wrote:
> >
> > Hi
> >
> > Thanks Zakelly and Yuan for your effort and update. Since we changed the
> > hardware, IMHO, if we are able to reach a consensus in the community that
> > there is no regression with the benchmarks, we could consider releasing
> rc1
> > without waiting for the new baseline scores which might take days.
> >
> > Best regards,
> > Jing
> >
> > On Tue, Sep 19, 2023 at 10:42 AM Yuan Mei 
> wrote:
> >
> > > Hey Zakelly,
> > >
> > > Thanks very much for the efforts to re-build the entire benchmark
> > > environment.
> > >
> > > As long as we have
> > > 1) the pipeline set up and ready (no need for the entire portal ready),
> > > 2) get benchmark comparison numbers (comparing with the commit just
> before
> > > the benchmark pipeline is down) and
> > > 3) confirmed no-regression, it should be good enough.
> > >
> > > Thanks again!
> > >
> > > Best
> > > Yuan
> > >
> > > On Tue, Sep 19, 2023 at 4:26 PM Zakelly Lan 
> wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I am working on rebuilding the benchmark pipeline and it's almost
> > > > done. However, due to the change in machines for benchmarking, I will
> > > > need a few more days to run tests and gather the baseline scores for
> > > > further comparison. Once the pipeline is fully ready, we will proceed
> > > > with the performance test for release 1.18.0.
> > > >
> > > > Please let me know if you have any concerns. Thank you all for your
> > > > patience.
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > > > On Mon, Sep 18, 2023 at 6:57 PM Jing Ge 
> > > > wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > The RC0 for Apache Flink 1.18.0 has been created. This RC is
> currently
> > > > for
> > > > > preview only to facilitate the integrated testing since the
> benchmark
> > > > tests
> > > > > are not available yet[1] and the release announcement is still
> under
> > > > > review. The RC1 will be released after all benchmarks tests are
> passed.
> > > > The
> > > > > related voting process will be triggered once the announcement is
> > > ready.
> > > > > The RC0 has all the artifacts that we would typically have for a
> > > release,
> > > > > except for the release note and the website pull request for the
> > > release
> > > > > announcement.
> > > > >
> > > > > The following contents are available for your review:
> > > > >
> > > > > - The preview source release and binary convenience releases [2],
> which
> > > > > are signed with the key with fingerprint 96AE0E32CBE6E0753CE6 [3].
> > > > > - all artifacts that would normally be deployed to the Maven
> > > > > Central Repository [4].
> > > > > - source code tag "release-1.18.0-rc0" [5]
> > > > >
> > > > > Your help testing the release will be greatly appreciated! And
> we'll
> > > > > create the rc1 release and the voting thread as soon as all the
> efforts
> > > > are
> > > > > finished.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/FLINK-33052
> > > > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc0/
> > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > [4]
> > > >
> https://repository.apache.org/content/repositories/orgapacheflink-1656/
> > > > > [5]
> https://github.com/apache/flink/releases/tag/release-1.18.0-rc0
> > > > >
> > > > > Best regards,
> > > > > Qingsheng, Sergei, Konstantin and Jing
> > > >
> > >
>

[DISCUSS] Backport fix to release 1.18

[Yuxin Tan]

Hi, dear community and release managers,

Thanks for bringing up the new release candidate for 1.18.

When testing for the batch scenario, I found an issue of frequent
flushing in Hybrid shuffle. It is a new bug introduced by 1.18 and
may significantly impact the performance of shuffle writing.

The fix ticket FLINK-33044[1] has been reviewed and approved. To
prevent the performance issues, I would like to backport the fix to release
1.18 unless there are any objections.

Glad to hear your suggestions.

In addition to this, sorry to start a new email thread to discuss it.
I attempted many times to send this email within the existing thread
titled "[ANNOUNCE] Release 1.18.0, release candidate #0", but the mail
failed to be delivered, that thread looks broken. It is very strange.
So I start the new discussion thread.

[1] https://issues.apache.org/jira/browse/FLINK-33044

Best,
Yuxin

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

[Yuxin Tan]

Hi, dear community and release managers,

Thanks for bringing this up.

When testing the release candidate #0 for the batch scenario, I found an
issue of frequent flushing in Hybrid shuffle. It is a new bug introduced by
1.18 and may significantly impact the performance of shuffle writing.

The fix tickit FLINK-33044[1] has been reviewed and approved. To prevent
any performance issues, I would like to backport the fix to version 1.18
unless there are any objections.

Glad to hear your suggestions.

[1] https://issues.apache.org/jira/browse/FLINK-33044

Best,
Yuxin


Zakelly Lan  于2023年9月19日周二 17:26写道：

> Hi Yuan and Jing,
>
> Thank you for sharing your thoughts. I completely agree that it is our
> top priority to ensure that there are no regressions from the last
> commit the previous benchmark pipeline covered to the final commit of
> this release. I will try to get this result first.
>
>
> Best,
> Zakelly
>
> On Tue, Sep 19, 2023 at 4:55 PM Jing Ge 
> wrote:
> >
> > Hi
> >
> > Thanks Zakelly and Yuan for your effort and update. Since we changed the
> > hardware, IMHO, if we are able to reach a consensus in the community that
> > there is no regression with the benchmarks, we could consider releasing
> rc1
> > without waiting for the new baseline scores which might take days.
> >
> > Best regards,
> > Jing
> >
> > On Tue, Sep 19, 2023 at 10:42 AM Yuan Mei 
> wrote:
> >
> > > Hey Zakelly,
> > >
> > > Thanks very much for the efforts to re-build the entire benchmark
> > > environment.
> > >
> > > As long as we have
> > > 1) the pipeline set up and ready (no need for the entire portal ready),
> > > 2) get benchmark comparison numbers (comparing with the commit just
> before
> > > the benchmark pipeline is down) and
> > > 3) confirmed no-regression, it should be good enough.
> > >
> > > Thanks again!
> > >
> > > Best
> > > Yuan
> > >
> > > On Tue, Sep 19, 2023 at 4:26 PM Zakelly Lan 
> wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I am working on rebuilding the benchmark pipeline and it's almost
> > > > done. However, due to the change in machines for benchmarking, I will
> > > > need a few more days to run tests and gather the baseline scores for
> > > > further comparison. Once the pipeline is fully ready, we will proceed
> > > > with the performance test for release 1.18.0.
> > > >
> > > > Please let me know if you have any concerns. Thank you all for your
> > > > patience.
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > > > On Mon, Sep 18, 2023 at 6:57 PM Jing Ge 
> > > > wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > The RC0 for Apache Flink 1.18.0 has been created. This RC is
> currently
> > > > for
> > > > > preview only to facilitate the integrated testing since the
> benchmark
> > > > tests
> > > > > are not available yet[1] and the release announcement is still
> under
> > > > > review. The RC1 will be released after all benchmarks tests are
> passed.
> > > > The
> > > > > related voting process will be triggered once the announcement is
> > > ready.
> > > > > The RC0 has all the artifacts that we would typically have for a
> > > release,
> > > > > except for the release note and the website pull request for the
> > > release
> > > > > announcement.
> > > > >
> > > > > The following contents are available for your review:
> > > > >
> > > > > - The preview source release and binary convenience releases [2],
> which
> > > > > are signed with the key with fingerprint 96AE0E32CBE6E0753CE6 [3].
> > > > > - all artifacts that would normally be deployed to the Maven
> > > > > Central Repository [4].
> > > > > - source code tag "release-1.18.0-rc0" [5]
> > > > >
> > > > > Your help testing the release will be greatly appreciated! And
> we'll
> > > > > create the rc1 release and the voting thread as soon as all the
> efforts
> > > > are
> > > > > finished.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/FLINK-33052
> > > > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc0/
> > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > [4]
> > > >
> https://repository.apache.org/content/repositories/orgapacheflink-1656/
> > > > > [5]
> https://github.com/apache/flink/releases/tag/release-1.18.0-rc0
> > > > >
> > > > > Best regards,
> > > > > Qingsheng, Sergei, Konstantin and Jing
> > > >
> > >
>

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

[Yuxin Tan]

Hi, dear community and release managers,

Thanks for bringing this up.

When testing the release candidate #0 for the batch scenario, I found an
issue of frequent flushing in Hybrid shuffle. It is a new bug introduced by
1.18 and may significantly impact the performance of shuffle writing.

The fix tickit FLINK-33044[1] has been reviewed and approved. To prevent
any performance issues, I would like to backport the fix to version 1.18
unless there are any objections.

Glad to hear your suggestions.

[1] https://issues.apache.org/jira/browse/FLINK-33044

Best,
Yuxin


Zakelly Lan  于2023年9月19日周二 17:26写道：

> Hi Yuan and Jing,
>
> Thank you for sharing your thoughts. I completely agree that it is our
> top priority to ensure that there are no regressions from the last
> commit the previous benchmark pipeline covered to the final commit of
> this release. I will try to get this result first.
>
>
> Best,
> Zakelly
>
> On Tue, Sep 19, 2023 at 4:55 PM Jing Ge 
> wrote:
> >
> > Hi
> >
> > Thanks Zakelly and Yuan for your effort and update. Since we changed the
> > hardware, IMHO, if we are able to reach a consensus in the community that
> > there is no regression with the benchmarks, we could consider releasing
> rc1
> > without waiting for the new baseline scores which might take days.
> >
> > Best regards,
> > Jing
> >
> > On Tue, Sep 19, 2023 at 10:42 AM Yuan Mei 
> wrote:
> >
> > > Hey Zakelly,
> > >
> > > Thanks very much for the efforts to re-build the entire benchmark
> > > environment.
> > >
> > > As long as we have
> > > 1) the pipeline set up and ready (no need for the entire portal ready),
> > > 2) get benchmark comparison numbers (comparing with the commit just
> before
> > > the benchmark pipeline is down) and
> > > 3) confirmed no-regression, it should be good enough.
> > >
> > > Thanks again!
> > >
> > > Best
> > > Yuan
> > >
> > > On Tue, Sep 19, 2023 at 4:26 PM Zakelly Lan 
> wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I am working on rebuilding the benchmark pipeline and it's almost
> > > > done. However, due to the change in machines for benchmarking, I will
> > > > need a few more days to run tests and gather the baseline scores for
> > > > further comparison. Once the pipeline is fully ready, we will proceed
> > > > with the performance test for release 1.18.0.
> > > >
> > > > Please let me know if you have any concerns. Thank you all for your
> > > > patience.
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > > > On Mon, Sep 18, 2023 at 6:57 PM Jing Ge 
> > > > wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > The RC0 for Apache Flink 1.18.0 has been created. This RC is
> currently
> > > > for
> > > > > preview only to facilitate the integrated testing since the
> benchmark
> > > > tests
> > > > > are not available yet[1] and the release announcement is still
> under
> > > > > review. The RC1 will be released after all benchmarks tests are
> passed.
> > > > The
> > > > > related voting process will be triggered once the announcement is
> > > ready.
> > > > > The RC0 has all the artifacts that we would typically have for a
> > > release,
> > > > > except for the release note and the website pull request for the
> > > release
> > > > > announcement.
> > > > >
> > > > > The following contents are available for your review:
> > > > >
> > > > > - The preview source release and binary convenience releases [2],
> which
> > > > > are signed with the key with fingerprint 96AE0E32CBE6E0753CE6 [3].
> > > > > - all artifacts that would normally be deployed to the Maven
> > > > > Central Repository [4].
> > > > > - source code tag "release-1.18.0-rc0" [5]
> > > > >
> > > > > Your help testing the release will be greatly appreciated! And
> we'll
> > > > > create the rc1 release and the voting thread as soon as all the
> efforts
> > > > are
> > > > > finished.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/FLINK-33052
> > > > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc0/
> > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > [4]
> > > >
> https://repository.apache.org/content/repositories/orgapacheflink-1656/
> > > > > [5]
> https://github.com/apache/flink/releases/tag/release-1.18.0-rc0
> > > > >
> > > > > Best regards,
> > > > > Qingsheng, Sergei, Konstantin and Jing
> > > >
> > >
>

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

[Yuxin Tan]

Hi, dear community and release managers,

Thanks for bringing this up.

When testing the release candidate #0 for the batch scenario, I found an
issue of frequent flushing in Hybrid shuffle. It is a new bug introduced by
1.18 and may significantly impact the performance of shuffle writing.

The fix ticket FLINK-33044[1] has been reviewed and approved. To prevent
any performance issues, I would like to backport the fix to version 1.18
unless there are any objections.

Glad to hear your suggestions.

[1] https://issues.apache.org/jira/browse/FLINK-33044

Best,
Yuxin


Zakelly Lan  于2023年9月19日周二 17:26写道：

> Hi Yuan and Jing,
>
> Thank you for sharing your thoughts. I completely agree that it is our
> top priority to ensure that there are no regressions from the last
> commit the previous benchmark pipeline covered to the final commit of
> this release. I will try to get this result first.
>
>
> Best,
> Zakelly
>
> On Tue, Sep 19, 2023 at 4:55 PM Jing Ge 
> wrote:
> >
> > Hi
> >
> > Thanks Zakelly and Yuan for your effort and update. Since we changed the
> > hardware, IMHO, if we are able to reach a consensus in the community that
> > there is no regression with the benchmarks, we could consider releasing
> rc1
> > without waiting for the new baseline scores which might take days.
> >
> > Best regards,
> > Jing
> >
> > On Tue, Sep 19, 2023 at 10:42 AM Yuan Mei 
> wrote:
> >
> > > Hey Zakelly,
> > >
> > > Thanks very much for the efforts to re-build the entire benchmark
> > > environment.
> > >
> > > As long as we have
> > > 1) the pipeline set up and ready (no need for the entire portal ready),
> > > 2) get benchmark comparison numbers (comparing with the commit just
> before
> > > the benchmark pipeline is down) and
> > > 3) confirmed no-regression, it should be good enough.
> > >
> > > Thanks again!
> > >
> > > Best
> > > Yuan
> > >
> > > On Tue, Sep 19, 2023 at 4:26 PM Zakelly Lan 
> wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I am working on rebuilding the benchmark pipeline and it's almost
> > > > done. However, due to the change in machines for benchmarking, I will
> > > > need a few more days to run tests and gather the baseline scores for
> > > > further comparison. Once the pipeline is fully ready, we will proceed
> > > > with the performance test for release 1.18.0.
> > > >
> > > > Please let me know if you have any concerns. Thank you all for your
> > > > patience.
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > > > On Mon, Sep 18, 2023 at 6:57 PM Jing Ge 
> > > > wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > The RC0 for Apache Flink 1.18.0 has been created. This RC is
> currently
> > > > for
> > > > > preview only to facilitate the integrated testing since the
> benchmark
> > > > tests
> > > > > are not available yet[1] and the release announcement is still
> under
> > > > > review. The RC1 will be released after all benchmarks tests are
> passed.
> > > > The
> > > > > related voting process will be triggered once the announcement is
> > > ready.
> > > > > The RC0 has all the artifacts that we would typically have for a
> > > release,
> > > > > except for the release note and the website pull request for the
> > > release
> > > > > announcement.
> > > > >
> > > > > The following contents are available for your review:
> > > > >
> > > > > - The preview source release and binary convenience releases [2],
> which
> > > > > are signed with the key with fingerprint 96AE0E32CBE6E0753CE6 [3].
> > > > > - all artifacts that would normally be deployed to the Maven
> > > > > Central Repository [4].
> > > > > - source code tag "release-1.18.0-rc0" [5]
> > > > >
> > > > > Your help testing the release will be greatly appreciated! And
> we'll
> > > > > create the rc1 release and the voting thread as soon as all the
> efforts
> > > > are
> > > > > finished.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/FLINK-33052
> > > > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc0/
> > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > [4]
> > > >
> https://repository.apache.org/content/repositories/orgapacheflink-1656/
> > > > > [5]
> https://github.com/apache/flink/releases/tag/release-1.18.0-rc0
> > > > >
> > > > > Best regards,
> > > > > Qingsheng, Sergei, Konstantin and Jing
> > > >
> > >
>

Re: [DISCUSS] Backport fix to release 1.18

[Yuxin Tan]

Hi, Jing

Thanks for the update. The delay is indeed long.

Please ignore it and join the discussion in the original thread.

Best,
Yuxin


Jing Ge  于2023年9月20日周三 22:01写道：

> Hi folks,
>
> Please ignore this email. It seems the dev mail server has some issues
> that emails have been sent but will be received by dev@flink.apache.org
> with a big delay at [1]. Participants who have been sent to or cc directly
> should get the email instantly. Let's consolidate the discussion in this
> thread [1].
>
> Best regards,
> Jing
>
>
> [1] https://lists.apache.org/thread/5x28rp3zct4p603hm4zdwx6kfr101w38
>
> On Wed, Sep 20, 2023 at 11:43 AM Yuxin Tan  wrote:
>
>>
>> Hi, dear community and release managers,
>>
>> Thanks for bringing up the new release candidate for 1.18.
>>
>> When testing for the batch scenario, I found an issue of frequent
>> flushing in Hybrid shuffle. It is a new bug introduced by 1.18 and
>> may significantly impact the performance of shuffle writing.
>>
>> The fix ticket FLINK-33044[1] has been reviewed and approved. To
>> prevent the performance issues, I would like to backport the fix to
>> release
>> 1.18 unless there are any objections.
>>
>> Glad to hear your suggestions.
>>
>> In addition to this, sorry to start a new email thread to discuss it.
>> I attempted many times to send this email within the existing thread
>> titled "[ANNOUNCE] Release 1.18.0, release candidate #0", but the mail
>> failed to be delivered, that thread looks broken. It is very strange.
>> So I start the new discussion thread.
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-33044
>>
>> Best,
>> Yuxin
>>
>

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

[Yuxin Tan]

Hi, dear community and release managers,

Thanks for bringing this up.

When testing the release candidate #0 for the batch scenario, I found an
issue of frequent flushing in Hybrid shuffle. It is a new bug introduced by
1.18 and may significantly impact the performance of shuffle writing.

The fix tickit FLINK-33044[1] has been reviewed and approved. To prevent
any performance issues, I would like to backport the fix to version 1.18
unless there are any objections.

Glad to hear your suggestions.

[1] https://issues.apache.org/jira/browse/FLINK-33044

Best,
Yuxin


Zakelly Lan  于2023年9月19日周二 17:26写道：

> Hi Yuan and Jing,
>
> Thank you for sharing your thoughts. I completely agree that it is our
> top priority to ensure that there are no regressions from the last
> commit the previous benchmark pipeline covered to the final commit of
> this release. I will try to get this result first.
>
>
> Best,
> Zakelly
>
> On Tue, Sep 19, 2023 at 4:55 PM Jing Ge 
> wrote:
> >
> > Hi
> >
> > Thanks Zakelly and Yuan for your effort and update. Since we changed the
> > hardware, IMHO, if we are able to reach a consensus in the community that
> > there is no regression with the benchmarks, we could consider releasing
> rc1
> > without waiting for the new baseline scores which might take days.
> >
> > Best regards,
> > Jing
> >
> > On Tue, Sep 19, 2023 at 10:42 AM Yuan Mei 
> wrote:
> >
> > > Hey Zakelly,
> > >
> > > Thanks very much for the efforts to re-build the entire benchmark
> > > environment.
> > >
> > > As long as we have
> > > 1) the pipeline set up and ready (no need for the entire portal ready),
> > > 2) get benchmark comparison numbers (comparing with the commit just
> before
> > > the benchmark pipeline is down) and
> > > 3) confirmed no-regression, it should be good enough.
> > >
> > > Thanks again!
> > >
> > > Best
> > > Yuan
> > >
> > > On Tue, Sep 19, 2023 at 4:26 PM Zakelly Lan 
> wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I am working on rebuilding the benchmark pipeline and it's almost
> > > > done. However, due to the change in machines for benchmarking, I will
> > > > need a few more days to run tests and gather the baseline scores for
> > > > further comparison. Once the pipeline is fully ready, we will proceed
> > > > with the performance test for release 1.18.0.
> > > >
> > > > Please let me know if you have any concerns. Thank you all for your
> > > > patience.
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > > > On Mon, Sep 18, 2023 at 6:57 PM Jing Ge 
> > > > wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > The RC0 for Apache Flink 1.18.0 has been created. This RC is
> currently
> > > > for
> > > > > preview only to facilitate the integrated testing since the
> benchmark
> > > > tests
> > > > > are not available yet[1] and the release announcement is still
> under
> > > > > review. The RC1 will be released after all benchmarks tests are
> passed.
> > > > The
> > > > > related voting process will be triggered once the announcement is
> > > ready.
> > > > > The RC0 has all the artifacts that we would typically have for a
> > > release,
> > > > > except for the release note and the website pull request for the
> > > release
> > > > > announcement.
> > > > >
> > > > > The following contents are available for your review:
> > > > >
> > > > > - The preview source release and binary convenience releases [2],
> which
> > > > > are signed with the key with fingerprint 96AE0E32CBE6E0753CE6 [3].
> > > > > - all artifacts that would normally be deployed to the Maven
> > > > > Central Repository [4].
> > > > > - source code tag "release-1.18.0-rc0" [5]
> > > > >
> > > > > Your help testing the release will be greatly appreciated! And
> we'll
> > > > > create the rc1 release and the voting thread as soon as all the
> efforts
> > > > are
> > > > > finished.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/FLINK-33052
> > > > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc0/
> > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > [4]
> > > >
> https://repository.apache.org/content/repositories/orgapacheflink-1656/
> > > > > [5]
> https://github.com/apache/flink/releases/tag/release-1.18.0-rc0
> > > > >
> > > > > Best regards,
> > > > > Qingsheng, Sergei, Konstantin and Jing
> > > >
> > >
>

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

[Yuxin Tan]

Hi, Jing, Qingsheng,

Thanks a lot.
The fix has been backported.

Best,
Yuxin


Jing Ge  于2023年9月21日周四 00:42写道：

> Hi Lijie,
>
> Thanks for reaching out. Please backport it to release-1.18.
>
> Best regards,
> Jing
>
> On Wed, Sep 20, 2023 at 4:35 PM Lijie Wang 
> wrote:
>
> > Hi community and release managers:
> >
> > We found a critical bug[1] of the rest client a few days ago, which may
> > cause the inode to be used up. Now the fix-PR[2] is ready for merging, I
> > hope to backport it to release-1.18.
> >
> > Please let me know if you have any concerns. Thanks.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-32974
> > [2] https://github.com/apache/flink/pull/23363
> >
> > Best,
> > Lijie
> >
> > Zakelly Lan  于2023年9月19日周二 17:26写道：
> >
> > > Hi Yuan and Jing,
> > >
> > > Thank you for sharing your thoughts. I completely agree that it is our
> > > top priority to ensure that there are no regressions from the last
> > > commit the previous benchmark pipeline covered to the final commit of
> > > this release. I will try to get this result first.
> > >
> > >
> > > Best,
> > > Zakelly
> > >
> > > On Tue, Sep 19, 2023 at 4:55 PM Jing Ge 
> > > wrote:
> > > >
> > > > Hi
> > > >
> > > > Thanks Zakelly and Yuan for your effort and update. Since we changed
> > the
> > > > hardware, IMHO, if we are able to reach a consensus in the community
> > that
> > > > there is no regression with the benchmarks, we could consider
> releasing
> > > rc1
> > > > without waiting for the new baseline scores which might take days.
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > On Tue, Sep 19, 2023 at 10:42 AM Yuan Mei 
> > > wrote:
> > > >
> > > > > Hey Zakelly,
> > > > >
> > > > > Thanks very much for the efforts to re-build the entire benchmark
> > > > > environment.
> > > > >
> > > > > As long as we have
> > > > > 1) the pipeline set up and ready (no need for the entire portal
> > ready),
> > > > > 2) get benchmark comparison numbers (comparing with the commit just
> > > before
> > > > > the benchmark pipeline is down) and
> > > > > 3) confirmed no-regression, it should be good enough.
> > > > >
> > > > > Thanks again!
> > > > >
> > > > > Best
> > > > > Yuan
> > > > >
> > > > > On Tue, Sep 19, 2023 at 4:26 PM Zakelly Lan  >
> > > wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > I am working on rebuilding the benchmark pipeline and it's almost
> > > > > > done. However, due to the change in machines for benchmarking, I
> > will
> > > > > > need a few more days to run tests and gather the baseline scores
> > for
> > > > > > further comparison. Once the pipeline is fully ready, we will
> > proceed
> > > > > > with the performance test for release 1.18.0.
> > > > > >
> > > > > > Please let me know if you have any concerns. Thank you all for
> your
> > > > > > patience.
> > > > > >
> > > > > > Best,
> > > > > > Zakelly
> > > > > >
> > > > > > On Mon, Sep 18, 2023 at 6:57 PM Jing Ge
>  > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > The RC0 for Apache Flink 1.18.0 has been created. This RC is
> > > currently
> > > > > > for
> > > > > > > preview only to facilitate the integrated testing since the
> > > benchmark
> > > > > > tests
> > > > > > > are not available yet[1] and the release announcement is still
> > > under
> > > > > > > review. The RC1 will be released after all benchmarks tests are
> > > passed.
> > > > > > The
> > > > > > > related voting process will be triggered once the announcement
> is
> > > > > ready.
> > > > > > > The RC0 has all the artifacts that we would typically have for
> a
> > > > > release,
> > > > > > > except for the release note and the website pull request for
> the
> > > > > release
> > > > > > > announcement.
> > > > > > >
> > > > > > > The following contents are available for your review:
> > > > > > >
> > > > > > > - The preview source release and binary convenience releases
> [2],
> > > which
> > > > > > > are signed with the key with fingerprint 96AE0E32CBE6E0753CE6
> > [3].
> > > > > > > - all artifacts that would normally be deployed to the Maven
> > > > > > > Central Repository [4].
> > > > > > > - source code tag "release-1.18.0-rc0" [5]
> > > > > > >
> > > > > > > Your help testing the release will be greatly appreciated! And
> > > we'll
> > > > > > > create the rc1 release and the voting thread as soon as all the
> > > efforts
> > > > > > are
> > > > > > > finished.
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-33052
> > > > > > > [2]
> > https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc0/
> > > > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > > > [4]
> > > > > >
> > >
> https://repository.apache.org/content/repositories/orgapacheflink-1656/
> > > > > > > [5]
> > > https://github.com/apache/flink/releases/tag/release-1.18.0-rc0
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Qingsheng, Sergei, Konstantin and Jing
> > > > > >
> > > > >
> 

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

[Yuxin Tan]

Hi, Zakelly,

No benchmark tests currently are affected by this issue. We
may add benchmarks to guard it later. Thanks.

Best,
Yuxin


Zakelly Lan  于2023年9月21日周四 11:56写道：

> Hi Jing,
>
> Sure, I will run the benchmark with this fix.
>
> Hi Yunxin,
>
> I'm not familiar with the hybrid shuffle. Is there any specific
> benchmark test that may be affected by this issue? I will pay special
> attention to it.
> Thanks.
>
>
> Best,
> Zakelly
>
> On Thu, Sep 21, 2023 at 10:08 AM Yuxin Tan  wrote:
> >
> > Hi, Jing, Qingsheng,
> >
> > Thanks a lot.
> > The fix has been backported.
> >
> > Best,
> > Yuxin
> >
> >
> > Jing Ge  于2023年9月21日周四 00:42写道：
> >
> > > Hi Lijie,
> > >
> > > Thanks for reaching out. Please backport it to release-1.18.
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Wed, Sep 20, 2023 at 4:35 PM Lijie Wang 
> > > wrote:
> > >
> > > > Hi community and release managers:
> > > >
> > > > We found a critical bug[1] of the rest client a few days ago, which
> may
> > > > cause the inode to be used up. Now the fix-PR[2] is ready for
> merging, I
> > > > hope to backport it to release-1.18.
> > > >
> > > > Please let me know if you have any concerns. Thanks.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FLINK-32974
> > > > [2] https://github.com/apache/flink/pull/23363
> > > >
> > > > Best,
> > > > Lijie
> > > >
> > > > Zakelly Lan  于2023年9月19日周二 17:26写道：
> > > >
> > > > > Hi Yuan and Jing,
> > > > >
> > > > > Thank you for sharing your thoughts. I completely agree that it is
> our
> > > > > top priority to ensure that there are no regressions from the last
> > > > > commit the previous benchmark pipeline covered to the final commit
> of
> > > > > this release. I will try to get this result first.
> > > > >
> > > > >
> > > > > Best,
> > > > > Zakelly
> > > > >
> > > > > On Tue, Sep 19, 2023 at 4:55 PM Jing Ge  >
> > > > > wrote:
> > > > > >
> > > > > > Hi
> > > > > >
> > > > > > Thanks Zakelly and Yuan for your effort and update. Since we
> changed
> > > > the
> > > > > > hardware, IMHO, if we are able to reach a consensus in the
> community
> > > > that
> > > > > > there is no regression with the benchmarks, we could consider
> > > releasing
> > > > > rc1
> > > > > > without waiting for the new baseline scores which might take
> days.
> > > > > >
> > > > > > Best regards,
> > > > > > Jing
> > > > > >
> > > > > > On Tue, Sep 19, 2023 at 10:42 AM Yuan Mei <
> yuanmei.w...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hey Zakelly,
> > > > > > >
> > > > > > > Thanks very much for the efforts to re-build the entire
> benchmark
> > > > > > > environment.
> > > > > > >
> > > > > > > As long as we have
> > > > > > > 1) the pipeline set up and ready (no need for the entire portal
> > > > ready),
> > > > > > > 2) get benchmark comparison numbers (comparing with the commit
> just
> > > > > before
> > > > > > > the benchmark pipeline is down) and
> > > > > > > 3) confirmed no-regression, it should be good enough.
> > > > > > >
> > > > > > > Thanks again!
> > > > > > >
> > > > > > > Best
> > > > > > > Yuan
> > > > > > >
> > > > > > > On Tue, Sep 19, 2023 at 4:26 PM Zakelly Lan <
> zakelly@gmail.com
> > > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > I am working on rebuilding the benchmark pipeline and it's
> almost
> > > > > > > > done. However, due to the change in machines for
> benchmarking, I
> > > > will
> > > > > > > > need a few 

Re: [Discuss] FLIP-366: Support standard YAML for FLINK configuration

[Yuxin Tan]

Hi, Junrui

+1 for the proposal.
Thanks for your effort.

Best,
Yuxin


Samrat Deb  于2023年9月22日周五 13:23写道：

> Hello Junrui,
>
> +1 for the proposal.
>
>
> Bests,
> Samrat
>
> On Fri, Sep 22, 2023 at 10:18 AM Shammon FY  wrote:
>
> > +1 for the proposal, thanks for driving.
> >
> > Bet,
> > Shammon FY
> >
> > On Fri, Sep 22, 2023 at 12:41 PM Yangze Guo  wrote:
> >
> > > Thanks for driving this, +1 for the proposal.
> > >
> > > Best,
> > > Yangze Guo
> > >
> > >
> > > On Fri, Sep 22, 2023 at 11:59 AM Lijie Wang 
> > > wrote:
> > > >
> > > > Hi Junrui,
> > > >
> > > > +1 for this proposal, thanks for driving.
> > > >
> > > > Best,
> > > > Lijie
> > > >
> > > > ConradJam  于2023年9月22日周五 10:07写道：
> > > >
> > > > > +1 Support for standard YAML format facilitates specification
> > > > >
> > > > > Jing Ge  于2023年9月22日周五 02:23写道：
> > > > >
> > > > > > Hi Junrui,
> > > > > >
> > > > > > +1 for following the standard. Thanks for your effort!
> > > > > >
> > > > > > Best regards,
> > > > > > Jing
> > > > > >
> > > > > > On Thu, Sep 21, 2023 at 5:09 AM Junrui Lee 
> > > wrote:
> > > > > >
> > > > > > > Hi Jane,
> > > > > > >
> > > > > > > Thank you for your valuable feedback and suggestions.
> > > > > > > I agree with your point about differentiating between
> > > > > "flink-config.yaml"
> > > > > > > and "flink-conf.yaml" to determine the standard syntax at a
> > glance.
> > > > > > >
> > > > > > > While I understand your suggestion of using
> > > "flink-conf-default.yaml"
> > > > > to
> > > > > > > represent the default YAML file for Flink 1.x, I have been
> > > considering
> > > > > > > the option of using "flink-configuration.yaml" as the file name
> > > for the
> > > > > > > new configuration file.
> > > > > > > This name "flink-configuration.yaml" provides a clear
> distinction
> > > > > between
> > > > > > > the new and old configuration files based on their names, and
> it
> > > does
> > > > > not
> > > > > > > introduce any additional semantics. Moreover, this name
> > > > > > > "flink-configuration.yaml" can continue to be used in future
> > > versions
> > > > > > > FLINK-2.0.
> > > > > > >
> > > > > > > WDYT? If we can reach a consensus on this, I will update the
> FLIP
> > > > > > > documentation
> > > > > > > accordingly.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Junrui
> > > > > > >
> > > > > > > Jane Chan  于2023年9月20日周三 23:38写道：
> > > > > > >
> > > > > > > > Hi Junrui,
> > > > > > > >
> > > > > > > > Thanks for driving this FLIP. +1 for adoption of the standard
> > > YAML
> > > > > > > syntax.
> > > > > > > > I just have one minor suggestion. It's a little bit
> challenging
> > > to
> > > > > > > > differentiate between `flink-config.yaml` and
> `flink-conf.yaml`
> > > to
> > > > > > > > determine which one uses the standard syntax at a glance. How
> > > about
> > > > > > > > using `flink-conf-default.yaml` to represent the default yaml
> > > file
> > > > > for
> > > > > > > > Flink 1.x?
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Jane
> > > > > > > >
> > > > > > > > On Wed, Sep 20, 2023 at 11:06 AM Junrui Lee <
> > jrlee@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi devs,
> > > > > > > > >
> > > > > > > > > I would like to start a discussion about FLIP-366:
> > > > > > > > > Support standard YAML for FLINK configuration[1]
> > > > > > > > >
> > > > > > > > > The current flink-conf.yaml parser in FLINK is not a
> standard
> > > YAML
> > > > > > > > parser,
> > > > > > > > > which has some shortcomings.
> > > > > > > > > Firstly, it does not support nested structure configuration
> > > items
> > > > > and
> > > > > > > > only
> > > > > > > > > supports key-value pairs, resulting in poor readability.
> > > Secondly,
> > > > > if
> > > > > > > the
> > > > > > > > > value is a collection type, such as a List or Map, users
> are
> > > > > required
> > > > > > > to
> > > > > > > > > write the value in a FLINK-specific pattern, which is
> > > inconvenient
> > > > > to
> > > > > > > > use.
> > > > > > > > > Additionally, the parser of FLINK has some differences in
> > > syntax
> > > > > > > compared
> > > > > > > > > to the standard YAML parser, such as the syntax for parsing
> > > > > comments
> > > > > > > and
> > > > > > > > > null values. These inconsistencies can cause confusion for
> > > users,
> > > > > as
> > > > > > > seen
> > > > > > > > > in FLINK-15358 and FLINK-32740.
> > > > > > > > >
> > > > > > > > > By supporting standard YAML, these issues can be resolved,
> > and
> > > > > users
> > > > > > > can
> > > > > > > > > create a Flink configuration file using third-party tools
> and
> > > > > > leverage
> > > > > > > > > some advanced YAML features. Therefore, we propose to
> support
> > > > > > standard
> > > > > > > > > YAML for FLINK configuration.
> > > > > > > > >
> > > > > > > > > You can find more details in the FLIP-366[1]. Looking
> forward
> > > to
> > > > > your
> > > > > > > > > feedback.
> > > > > > > > >
> > > > > > > > > [1]
> > >

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

[Yuxin Tan]

Congratulations! Thanks for the efforts.


Best,
Yuxin


Samrat Deb  于2024年3月21日周四 20:28写道：

> Congratulations !
>
> Bests
> Samrat
>
> On Thu, 21 Mar 2024 at 5:52 PM, Ahmed Hamdy  wrote:
>
> > Congratulations, great work and great news.
> > Best Regards
> > Ahmed Hamdy
> >
> >
> > On Thu, 21 Mar 2024 at 11:41, Benchao Li  wrote:
> >
> > > Congratulations, and thanks for the great work!
> > >
> > > Yuan Mei  于2024年3月21日周四 18:31写道：
> > > >
> > > > Thanks for driving these efforts!
> > > >
> > > > Congratulations
> > > >
> > > > Best
> > > > Yuan
> > > >
> > > > On Thu, Mar 21, 2024 at 4:35 PM Yu Li  wrote:
> > > >
> > > > > Congratulations and look forward to its further development!
> > > > >
> > > > > Best Regards,
> > > > > Yu
> > > > >
> > > > > On Thu, 21 Mar 2024 at 15:54, ConradJam 
> wrote:
> > > > > >
> > > > > > Congrattulations!
> > > > > >
> > > > > > Leonard Xu  于2024年3月20日周三 21:36写道：
> > > > > >
> > > > > > > Hi devs and users,
> > > > > > >
> > > > > > > We are thrilled to announce that the donation of Flink CDC as a
> > > > > > > sub-project of Apache Flink has completed. We invite you to
> > explore
> > > > > the new
> > > > > > > resources available:
> > > > > > >
> > > > > > > - GitHub Repository: https://github.com/apache/flink-cdc
> > > > > > > - Flink CDC Documentation:
> > > > > > > https://nightlies.apache.org/flink/flink-cdc-docs-stable
> > > > > > >
> > > > > > > After Flink community accepted this donation[1], we have
> > completed
> > > > > > > software copyright signing, code repo migration, code cleanup,
> > > website
> > > > > > > migration, CI migration and github issues migration etc.
> > > > > > > Here I am particularly grateful to Hang Ruan, Zhongqaing Gong,
> > > > > Qingsheng
> > > > > > > Ren, Jiabao Sun, LvYanquan, loserwang1024 and other
> contributors
> > > for
> > > > > their
> > > > > > > contributions and help during this process！
> > > > > > >
> > > > > > >
> > > > > > > For all previous contributors: The contribution process has
> > > slightly
> > > > > > > changed to align with the main Flink project. To report bugs or
> > > > > suggest new
> > > > > > > features, please open tickets
> > > > > > > Apache Jira (https://issues.apache.org/jira).  Note that we
> will
> > > no
> > > > > > > longer accept GitHub issues for these purposes.
> > > > > > >
> > > > > > >
> > > > > > > Welcome to explore the new repository and documentation. Your
> > > feedback
> > > > > and
> > > > > > > contributions are invaluable as we continue to improve Flink
> CDC.
> > > > > > >
> > > > > > > Thanks everyone for your support and happy exploring Flink CDC!
> > > > > > >
> > > > > > > Best，
> > > > > > > Leonard
> > > > > > > [1]
> > > https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > Best
> > > > > >
> > > > > > ConradJam
> > > > >
> > >
> > >
> > >
> > > --
> > >
> > > Best,
> > > Benchao Li
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

[Yuxin Tan]

Congratulations, Zakelly!

Best,
Yuxin


Xuannan Su  于2024年4月16日周二 10:30写道：

> Congratulations Zakelly!
>
> Best regards,
> Xuannan
>
> On Mon, Apr 15, 2024 at 4:31 PM Jing Ge 
> wrote:
> >
> > Congratulations Zakelly!
> >
> > Best regards,
> > Jing
> >
> > On Mon, Apr 15, 2024 at 4:26 PM Xia Sun  wrote:
> >
> > > Congratulations Zakelly!
> > >
> > >  Best,
> > >  Xia
> > >
> > > Leonard Xu  于2024年4月15日周一 16:16写道：
> > >
> > > > Congratulations Zakelly!
> > > >
> > > >
> > > > Best,
> > > > Leonard
> > > > > 2024年4月15日 下午3:56，Samrat Deb  写道：
> > > > >
> > > > > Congratulations Zakelly!
> > > >
> > > >
> > >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Lincoln Lee

[Yuxin Tan]

Congratulations, Lincoln!

Best,
Yuxin


Xuannan Su  于2024年4月16日周二 10:26写道：

> Congratulations, Lincoln!
>
> Best regards,
> Xuannan
>
> On Tue, Apr 16, 2024 at 10:04 AM Hang Ruan  wrote:
> >
> > Congratulations, Lincoln!
> >
> > Best,
> > Hang
> >
> > yh z  于2024年4月16日周二 09:14写道：
> >
> > > Congratulations, Lincoln!
> > >
> > > Best,
> > > Yunhong (Swuferhong)
> > >
> > >
> > > Swapnal Varma  于2024年4月15日周一 18:50写道：
> > >
> > > > Congratulations, Lincoln!
> > > >
> > > > Best,
> > > > Swapnal
> > > >
> > > >
> > > > On Mon, 15 Apr 2024, 15:16 Jacky Lau,  wrote:
> > > >
> > > > > Congratulations, Lincoln!
> > > > >
> > > > > Best,
> > > > > Jacky Lau
> > > > >
> > > > > Jinzhong Li  于2024年4月15日周一 15:45写道：
> > > > >
> > > > > > Congratulations, Lincoln!
> > > > > >
> > > > > > Best,
> > > > > > Jinzhong Li
> > > > > >
> > > > > > On Mon, Apr 15, 2024 at 2:56 PM Hangxiang Yu <
> master...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Congratulations, Lincoln!
> > > > > > >
> > > > > > > On Mon, Apr 15, 2024 at 10:17 AM Zakelly Lan <
> > > zakelly@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Congratulations, Lincoln!
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Zakelly
> > > > > > > >
> > > > > > > > On Sat, Apr 13, 2024 at 12:48 AM Ferenc Csaky
> > > > > >  > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Congratulations, Lincoln!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Ferenc
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Friday, April 12th, 2024 at 15:54,
> > > > > lorenzo.affe...@ververica.com
> > > > > > > > .INVALID
> > > > > > > > >  wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Huge congrats! Well done!
> > > > > > > > > > On Apr 12, 2024 at 13:56 +0200, Ron liu
> ron9@gmail.com,
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Congratulations, Lincoln!
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Ron
> > > > > > > > > > >
> > > > > > > > > > > Junrui Lee jrlee@gmail.com 于2024年4月12日周五 18:54写道：
> > > > > > > > > > >
> > > > > > > > > > > > Congratulations, Lincoln!
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Junrui
> > > > > > > > > > > >
> > > > > > > > > > > > Aleksandr Pilipenko z3d...@gmail.com 于2024年4月12日周五
> > > > 18:29写道：
> > > > > > > > > > > >
> > > > > > > > > > > > > > Congratulations, Lincoln!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best Regards
> > > > > > > > > > > > > > Aleksandr
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best,
> > > > > > > Hangxiang.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Jing Ge

[Yuxin Tan]

Congratulations, Jing!

Best,
Yuxin


Danny Cranmer  于2024年4月15日周一 20:26写道：

> Congrats Jing!
>
> Best Regards,
> Danny
>
> On Mon, Apr 15, 2024 at 11:51 AM Swapnal Varma 
> wrote:
>
> > Congratulations, Jing!
> >
> > Best,
> > Swapnal
> >
> > On Mon, 15 Apr 2024, 15:14 Jacky Lau,  wrote:
> >
> > > Congratulations, Jing!
> > >
> > > Best,
> > > Jacky Lau
> > >
> >
>

Re: [VOTE] FLIP-442: General Improvement to Configuration for Flink 2.0

[Yuxin Tan]

+1 (non-binding)

Best,
Yuxin


Zakelly Lan  于2024年4月17日周三 16:51写道：

> +1 binding
>
>
> Best,
> Zakelly
>
> On Wed, Apr 17, 2024 at 2:05 PM Rui Fan <1996fan...@gmail.com> wrote:
>
> > +1(binding)
> >
> > Best,
> > Rui
> >
> > On Wed, Apr 17, 2024 at 1:02 PM Xuannan Su 
> wrote:
> >
> > > Hi everyone,
> > >
> > > Thanks for all the feedback about the FLIP-442: General Improvement to
> > > Configuration for Flink 2.0 [1] [2].
> > >
> > > I'd like to start a vote for it. The vote will be open for at least 72
> > > hours(excluding weekends,until APR 22, 12:00AM GMT) unless there is an
> > > objection or an insufficient number of votes.
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-442%3A+General+Improvement+to+Configuration+for+Flink+2.0
> > > [2] https://lists.apache.org/thread/15k0stwyoknhxvd643ctwjw3fd17pqwk
> > >
> > >
> > > Best regards,
> > > Xuannan
> > >
> >
>

Re: [DISCUSSION] FLIP-450: Improve Runtime Configuration for Flink 2.0

[Yuxin Tan]

Thanks for the effort, Xuannan.

+1 for the proposal.

Best,
Yuxin


Xintong Song  于2024年4月29日周一 15:40写道：

> Thanks for driving this effort, Xuannan.
>
> +1 for the proposed changes.
>
> Just one suggestion: Some of the proposed changes involve not solely
> changing the configuration options, but are bound to changing / removal of
> certain features. E.g., the removal of hash-blocking shuffle and legacy
> hybrid shuffle mode, and the behavior change of overdraft network buffers.
> Therefore, it might be nicer to provide an implementation plan with a list
> of related tasks in the FLIP. This should not block the FLIP though.
>
> Best,
>
> Xintong
>
>
>
> On Thu, Apr 25, 2024 at 4:35 PM Xuannan Su  wrote:
>
> > Hi all,
> >
> > I'd like to start a discussion on FLIP-450: Improve Runtime
> > Configuration for Flink 2.0 [1]. As Flink moves toward 2.0, we have
> > revisited all runtime configurations and identified several
> > improvements to enhance user-friendliness and maintainability. In this
> > FLIP, we aim to refine the runtime configuration.
> >
> > Looking forward to everyone's feedback and suggestions. Thank you!
> >
> > Best regards,
> > Xuannan
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-450%3A+Improve+Runtime+Configuration+for+Flink+2.0
> >
>

[DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan]

Hi all,

I would like to start a discussion on FLIP-459 Support Flink hybrid shuffle
integration with
Apache Celeborn[1]. Flink hybrid shuffle supports transitions between
memory, disk, and
remote storage to improve performance and job stability. Concurrently,
Apache Celeborn
provides a stable, performant, scalable remote shuffle service. This
integration proposal is to
harness the benefits from both hybrid shuffle and Celeborn simultaneously.

Note that this proposal has two parts.
1. The Flink-side modifications are in FLIP-459[1].
2. The Celeborn-side changes are in CIP-6[2].

Looking forward to everyone's feedback and suggestions. Thank you!

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
[2]
https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn

Best,
Yuxin

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan]

Hi, Xintong,

>  I think we can also publish the prototype codes so the
community can better understand and help with it.

Ok, I agree on the point. I will prepare and publish the code
recently.

Rui,

> Kindly reminder: the image of CIP-6[1] cannot be loaded.

Thanks for the reminder. I've updated the images.


Best,
Yuxin


Rui Fan <1996fan...@gmail.com> 于2024年5月29日周三 09:33写道：

> Thanks Yuxin for driving this proposal!
>
> Kindly reminder: the image of CIP-6[1] cannot be loaded.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
>
> Best,
> Rui
>
> On Wed, May 29, 2024 at 9:03 AM Xintong Song 
> wrote:
>
> > +1 for this proposal.
> >
> > We have been prototyping this feature internally at Alibaba for a couple
> of
> > months. Yuxin, I think we can also publish the prototype codes so the
> > community can better understand and help with it.
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Tue, May 28, 2024 at 8:34 PM Yuxin Tan 
> wrote:
> >
> > > Hi all,
> > >
> > > I would like to start a discussion on FLIP-459 Support Flink hybrid
> > shuffle
> > > integration with
> > > Apache Celeborn[1]. Flink hybrid shuffle supports transitions between
> > > memory, disk, and
> > > remote storage to improve performance and job stability. Concurrently,
> > > Apache Celeborn
> > > provides a stable, performant, scalable remote shuffle service. This
> > > integration proposal is to
> > > harness the benefits from both hybrid shuffle and Celeborn
> > simultaneously.
> > >
> > > Note that this proposal has two parts.
> > > 1. The Flink-side modifications are in FLIP-459[1].
> > > 2. The Celeborn-side changes are in CIP-6[2].
> > >
> > > Looking forward to everyone's feedback and suggestions. Thank you!
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
> > > [2]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
> > >
> > > Best,
> > > Yuxin
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Weijie Guo

[Yuxin Tan]

Congratulations, Weijie!

Best,
Yuxin


Yuepeng Pan  于2024年6月4日周二 14:57写道：

> Congratulations !
>
>
> Best,
> Yuepeng Pan
>
> At 2024-06-04 14:45:45, "Xintong Song"  wrote:
> >Hi everyone,
> >
> >On behalf of the PMC, I'm very happy to announce that Weijie Guo has
> joined
> >the Flink PMC!
> >
> >Weijie has been an active member of the Apache Flink community for many
> >years. He has made significant contributions in many components, including
> >runtime, shuffle, sdk, connectors, etc. He has driven / participated in
> >many FLIPs, authored and reviewed hundreds of PRs, been consistently
> active
> >on mailing lists, and also helped with release management of 1.20 and
> >several other bugfix releases.
> >
> >Congratulations and welcome Weijie!
> >
> >Best,
> >
> >Xintong (on behalf of the Flink PMC)
>

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan]

Hi, Venkatakrishnan,

Thanks for joining the discussion. We appreciate your interest
in contributing to the work. Once the FLIP and CIP proposals
have been approved, we will create some JIRA tickets in Flink
and Celeborn projects. Please feel free to take a look at the
tickets and select any that resonate with your interests.

Best,
Yuxin


Venkatakrishnan Sowrirajan  于2024年5月31日周五 23:11写道：

> Thanks for this FLIP. We are also interested in learning/contributing to
> the hybrid shuffle integration with celeborn for batch executions.
>
> On Tue, May 28, 2024, 7:07 PM Yuxin Tan  wrote:
>
> > Hi, Xintong,
> >
> > >  I think we can also publish the prototype codes so the
> > community can better understand and help with it.
> >
> > Ok, I agree on the point. I will prepare and publish the code
> > recently.
> >
> > Rui,
> >
> > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> >
> > Thanks for the reminder. I've updated the images.
> >
> >
> > Best,
> > Yuxin
> >
> >
> > Rui Fan <1996fan...@gmail.com> 于2024年5月29日周三 09:33写道：
> >
> > > Thanks Yuxin for driving this proposal!
> > >
> > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> > >
> > > [1]
> > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > >
> > > Best,
> > > Rui
> > >
> > > On Wed, May 29, 2024 at 9:03 AM Xintong Song 
> > > wrote:
> > >
> > > > +1 for this proposal.
> > > >
> > > > We have been prototyping this feature internally at Alibaba for a
> > couple
> > > of
> > > > months. Yuxin, I think we can also publish the prototype codes so the
> > > > community can better understand and help with it.
> > > >
> > > > Best,
> > > >
> > > > Xintong
> > > >
> > > >
> > > >
> > > > On Tue, May 28, 2024 at 8:34 PM Yuxin Tan 
> > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I would like to start a discussion on FLIP-459 Support Flink hybrid
> > > > shuffle
> > > > > integration with
> > > > > Apache Celeborn[1]. Flink hybrid shuffle supports transitions
> between
> > > > > memory, disk, and
> > > > > remote storage to improve performance and job stability.
> > Concurrently,
> > > > > Apache Celeborn
> > > > > provides a stable, performant, scalable remote shuffle service.
> This
> > > > > integration proposal is to
> > > > > harness the benefits from both hybrid shuffle and Celeborn
> > > > simultaneously.
> > > > >
> > > > > Note that this proposal has two parts.
> > > > > 1. The Flink-side modifications are in FLIP-459[1].
> > > > > 2. The Celeborn-side changes are in CIP-6[2].
> > > > >
> > > > > Looking forward to everyone's feedback and suggestions. Thank you!
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-459*3A*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;JSsrKysrKysr!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9MaOGE7hQ$
> > > > > [2]
> > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > > > >
> > > > > Best,
> > > > > Yuxin
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

[Yuxin Tan]

Congratulations, Rui!

Best,
Yuxin


Xuannan Su  于2024年6月6日周四 09:58写道：

> Congratulations!
>
> Best regards,
> Xuannan
>
> On Thu, Jun 6, 2024 at 9:53 AM Hangxiang Yu  wrote:
> >
> > Congratulations, Rui !
> >
> > On Thu, Jun 6, 2024 at 9:18 AM Lincoln Lee 
> wrote:
> >
> > > Congratulations, Rui!
> > >
> > > Best,
> > > Lincoln Lee
> > >
> > >
> > > Lijie Wang  于2024年6月6日周四 09:11写道：
> > >
> > > > Congratulations, Rui!
> > > >
> > > > Best,
> > > > Lijie
> > > >
> > > > Rodrigo Meneses  于2024年6月5日周三 21:35写道：
> > > >
> > > > > All the best
> > > > >
> > > > > On Wed, Jun 5, 2024 at 5:56 AM xiangyu feng 
> > > > wrote:
> > > > >
> > > > > > Congratulations, Rui!
> > > > > >
> > > > > > Regards,
> > > > > > Xiangyu Feng
> > > > > >
> > > > > > Feng Jin  于2024年6月5日周三 20:42写道：
> > > > > >
> > > > > > > Congratulations, Rui!
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Feng Jin
> > > > > > >
> > > > > > > On Wed, Jun 5, 2024 at 8:23 PM Yanfei Lei  >
> > > > wrote:
> > > > > > >
> > > > > > > > Congratulations, Rui!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Yanfei
> > > > > > > >
> > > > > > > > Luke Chen  于2024年6月5日周三 20:08写道：
> > > > > > > > >
> > > > > > > > > Congrats, Rui!
> > > > > > > > >
> > > > > > > > > Luke
> > > > > > > > >
> > > > > > > > > On Wed, Jun 5, 2024 at 8:02 PM Jiabao Sun <
> > > jiabao...@apache.org>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Congrats, Rui. Well-deserved!
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Jiabao
> > > > > > > > > >
> > > > > > > > > > Zhanghao Chen  于2024年6月5日周三
> > > > 19:29写道：
> > > > > > > > > >
> > > > > > > > > > > Congrats, Rui!
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Zhanghao Chen
> > > > > > > > > > > 
> > > > > > > > > > > From: Piotr Nowojski 
> > > > > > > > > > > Sent: Wednesday, June 5, 2024 18:01
> > > > > > > > > > > To: dev ; rui fan <
> > > > 1996fan...@gmail.com>
> > > > > > > > > > > Subject: [ANNOUNCE] New Apache Flink PMC Member - Fan
> Rui
> > > > > > > > > > >
> > > > > > > > > > > Hi everyone,
> > > > > > > > > > >
> > > > > > > > > > > On behalf of the PMC, I'm very happy to announce
> another
> > > new
> > > > > > Apache
> > > > > > > > Flink
> > > > > > > > > > > PMC Member - Fan Rui.
> > > > > > > > > > >
> > > > > > > > > > > Rui has been active in the community since August 2019.
> > > > During
> > > > > > this
> > > > > > > > time
> > > > > > > > > > he
> > > > > > > > > > > has contributed a lot of new features. Among others:
> > > > > > > > > > >   - Decoupling Autoscaler from Kubernetes Operator, and
> > > > > > supporting
> > > > > > > > > > > Standalone Autoscaler
> > > > > > > > > > >   - Improvements to checkpointing, flamegraphs, restart
> > > > > > strategies,
> > > > > > > > > > > watermark alignment, network shuffles
> > > > > > > > > > >   - Optimizing the memory and CPU usage of large
> operators,
> > > > > > greatly
> > > > > > > > > > > reducing the risk and probability of TaskManager OOM
> > > > > > > > > > >
> > > > > > > > > > > He reviewed a significant amount of PRs and has been
> active
> > > > > both
> > > > > > on
> > > > > > > > the
> > > > > > > > > > > mailing lists and in Jira helping to both maintain and
> grow
> > > > > > Apache
> > > > > > > > > > Flink's
> > > > > > > > > > > community. He is also our current Flink 1.20 release
> > > manager.
> > > > > > > > > > >
> > > > > > > > > > > In the last 12 months, Rui has been the most active
> > > > contributor
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > Flink Kubernetes Operator project, while being the 2nd
> most
> > > > > > active
> > > > > > > > Flink
> > > > > > > > > > > contributor at the same time.
> > > > > > > > > > >
> > > > > > > > > > > Please join me in welcoming and congratulating Fan Rui!
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Piotrek (on behalf of the Flink PMC)
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Best,
> > Hangxiang.
>

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan]

Thanks Junrui for your question.

> I wonder if the current interface design support the
future adaptation for batch job recovery

I noticed that FLIP-383 supports batch job recovery by introducing
some new APIs. These APIs can also be added to the Tier-related
interfaces to facilitate the feature. Since these modifications are not
directly related to the current integration tasks and the integration
does not conflict with the batch job recovery, I propose that this FLIP
doesn't involve these particular changes. Moreover, considering that
the Tier interfaces are not public currently, it is also feasible to add
the interfaces directly if necessary.
WDYT?

Best,
Yuxin


Junrui Lee  于2024年6月6日周四 11:02写道：

> Thanks Yuxin for driving this proposal!
>
> I have a question about the public interface compatibility in the context
> of FLIP-459. As we've supported batch job recovery from jobMaster failures
> in FLIP-383 which will be released in Flink 1.20. I wonder if the current
> interface design support the future adaptation for batch job recovery?
>
> Looking forward to your feedback.
>
> Best,
> Junrui.
>
> weijie guo  于2024年6月5日周三 10:13写道：
>
> > Thanks Yuxin for the proposal!
> >
> > When we first proposed Hybrid Shuffle, I wanted to support pluggable
> > storage tier in the future. However, limited by the architecture of the
> > legacy Hybrid Shuffle at that time, this idea has not been realized. The
> > new architecture abstracts the tier nicely, and now it's time to
> introduce
> > support for external storage.
> >
> > Big +1 for this one!
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > rexxiong  于2024年6月5日周三 00:08写道：
> >
> > > Thanks Yuxin for the proposal. +1,  as a member of the Apache Celeborn
> > > community, I am very excited about the integration of Flink's Hybrid
> > > Shuffle with Apache Celeborn. The whole design of CIP-6 looks good to
> > me. I
> > > am looking forward to this integration.
> > >
> > > Thanks,
> > > Jiashu Xiong
> > >
> > > Ethan Feng  于2024年6月4日周二 16:47写道：
> > >
> > > > +1 for this proposal.
> > > >
> > > > After internally reviewing the prototype of CIP-6, this would improve
> > > > performance and stability for Flink users using Celeborn.
> > > >
> > > > Expect to see this feature come out to the community.
> > > >
> > > > As I come from the Celeborn community, I hope more users can try to
> > > > use Celeborn when there are Flink batch jobs.
> > > >
> > > > Thanks,
> > > > Ethan Feng
> > > >
> > > > Yuxin Tan  于2024年6月4日周二 16:34写道：
> > > > >
> > > > > Hi, Venkatakrishnan,
> > > > >
> > > > > Thanks for joining the discussion. We appreciate your interest
> > > > > in contributing to the work. Once the FLIP and CIP proposals
> > > > > have been approved, we will create some JIRA tickets in Flink
> > > > > and Celeborn projects. Please feel free to take a look at the
> > > > > tickets and select any that resonate with your interests.
> > > > >
> > > > > Best,
> > > > > Yuxin
> > > > >
> > > > >
> > > > > Venkatakrishnan Sowrirajan  于2024年5月31日周五
> 23:11写道：
> > > > >
> > > > > > Thanks for this FLIP. We are also interested in
> > learning/contributing
> > > > to
> > > > > > the hybrid shuffle integration with celeborn for batch
> executions.
> > > > > >
> > > > > > On Tue, May 28, 2024, 7:07 PM Yuxin Tan 
> > > > wrote:
> > > > > >
> > > > > > > Hi, Xintong,
> > > > > > >
> > > > > > > >  I think we can also publish the prototype codes so the
> > > > > > > community can better understand and help with it.
> > > > > > >
> > > > > > > Ok, I agree on the point. I will prepare and publish the code
> > > > > > > recently.
> > > > > > >
> > > > > > > Rui,
> > > > > > >
> > > > > > > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> > > > > > >
> > > > > > > Thanks for the reminder. I've updated the images.
> > > > > > >
> > > > > > >
> > > > > > > Best,
> &g

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan]

Thanks Zhu for the suggestion.
I have updated the description of the option.

Best,
Yuxin


Zhu Zhu  于2024年6月6日周四 14:59写道：

> +1
>
> Maybe explain in the description of
> `taskmanager.network.hybrid-shuffle.external-remote-tier-factory.class`
> that it only accepts Celeborn as the remote shuffle tier at this moment?
>
> Thanks,
> Zhu
>
> Junrui Lee  于2024年6月6日周四 13:49写道：
>
> > Thanks Yuxin for your answer. +1 for this proposal.
> >
> > Best,
> > Junrui.
> >
> > Yuxin Tan  于2024年6月6日周四 13:42写道：
> >
> > > Thanks Junrui for your question.
> > >
> > > > I wonder if the current interface design support the
> > > future adaptation for batch job recovery
> > >
> > > I noticed that FLIP-383 supports batch job recovery by introducing
> > > some new APIs. These APIs can also be added to the Tier-related
> > > interfaces to facilitate the feature. Since these modifications are not
> > > directly related to the current integration tasks and the integration
> > > does not conflict with the batch job recovery, I propose that this FLIP
> > > doesn't involve these particular changes. Moreover, considering that
> > > the Tier interfaces are not public currently, it is also feasible to
> add
> > > the interfaces directly if necessary.
> > > WDYT?
> > >
> > > Best,
> > > Yuxin
> > >
> > >
> > > Junrui Lee  于2024年6月6日周四 11:02写道：
> > >
> > > > Thanks Yuxin for driving this proposal!
> > > >
> > > > I have a question about the public interface compatibility in the
> > context
> > > > of FLIP-459. As we've supported batch job recovery from jobMaster
> > > failures
> > > > in FLIP-383 which will be released in Flink 1.20. I wonder if the
> > current
> > > > interface design support the future adaptation for batch job
> recovery?
> > > >
> > > > Looking forward to your feedback.
> > > >
> > > > Best,
> > > > Junrui.
> > > >
> > > > weijie guo  于2024年6月5日周三 10:13写道：
> > > >
> > > > > Thanks Yuxin for the proposal!
> > > > >
> > > > > When we first proposed Hybrid Shuffle, I wanted to support
> pluggable
> > > > > storage tier in the future. However, limited by the architecture of
> > the
> > > > > legacy Hybrid Shuffle at that time, this idea has not been
> realized.
> > > The
> > > > > new architecture abstracts the tier nicely, and now it's time to
> > > > introduce
> > > > > support for external storage.
> > > > >
> > > > > Big +1 for this one!
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Weijie
> > > > >
> > > > >
> > > > > rexxiong  于2024年6月5日周三 00:08写道：
> > > > >
> > > > > > Thanks Yuxin for the proposal. +1,  as a member of the Apache
> > > Celeborn
> > > > > > community, I am very excited about the integration of Flink's
> > Hybrid
> > > > > > Shuffle with Apache Celeborn. The whole design of CIP-6 looks
> good
> > to
> > > > > me. I
> > > > > > am looking forward to this integration.
> > > > > >
> > > > > > Thanks,
> > > > > > Jiashu Xiong
> > > > > >
> > > > > > Ethan Feng  于2024年6月4日周二 16:47写道：
> > > > > >
> > > > > > > +1 for this proposal.
> > > > > > >
> > > > > > > After internally reviewing the prototype of CIP-6, this would
> > > improve
> > > > > > > performance and stability for Flink users using Celeborn.
> > > > > > >
> > > > > > > Expect to see this feature come out to the community.
> > > > > > >
> > > > > > > As I come from the Celeborn community, I hope more users can
> try
> > to
> > > > > > > use Celeborn when there are Flink batch jobs.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Ethan Feng
> > > > > > >
> > > > > > > Yuxin Tan  于2024年6月4日周二 16:34写道：
> > > > > > > >
> > > > > > > > Hi, Venkatakrishnan,
> > > > > > > >
> > > > &g

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan]

Hi all,

Thanks for all the feedback and suggestions so far.

If there is no further comment, we will open the voting thread tomorrow.

Best,
Yuxin


Yuxin Tan  于2024年6月6日周四 15:40写道：

> Thanks Zhu for the suggestion.
> I have updated the description of the option.
>
> Best,
> Yuxin
>
>
> Zhu Zhu  于2024年6月6日周四 14:59写道：
>
>> +1
>>
>> Maybe explain in the description of
>> `taskmanager.network.hybrid-shuffle.external-remote-tier-factory.class`
>> that it only accepts Celeborn as the remote shuffle tier at this moment?
>>
>> Thanks,
>> Zhu
>>
>> Junrui Lee  于2024年6月6日周四 13:49写道：
>>
>> > Thanks Yuxin for your answer. +1 for this proposal.
>> >
>> > Best,
>> > Junrui.
>> >
>> > Yuxin Tan  于2024年6月6日周四 13:42写道：
>> >
>> > > Thanks Junrui for your question.
>> > >
>> > > > I wonder if the current interface design support the
>> > > future adaptation for batch job recovery
>> > >
>> > > I noticed that FLIP-383 supports batch job recovery by introducing
>> > > some new APIs. These APIs can also be added to the Tier-related
>> > > interfaces to facilitate the feature. Since these modifications are
>> not
>> > > directly related to the current integration tasks and the integration
>> > > does not conflict with the batch job recovery, I propose that this
>> FLIP
>> > > doesn't involve these particular changes. Moreover, considering that
>> > > the Tier interfaces are not public currently, it is also feasible to
>> add
>> > > the interfaces directly if necessary.
>> > > WDYT?
>> > >
>> > > Best,
>> > > Yuxin
>> > >
>> > >
>> > > Junrui Lee  于2024年6月6日周四 11:02写道：
>> > >
>> > > > Thanks Yuxin for driving this proposal!
>> > > >
>> > > > I have a question about the public interface compatibility in the
>> > context
>> > > > of FLIP-459. As we've supported batch job recovery from jobMaster
>> > > failures
>> > > > in FLIP-383 which will be released in Flink 1.20. I wonder if the
>> > current
>> > > > interface design support the future adaptation for batch job
>> recovery?
>> > > >
>> > > > Looking forward to your feedback.
>> > > >
>> > > > Best,
>> > > > Junrui.
>> > > >
>> > > > weijie guo  于2024年6月5日周三 10:13写道：
>> > > >
>> > > > > Thanks Yuxin for the proposal!
>> > > > >
>> > > > > When we first proposed Hybrid Shuffle, I wanted to support
>> pluggable
>> > > > > storage tier in the future. However, limited by the architecture
>> of
>> > the
>> > > > > legacy Hybrid Shuffle at that time, this idea has not been
>> realized.
>> > > The
>> > > > > new architecture abstracts the tier nicely, and now it's time to
>> > > > introduce
>> > > > > support for external storage.
>> > > > >
>> > > > > Big +1 for this one!
>> > > > >
>> > > > > Best regards,
>> > > > >
>> > > > > Weijie
>> > > > >
>> > > > >
>> > > > > rexxiong  于2024年6月5日周三 00:08写道：
>> > > > >
>> > > > > > Thanks Yuxin for the proposal. +1,  as a member of the Apache
>> > > Celeborn
>> > > > > > community, I am very excited about the integration of Flink's
>> > Hybrid
>> > > > > > Shuffle with Apache Celeborn. The whole design of CIP-6 looks
>> good
>> > to
>> > > > > me. I
>> > > > > > am looking forward to this integration.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Jiashu Xiong
>> > > > > >
>> > > > > > Ethan Feng  于2024年6月4日周二 16:47写道：
>> > > > > >
>> > > > > > > +1 for this proposal.
>> > > > > > >
>> > > > > > > After internally reviewing the prototype of CIP-6, this would
>> > > improve
>> > > > > > > performance and stability for Flink users using Celeborn.
>> > > > > > >
>> > > > > > > Expect to s

[VOTE] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan]

Hi everyone,

Thanks for all the feedback about the FLIP-459 Support Flink
hybrid shuffle integration with Apache Celeborn[1].
The discussion thread is here [2].

I'd like to start a vote for it. The vote will be open for at least
72 hours unless there is an objection or insufficient votes.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
[2] https://lists.apache.org/thread/gy7sm7qrf7yrv1rl5f4vtk5fo463ts33

Best,
Yuxin

Re: [VOTE] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan]

Hi all,

Thanks for all your votes, I hereby close the vote and I'll announce
the results in a separate email.

Best,
Yuxin


Zhu Zhu  于2024年6月12日周三 11:15写道：

> +1 (binding)
>
> Thanks,
> Zhu
>
> Yuepeng Pan  于2024年6月11日周二 17:04写道：
>
> > +1 (non-binding)
> >
> > Best regards,
> > Yuepeng Pan
> >
> > At 2024-06-11 16:34:12, "Rui Fan" <1996fan...@gmail.com> wrote:
> > >+1（binding）
> > >
> > >Best,
> > >Rui
> > >
> > >On Tue, Jun 11, 2024 at 4:14 PM Muhammet Orazov
> > > wrote:
> > >
> > >> +1 (non-binding)
> > >>
> > >> Thanks Yuxin for driving this!
> > >>
> > >> Best,
> > >> Muhammet
> > >>
> > >>
> > >> On 2024-06-07 08:02, Yuxin Tan wrote:
> > >> > Hi everyone,
> > >> >
> > >> > Thanks for all the feedback about the FLIP-459 Support Flink
> > >> > hybrid shuffle integration with Apache Celeborn[1].
> > >> > The discussion thread is here [2].
> > >> >
> > >> > I'd like to start a vote for it. The vote will be open for at least
> > >> > 72 hours unless there is an objection or insufficient votes.
> > >> >
> > >> > [1]
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
> > >> > [2]
> https://lists.apache.org/thread/gy7sm7qrf7yrv1rl5f4vtk5fo463ts33
> > >> >
> > >> > Best,
> > >> > Yuxin
> > >>
> >
>

[RESULT][VOTE] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan]

Hi, all

I am happy to say that FLIP-459 Support Flink
hybrid shuffle integration with Apache Celeborn [1]
has been accepted.

There are 11 votes, of which 4 are binding[2].

Xintong Song (binding)
Zhu Zhu (binding)
weijie guo (binding)
Jim Hughes (non-binding)
Jeyhun Karimov (non-binding)
Ahmed Hamdy (non-binding)
Venkatakrishnan Sowrirajan (non-binding)
Junrui Lee (non-binding)
Muhammet Orazov (non-binding)
Rui Fan (binding)
Yuepeng Pan (non-binding)

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
[2] https://lists.apache.org/thread/l64ykk3n8c2gc40gjbowt0ozs0x0jmqm

Best,
Yuxin

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan]

Hi, Venkatakrishnan

The current proposal is designed to initialize a single remote
tier, such as Celeborn, if the remote tier factory is configured.
However, this is a temporary approach because the tier interfaces
are not stable. Our aim is to eventually support multiple storage
tiers: memory tier, disk tier, and remote tier. The remote tier
could be cloud storage or a remote shuffle service like Celeborn.

Initially, we don't expect to need both a cloud storage and a
remote shuffle service as two remote tiers concurrently due to
their large capacities. However, we're open and receptive to
adding both tiers concurrently if future discussions suggest it's
necessary. The architecture is designed for flexibility, allowing
each tier to function independently and be easily added to the
tiered storage with minimal modifications.

Best,
Yuxin


Venkatakrishnan Sowrirajan  于2024年6月17日周一 15:30写道：

> Yuxin,
>
> One question, in the current proposal is it limited to only one tiered
> storage implementation for eg: celeborn? Is it possible to have multiple
> tiered storages like a separate cloud storage and Celeborn or some such?
>
>
> On Thu, Jun 6, 2024, 9:46 AM Jeyhun Karimov  wrote:
>
> > Hi Yuxin,
> >
> > +1 for this proposal.
> > This change will greatly alleviate the pressure on local storage
> resources
> > (especially when there is limited local storage)
> > particularly in the context of cloud-native environments.
> >
> > Regards,
> > Jeyhun
> >
> > On Thu, Jun 6, 2024 at 1:20 PM Yuxin Tan  wrote:
> >
> > > Hi all,
> > >
> > > Thanks for all the feedback and suggestions so far.
> > >
> > > If there is no further comment, we will open the voting thread
> tomorrow.
> > >
> > > Best,
> > > Yuxin
> > >
> > >
> > > Yuxin Tan  于2024年6月6日周四 15:40写道：
> > >
> > > > Thanks Zhu for the suggestion.
> > > > I have updated the description of the option.
> > > >
> > > > Best,
> > > > Yuxin
> > > >
> > > >
> > > > Zhu Zhu  于2024年6月6日周四 14:59写道：
> > > >
> > > >> +1
> > > >>
> > > >> Maybe explain in the description of
> > > >>
> > `taskmanager.network.hybrid-shuffle.external-remote-tier-factory.class`
> > > >> that it only accepts Celeborn as the remote shuffle tier at this
> > moment?
> > > >>
> > > >> Thanks,
> > > >> Zhu
> > > >>
> > > >> Junrui Lee  于2024年6月6日周四 13:49写道：
> > > >>
> > > >> > Thanks Yuxin for your answer. +1 for this proposal.
> > > >> >
> > > >> > Best,
> > > >> > Junrui.
> > > >> >
> > > >> > Yuxin Tan  于2024年6月6日周四 13:42写道：
> > > >> >
> > > >> > > Thanks Junrui for your question.
> > > >> > >
> > > >> > > > I wonder if the current interface design support the
> > > >> > > future adaptation for batch job recovery
> > > >> > >
> > > >> > > I noticed that FLIP-383 supports batch job recovery by
> introducing
> > > >> > > some new APIs. These APIs can also be added to the Tier-related
> > > >> > > interfaces to facilitate the feature. Since these modifications
> > are
> > > >> not
> > > >> > > directly related to the current integration tasks and the
> > > integration
> > > >> > > does not conflict with the batch job recovery, I propose that
> this
> > > >> FLIP
> > > >> > > doesn't involve these particular changes. Moreover, considering
> > that
> > > >> > > the Tier interfaces are not public currently, it is also
> feasible
> > to
> > > >> add
> > > >> > > the interfaces directly if necessary.
> > > >> > > WDYT?
> > > >> > >
> > > >> > > Best,
> > > >> > > Yuxin
> > > >> > >
> > > >> > >
> > > >> > > Junrui Lee  于2024年6月6日周四 11:02写道：
> > > >> > >
> > > >> > > > Thanks Yuxin for driving this proposal!
> > > >> > > >
> > > >> > > > I have a question about the public interface compatibility in
> > the
> > > >> > context
> > >

Re: [VOTE] Release flink-shaded 19.0, release candidate #1

[Yuxin Tan]

+1 (non-binding)

- build from source
- verified website pr
- verified hash and signature

Best,
Yuxin


weijie guo  于2024年7月2日周二 15:27写道：

> +1 (binding)
>
> - Verified hash and signature
> - Build from source
> - Verified website PR
>
> Best regards,
>
> Weijie
>
>
> Martijn Visser  于2024年7月1日周一 21:30写道：
>
> > +1 (binding)
> >
> > - Validated hashes
> > - Verified signature
> > - Verified that no binaries exist in the source archive
> > - Build the source with Maven
> > - Verified licenses
> > - Verified web PRs
> >
> > On Fri, Jun 28, 2024 at 2:02 PM Timo Walther  wrote:
> >
> > > +1 (binding)
> > >
> > > Thanks for fixing the JSON functions!
> > >
> > > Timo
> > >
> > > On 28.06.24 12:54, Dawid Wysakowicz wrote:
> > > > Hi everyone,
> > > > Please review and vote on the release candidate 1 for the version
> 19.0,
> > > as
> > > > follows:
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > >
> > > > The complete staging area is available for your review, which
> includes:
> > > > * JIRA release notes [1],
> > > > * the official Apache source release to be deployed to
> dist.apache.org
> > > [2],
> > > > which are signed with the key with fingerprint
> > > > EA93A435B4E2C9B4C9F533F631D2DD10BFC15A2D [3],
> > > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > > * source code tag release-19.0-rc1 [5],
> > > > * website pull request listing the new release [6].
> > > >
> > > > The vote will be open for at least 72 hours. It is adopted by
> majority
> > > > approval, with at least 3 PMC affirmative votes.
> > > >
> > > > Thanks,
> > > > Dawid
> > > >
> > > > [1] https://issues.apache.org/jira/projects/FLINK/versions/12353853
> > > > [2]
> https://dist.apache.org/repos/dist/dev/flink/flink-shaded-19.0-rc1
> > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > [4]
> > >
> https://repository.apache.org/content/repositories/orgapacheflink-1743/
> > > > [5]
> > https://github.com/apache/flink-shaded/releases/tag/release-19.0-rc1
> > > > [6] https://github.com/apache/flink-web/pull/749
> > > >
> > >
> > >
> >
>

Re: Re: [DISCUSS] FLIP-466: Introduce ProcessFunction Attribute in DataStream API V2

[Yuxin Tan]

Hi, Wencong,
Thanks for driving the FLIP.

+1 for this FLIP.

I believe these hints will improve the performance in many use cases.
I only have a minor question about the Idempotence annotation. When
this annotation is added, how does StreamTask optimize the frequency?
Does it ensure a single output, or does it merely reduce the frequency
of the outputs?

Best,
Yuxin


Wencong Liu  于2024年7月1日周一 16:39写道：

> Hi, Jeyhun,
> Thanks for the reply.
>
>
> > Integrate these annotations with the execution plan.
> I believe DataStream is an Imperative API, which means
> that the actual execution plan is basically consistent with
> the computational logic expressed by the user with DataStream,
> and it is different from SQL, so the significance of supporting
> getExecutionPlan in the short term may not be great. If it is to
> be supported later, it is possible to consider the impact of Hints.
>
>
> > Check for misuse of attributes or ignore it.
> For illegal use (annotated on the inappropriate ProcessFunction),
> an exception will be thrown. For legal use, the framework can also
> choose to ignore it.
>
>
> > A framework to include attributes.
> Yes, we will provide a basic framework in the implementation
> to help developers for extension.
>
>
> Best,
> Wencong
>
>
> At 2024-06-28 02:06:37, "Jeyhun Karimov"  wrote:
> >Hi Wencong,
> >
> >Thanks for the FLIP. +1 for it.
> >
> >Providing hints to users will enable more optimization potential for DSv2.
> >I have a few questions.
> >
> >I think currently, DSv2 ExecutionEnvironment does not support getting
> >execution plan (getExecutionPlan()).
> >Do you plan to integrate these annotations with the execution plan?
> >
> >Any plans to check for misuse of attributes? Or any plans for a framework
> >to implicitly include attributes?
> >
> >Also, now that we make analogy with SQL hints, SQL query planners usually
> >ignore wrong hints and continue with its best plan.
> >Do we want to consider this approach? Or should we throw exception
> whenever
> >the hint (attribute in this case) is wrong?
> >
> >
> >Regards,
> >Jeyhun
> >
> >
> >On Thu, Jun 27, 2024 at 7:47 AM Xintong Song 
> wrote:
> >
> >> +1 for this FLIP.
> >>
> >> I think this is similar to SQL hints, where users can provide optional
> >> information to help the engine execute the workload more efficiently.
> >> Having a unified mechanism for such kind of hints should improve
> usability
> >> compared to introducing tons of configuration knobs.
> >>
> >> Best,
> >>
> >> Xintong
> >>
> >>
> >>
> >> On Wed, Jun 26, 2024 at 8:09 PM Wencong Liu 
> wrote:
> >>
> >> > Hi devs,
> >> >
> >> >
> >> > I'm proposing a new FLIP[1] to introduce the ProcessFunction
> Attribute in
> >> > the
> >> > DataStream API V2. The goal is to optimize job execution by leveraging
> >> > additional information about users' ProcessFunction logic. The
> proposal
> >> > includes
> >> > the following scenarios where the ProcessFunction Attribute can
> >> > significantly
> >> > enhance optimization:
> >> >
> >> >
> >> > Scenario 1: If the framework recognizes that the ProcessFunction
> outputs
> >> > data
> >> > only after all input is received, the downstream operators can be
> >> > scheduled until
> >> > the ProcessFunction is finished, which effectively reduces resource
> >> > consumption.
> >> > Ignoring this information could lead to premature scheduling of
> >> downstream
> >> > operators with no data to process. This scenario is addressed and
> >> > optimized by FLIP-331[2].
> >> >
> >> >
> >> > Scenario 2: For stream processing, where users are only interested in
> the
> >> > latest
> >> > result per key at the current time, the framework can optimize by
> >> > adjusting the
> >> > frequency of ProcessFunction outputs. This reduces shuffle data volume
> >> and
> >> > downstream operator workload. If this optimization is ignored, each
> new
> >> > input
> >> > would trigger a new output. This scenario is addressed and
> >> > optimized by FLIP-365[3].
> >> >
> >> >
> >> > Scenario 3: If a user's ProcessFunction neither caches inputs nor
> >> outputs,
> >> > recognizing this can enable object reuse for this data within the
> >> > OperatorChain,
> >> > enhancing performance. Without this optimization, data would be copied
> >> > before
> >> > being passed to the next operator. This scenario is addressed and
> >> > optimized by FLIP-329[4].
> >> >
> >> >
> >> > To unify the mechanism for utilizing additional information and
> >> optimizing
> >> > jobs,
> >> > we propose introducing the ProcessFunction Attribute represented by
> >> > Java annotations, which allow users to provide relevant information
> about
> >> > their
> >> > ProcessFunctions. The framework can then use this to optimize job
> >> > execution.
> >> > Importantly, regular job execution remains unaffected whether users
> use
> >> > this
> >> > attribute or not.
> >> >
> >> >
> >> > Looking forward to discussing this in the upcoming FLIP.
> >> >
> >> >
> >> > Best regards,
> >> > Wen

Re: [DISCUSS] FLIP-266: Simplify network memory configurations for TaskManager

[Yuxin Tan]

ired-buffer-per-gate.max) to
> control
> > the network memory usage.
> > 1. Is this configuration at the job level or cluster level? As the FLIP
> > described, the default values of the Batch job and Stream job are
> > different, If an explicit value is set for cluster level, will it affect
> > all Batch jobs and Stream jobs on the cluster?
> >
> > 2. The default value of Batch Job depends on the value of
> >
> ExclusiveBuffersPerChannel(taskmanager.network.memory.buffers-per-channel),
> > if the value of ExclusiveBuffersPerChannel changed, does
> > "taskmanager.memory.network.required-buffer-per-gate.max" need to change
> > with it?
> >
> >
> > Best,
> > Yanfei
> >
> > Dong Lin  于2022年12月25日周日 08:58写道：
> >
> > > Hi Yuxin,
> > >
> > > Thanks for proposing the FLIP!
> > >
> > > The motivation section makes sense. But it seems that the proposed
> change
> > > section mixes the proposed config with the evaluation results. It is a
> bit
> > > hard to understand what configs are proposed and how to describe these
> > > configs to users. Given that the configuration setting is part of
> public
> > > interfaces, it might be helpful to add a dedicated public interface
> section
> > > to describe the config key and config semantics, as suggested in the
> FLIP
> > > template here
> > > <
> > >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > >
> > > .
> > >
> > > This FLIP seems to add more configs without removing any config from
> Flink.
> > > Intuitively this can make the Flink configuration harder rather than
> > > simpler. Maybe we can get a better idea after we add a public interface
> > > section to clarify those configs.
> > >
> > > Thanks,
> > > Dong
> > >
> > >
> > > On Mon, Dec 19, 2022 at 3:36 PM Yuxin Tan 
> wrote:
> > >
> > > > Hi, devs,
> > > >
> > > > I'd like to start a discussion about FLIP-266: Simplify network
> memory
> > > > configurations for TaskManager[1].
> > > >
> > > > When using Flink, users may encounter the following issues that
> affect
> > > > usability.
> > > > 1. The job may fail with an "Insufficient number of network buffers"
> > > > exception.
> > > > 2. Flink network memory size adjustment is complex.
> > > > When encountering these issues, users can solve some problems by
> adding
> > > or
> > > > adjusting parameters. However, multiple memory config options should
> be
> > > > changed. The config option adjustment requires understanding the
> detailed
> > > > internal implementation, which is impractical for most users.
> > > >
> > > > To simplify network memory configurations for TaskManager and improve
> > > Flink
> > > > usability, this FLIP proposed some optimization solutions for the
> issues.
> > > >
> > > > Looking forward to your feedback.
> > > >
> > > > [1]
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-266%3A+Simplify+network+memory+configurations+for+TaskManager
> > > >
> > > > Best regards,
> > > > Yuxin
> > > >
> > >
>

Re: [DISCUSS] FLIP-266: Simplify network memory configurations for TaskManager

[Yuxin Tan]

Hi, Weihua

Thanks for your suggestions.

> 1. How about reducing ExclusiveBuffersPerChannel to 1 first when the
total buffer is not enough?

I think it's a good idea. Will try and check the results in PoC. Before all
read buffers use floating buffers, I will try to use
(ExclusiveBuffersPerChannel - i)
buffers per channel first. For example, if the user has configured
ExclusiveBuffersPerChannel to 4, it will check whether all read buffers
are sufficient from 4 to 1. Only when ExclusiveBuffersPerChannel of
all channels is 1 and all read buffers are insufficient, all read buffers
will use floating buffers.
If the test results prove better, the FLIP will use this method.

> 2. Do we really need to change the default value of
'taskmanager.memory.network.max'?

Changing taskmanager.memory.network.max will indeed affect some
users, but the user only is affected when the 3 conditions are fulfilled.
1) Flink total TM memory is larger than 10g (because the network memory
ratio is 0.1).
2) taskmanager.memory.network.max was not initially configured.
3) Other memory, such as managed memory or heap memory, is insufficient.
I think the number of jobs fulfilling the conditions is small because when
TM
uses such a large amount of memory, the network memory requirement may
also be large. And when encountering the issue, the rollback method is very
simple,
configuring taskmanager.memory.network.max as 1g or other values.
In addition, the reason for modifying the default value is to simplify the
network
configurations in most scenarios. This change does affect a few usage
scenarios,
but we should admit that setting the default to any value may not meet
the requirements of all scenarios.

Best,
Yuxin


Weihua Hu  于2022年12月26日周一 20:35写道：

> Hi Yuxin,
> Thanks for the proposal.
>
> "Insufficient number of network buffers" exceptions also bother us. It's
> too hard for users to figure out
> how much network buffer they really need. It relates to partitioner type,
> parallelism, slots per taskmanager.
>
> Since streaming jobs are our primary scenario, I have some questions about
> streaming jobs.
>
> 1. In this FLIP, all read buffers will use floating buffers when the total
> buffer is more than
> 'taskmanager.memory.network.read-required-buffer.max'. Competition in
> buffer allocation led to preference regression.
> How about reducing ExclusiveBuffersPerChannel to 1 first when the total
> buffer is not enough?
> Will this reduce performance regression in streaming?
>
> 2. Changing taskmanager.memory.network.max will affect user migration from
> the lower version.
> IMO, network buffer size should not increase with total memory, especially
> for streaming jobs with application mode.
> For example, some ETL jobs with rescale partitioner only require a few
> network buffers.
> And we already have 'taskmanager.memory.network.read-required-buffer.max'
> to control maximum read network buffer usage.
> Do we really need to change the default value of
> 'taskmanager.memory.network.max'?
>
> Best,
> Weihua
>
>
> On Mon, Dec 26, 2022 at 6:26 PM Yuxin Tan  wrote:
>
> > Hi, all
> > Thanks for the reply and feedback for everyone!
> >
> >
> > After combining everyone's comments, the main concerns, and corresponding
> > adjustments are as follows.
> >
> >
> > @Guowei Ma, Thanks for your feedback.
> > > should we introduce a _new_ non-orthogonal
> > option(`taskmanager.memory.network.required-buffer-per-gate.max`). That
> is
> > to say, the option will affect both streaming and batch shuffle behavior
> at
> > the
> > same time.
> >
> > 1. Because the default option can meet most requirements no matter in
> > Streaming
> > or Batch scenarios. We do not want users to adjust this default config
> > option by
> > design. This configuration option is added only to preserve the
> possibility
> > of
> > modification options for users.
> > 2. In a few cases, if you really want to adjust this option, users may
> not
> > expect to
> > adjust the option according to Streaming or Batch, for example, according
> > to the
> > parallelism of the job.
> > 3. Regarding the performance of streaming shuffle, the same problem of
> > insufficient memory also exists for Streaming jobs. We introduced this
> > configuration
> > to enable users to decouple memory and parallelism, but it will affect
> some
> > performance. By default, the feature is disabled and does not affect
> > performance.
> > However, the added configuration enables users to choose to decouple
> memory
> > usage and parallelism for Streaming jobs.
> >
> > > It's better no

Re: [DISCUSS] FLIP-266: Simplify network memory configurations for TaskManager

[Yuxin Tan]

Hi, Roman

Thanks for the replay.

ExclusiveBuffersPerChannel and FloatingBuffersPerGate are obtained from
configurations, which are not calculated. I have described them in the FLIP
motivation section.

> 3. Each gate requires at least one buffer...
The timeout exception occurs when the ExclusiveBuffersPerChannel
can not be requested from NetworkBufferPool, which is not caused by the
change of this Flip. In addition, if  we have set the
ExclusiveBuffersPerChannel
to 0 when using floating buffers, which can also decrease the probability
of
this exception.

> 4. It would be great to have experimental results for jobs with different
exchange types.
Thanks for the suggestion. I have a test about different exchange types,
forward
and rescale, and the results show no differences from the all-to-all type,
which
is also understandable, because the network memory usage is calculated
with numChannels, independent of the edge type.

Best,
Yuxin


Roman Khachatryan  于2022年12月28日周三 05:27写道：

> Hi everyone,
>
> Thanks for the proposal and the discussion.
>
> I couldn't find much details on how exactly the values of
> ExclusiveBuffersPerChannel and FloatingBuffersPerGate are calculated.
> I guess that
> - the threshold evaluation is done on JM
> - floating buffers calculation is done on TM based on the current memory
> available; so it is not taking into account any future tasks submitted for
> that (or other) job
> Is that correct?
>
> If so, I see the following potential issues:
>
> 1. Each (sub)task might have different values because the actual
> available memory might be different. E.g. some tasks might use exclusive
> buffers and others only floating. That could lead to significant skew
> in processing speed, and in turn to issues with checkpoints and watermarks.
>
> 2. Re-deployment of a task (e.g. on job failure) might lead to a completely
> different memory configuration. That, coupled with different values per
> subtask and operator, makes the performance analysis more difficult.
>
> (Regardless of whether it's done on TM or JM):
> 3. Each gate requires at least one buffer [1]. So, in case when no memory
> is available, TM will throw an Allocation timeout exception instead of
> Insufficient buffers exception immediately. A delay here (allocation
> timeout) seems like a regression.
> Besides that, the regression depends on how much memory is actually
> available and how much it is contended, doesn't it?
> Should there still be a lower threshold of available memory, below which
> the job (task) isn't accepted?
> 4. The same threshold for all types of shuffles will likely result in using
> exclusive buffers
> for point-wise connections and floating buffers for all-to-all ones. I'm
> not sure if that's always optimal. It would be great to have experimental
> results for jobs with different exchange types, WDYT?
>
> [1]
> https://issues.apache.org/jira/browse/FLINK-24035
>
> Regards,
> Roman
>
>
> On Tue, Dec 27, 2022 at 4:12 AM Yuxin Tan  wrote:
>
> > Hi, Weihua
> >
> > Thanks for your suggestions.
> >
> > > 1. How about reducing ExclusiveBuffersPerChannel to 1 first when the
> > total buffer is not enough?
> >
> > I think it's a good idea. Will try and check the results in PoC. Before
> all
> > read buffers use floating buffers, I will try to use
> > (ExclusiveBuffersPerChannel - i)
> > buffers per channel first. For example, if the user has configured
> > ExclusiveBuffersPerChannel to 4, it will check whether all read buffers
> > are sufficient from 4 to 1. Only when ExclusiveBuffersPerChannel of
> > all channels is 1 and all read buffers are insufficient, all read buffers
> > will use floating buffers.
> > If the test results prove better, the FLIP will use this method.
> >
> > > 2. Do we really need to change the default value of
> > 'taskmanager.memory.network.max'?
> >
> > Changing taskmanager.memory.network.max will indeed affect some
> > users, but the user only is affected when the 3 conditions are fulfilled.
> > 1) Flink total TM memory is larger than 10g (because the network memory
> > ratio is 0.1).
> > 2) taskmanager.memory.network.max was not initially configured.
> > 3) Other memory, such as managed memory or heap memory, is insufficient.
> > I think the number of jobs fulfilling the conditions is small because
> when
> > TM
> > uses such a large amount of memory, the network memory requirement may
> > also be large. And when encountering the issue, the rollback method is
> very
> > simple,
> > configuring taskmanager.memory.network.max as 1g or other values.
> > In addition, the reason for mod

Re: [DISCUSS] FLIP-266: Simplify network memory configurations for TaskManager

[Yuxin Tan]

Hi, JasonLee

Thanks for the feedback.

> How can we determine how many RS and IG a DAG has?

Network memory is related to the parallelism and the complexity of the task
DAG, which I think is correct. However, this Flip can only improve the part
issue of the total issue, mainly focusing on memory optimization of Shuffle
reading. We can only limit the total read buffers in one InputGate, but can
not determine the number of RS and IG. The good news is that this is an
independent problem. Maybe we could try to optimize and solve that
problem later.


Best,
Yuxin


JasonLee <17610775...@163.com> 于2022年12月28日周三 19:18写道：

> Hi Yuxin
>
>
> Thanks for the proposal, big + 1 for this FLIP.
>
>
>
> It is difficult for users to calculate the size of network memory. If the
> setting is too small, the task cannot be started. If the setting is too
> large, there may be a waste of resources. As far as possible, Flink
> framework can automatically set a reasonable value, but I have a small
> problem. network memory is not only related to the parallelism of the task,
> but also to the complexity of the task DAG. The more complex a DAG is,
> shuffle write and shuffle read require larger buffers. How can we determine
> how many RS and IG a DAG has?
>
>
>
> Best
> JasonLee
>
>
>  Replied Message 
> | From | Yuxin Tan |
> | Date | 12/28/2022 18:29 |
> | To |  |
> | Subject | Re: [DISCUSS] FLIP-266: Simplify network memory configurations
> for TaskManager |
> Hi, Roman
>
> Thanks for the replay.
>
> ExclusiveBuffersPerChannel and FloatingBuffersPerGate are obtained from
> configurations, which are not calculated. I have described them in the FLIP
> motivation section.
>
> 3. Each gate requires at least one buffer...
> The timeout exception occurs when the ExclusiveBuffersPerChannel
> can not be requested from NetworkBufferPool, which is not caused by the
> change of this Flip. In addition, if  we have set the
> ExclusiveBuffersPerChannel
> to 0 when using floating buffers, which can also decrease the probability
> of
> this exception.
>
> 4. It would be great to have experimental results for jobs with different
> exchange types.
> Thanks for the suggestion. I have a test about different exchange types,
> forward
> and rescale, and the results show no differences from the all-to-all type,
> which
> is also understandable, because the network memory usage is calculated
> with numChannels, independent of the edge type.
>
> Best,
> Yuxin
>
>
> Roman Khachatryan  于2022年12月28日周三 05:27写道：
>
> Hi everyone,
>
> Thanks for the proposal and the discussion.
>
> I couldn't find much details on how exactly the values of
> ExclusiveBuffersPerChannel and FloatingBuffersPerGate are calculated.
> I guess that
> - the threshold evaluation is done on JM
> - floating buffers calculation is done on TM based on the current memory
> available; so it is not taking into account any future tasks submitted for
> that (or other) job
> Is that correct?
>
> If so, I see the following potential issues:
>
> 1. Each (sub)task might have different values because the actual
> available memory might be different. E.g. some tasks might use exclusive
> buffers and others only floating. That could lead to significant skew
> in processing speed, and in turn to issues with checkpoints and watermarks.
>
> 2. Re-deployment of a task (e.g. on job failure) might lead to a completely
> different memory configuration. That, coupled with different values per
> subtask and operator, makes the performance analysis more difficult.
>
> (Regardless of whether it's done on TM or JM):
> 3. Each gate requires at least one buffer [1]. So, in case when no memory
> is available, TM will throw an Allocation timeout exception instead of
> Insufficient buffers exception immediately. A delay here (allocation
> timeout) seems like a regression.
> Besides that, the regression depends on how much memory is actually
> available and how much it is contended, doesn't it?
> Should there still be a lower threshold of available memory, below which
> the job (task) isn't accepted?
> 4. The same threshold for all types of shuffles will likely result in using
> exclusive buffers
> for point-wise connections and floating buffers for all-to-all ones. I'm
> not sure if that's always optimal. It would be great to have experimental
> results for jobs with different exchange types, WDYT?
>
> [1]
> https://issues.apache.org/jira/browse/FLINK-24035
>
> Regards,
> Roman
>
>
> On Tue, Dec 27, 2022 at 4:12 AM Yuxin Tan  wrote:
>
> Hi, Weihua
>
> Thanks for your suggestions.
>
> 1. How about reducing ExclusiveBuffersPerChannel to 1 fir

Re: [DISCUSS] FLIP-266: Simplify network memory configurations for TaskManager

[Yuxin Tan]

 a waste of resources. As far as possible, Flink
> > framework can automatically set a reasonable value, but I have a small
> > problem. network memory is not only related to the parallelism of the
> task,
> > but also to the complexity of the task DAG. The more complex a DAG is,
> > shuffle write and shuffle read require larger buffers. How can we
> determine
> > how many RS and IG a DAG has?
> >
> >
> >
> > Best
> > JasonLee
> >
> >
> >  Replied Message 
> > | From | Yuxin Tan |
> > | Date | 12/28/2022 18:29 |
> > | To |  |
> > | Subject | Re: [DISCUSS] FLIP-266: Simplify network memory
> configurations
> > for TaskManager |
> > Hi, Roman
> >
> > Thanks for the replay.
> >
> > ExclusiveBuffersPerChannel and FloatingBuffersPerGate are obtained from
> > configurations, which are not calculated. I have described them in the
> FLIP
> > motivation section.
> >
> > 3. Each gate requires at least one buffer...
> > The timeout exception occurs when the ExclusiveBuffersPerChannel
> > can not be requested from NetworkBufferPool, which is not caused by the
> > change of this Flip. In addition, if  we have set the
> > ExclusiveBuffersPerChannel
> > to 0 when using floating buffers, which can also decrease the probability
> > of
> > this exception.
> >
> > 4. It would be great to have experimental results for jobs with different
> > exchange types.
> > Thanks for the suggestion. I have a test about different exchange types,
> > forward
> > and rescale, and the results show no differences from the all-to-all
> type,
> > which
> > is also understandable, because the network memory usage is calculated
> > with numChannels, independent of the edge type.
> >
> > Best,
> > Yuxin
> >
> >
> > Roman Khachatryan  于2022年12月28日周三 05:27写道：
> >
> > Hi everyone,
> >
> > Thanks for the proposal and the discussion.
> >
> > I couldn't find much details on how exactly the values of
> > ExclusiveBuffersPerChannel and FloatingBuffersPerGate are calculated.
> > I guess that
> > - the threshold evaluation is done on JM
> > - floating buffers calculation is done on TM based on the current memory
> > available; so it is not taking into account any future tasks submitted
> for
> > that (or other) job
> > Is that correct?
> >
> > If so, I see the following potential issues:
> >
> > 1. Each (sub)task might have different values because the actual
> > available memory might be different. E.g. some tasks might use exclusive
> > buffers and others only floating. That could lead to significant skew
> > in processing speed, and in turn to issues with checkpoints and
> watermarks.
> >
> > 2. Re-deployment of a task (e.g. on job failure) might lead to a
> completely
> > different memory configuration. That, coupled with different values per
> > subtask and operator, makes the performance analysis more difficult.
> >
> > (Regardless of whether it's done on TM or JM):
> > 3. Each gate requires at least one buffer [1]. So, in case when no memory
> > is available, TM will throw an Allocation timeout exception instead of
> > Insufficient buffers exception immediately. A delay here (allocation
> > timeout) seems like a regression.
> > Besides that, the regression depends on how much memory is actually
> > available and how much it is contended, doesn't it?
> > Should there still be a lower threshold of available memory, below which
> > the job (task) isn't accepted?
> > 4. The same threshold for all types of shuffles will likely result in
> using
> > exclusive buffers
> > for point-wise connections and floating buffers for all-to-all ones. I'm
> > not sure if that's always optimal. It would be great to have experimental
> > results for jobs with different exchange types, WDYT?
> >
> > [1]
> > https://issues.apache.org/jira/browse/FLINK-24035
> >
> > Regards,
> > Roman
> >
> >
> > On Tue, Dec 27, 2022 at 4:12 AM Yuxin Tan 
> wrote:
> >
> > Hi, Weihua
> >
> > Thanks for your suggestions.
> >
> > 1. How about reducing ExclusiveBuffersPerChannel to 1 first when the
> > total buffer is not enough?
> >
> > I think it's a good idea. Will try and check the results in PoC. Before
> > all
> > read buffers use floating buffers, I will try to use
> > (ExclusiveBuffersPerChannel - i)
> > buffers per channel first. For example, if the user has configu

Re: [DISCUSS] FLIP-266: Simplify network memory configurations for TaskManager

[Yuxin Tan]

Hi, Roman

Sorry about that I missed one question just now.

>  if the two configuration options are still in use, why does the FLIP
propose to deprecate them?
These two configs are usually used to avoid the memory issue, but
after introducing the improvement, generally, I think it is no longer
necessary to adjust these two configurations to avoid the issue. So
I propose to deprecate them in the future when the @Experimental
annotation of the newly added config is removed.

Best,
Yuxin


Roman Khachatryan  于2022年12月28日周三 20:10写道：

> Thanks for your reply Yuxin,
>
> > ExclusiveBuffersPerChannel and FloatingBuffersPerGate are obtained from
> > configurations, which are not calculated. I have described them in the
> FLIP
> > motivation section.
>
> The motivation section says about floating buffers:
> > FloatingBuffersPerGate is within the range of
> [numFloatingBufferThreashold, ExclusiveBuffersPerChannel * numChannels +
> DefaultFloatingBuffersPerGate] ...
> So my question is what value exactly in this range will it have and how and
> where will it be computed?
>
> As for the ExclusiveBuffersPerChannel, there was a proposal in the thread
> to calculate it dynamically (by linear search
> from taskmanager.network.memory.buffers-per-channel down to 0).
>
> Also, if the two configuration options are still in use, why does the FLIP
> propose to deprecate them?
>
> Besides that, wouldn't it be more clear to separate motivation from the
> proposed changes?
>
> Regards,
> Roman
>
>
> On Wed, Dec 28, 2022 at 12:19 PM JasonLee <17610775...@163.com> wrote:
>
> > Hi Yuxin
> >
> >
> > Thanks for the proposal, big + 1 for this FLIP.
> >
> >
> >
> > It is difficult for users to calculate the size of network memory. If the
> > setting is too small, the task cannot be started. If the setting is too
> > large, there may be a waste of resources. As far as possible, Flink
> > framework can automatically set a reasonable value, but I have a small
> > problem. network memory is not only related to the parallelism of the
> task,
> > but also to the complexity of the task DAG. The more complex a DAG is,
> > shuffle write and shuffle read require larger buffers. How can we
> determine
> > how many RS and IG a DAG has?
> >
> >
> >
> > Best
> > JasonLee
> >
> >
> >  Replied Message 
> > | From | Yuxin Tan |
> > | Date | 12/28/2022 18:29 |
> > | To |  |
> > | Subject | Re: [DISCUSS] FLIP-266: Simplify network memory
> configurations
> > for TaskManager |
> > Hi, Roman
> >
> > Thanks for the replay.
> >
> > ExclusiveBuffersPerChannel and FloatingBuffersPerGate are obtained from
> > configurations, which are not calculated. I have described them in the
> FLIP
> > motivation section.
> >
> > 3. Each gate requires at least one buffer...
> > The timeout exception occurs when the ExclusiveBuffersPerChannel
> > can not be requested from NetworkBufferPool, which is not caused by the
> > change of this Flip. In addition, if  we have set the
> > ExclusiveBuffersPerChannel
> > to 0 when using floating buffers, which can also decrease the probability
> > of
> > this exception.
> >
> > 4. It would be great to have experimental results for jobs with different
> > exchange types.
> > Thanks for the suggestion. I have a test about different exchange types,
> > forward
> > and rescale, and the results show no differences from the all-to-all
> type,
> > which
> > is also understandable, because the network memory usage is calculated
> > with numChannels, independent of the edge type.
> >
> > Best,
> > Yuxin
> >
> >
> > Roman Khachatryan  于2022年12月28日周三 05:27写道：
> >
> > Hi everyone,
> >
> > Thanks for the proposal and the discussion.
> >
> > I couldn't find much details on how exactly the values of
> > ExclusiveBuffersPerChannel and FloatingBuffersPerGate are calculated.
> > I guess that
> > - the threshold evaluation is done on JM
> > - floating buffers calculation is done on TM based on the current memory
> > available; so it is not taking into account any future tasks submitted
> for
> > that (or other) job
> > Is that correct?
> >
> > If so, I see the following potential issues:
> >
> > 1. Each (sub)task might have different values because the actual
> > available memory might be different. E.g. some tasks might use exclusive
> > buffers and others only floating. That could lead to significant skew
> > in processing speed, and in turn to

Re: [DISCUSS] FLIP-266: Simplify network memory configurations for TaskManager

[Yuxin Tan]

Hi, everyone

Thanks for the reply and the discussion.

We discussed this with @Guowei Ma, @Dong Lin, and @Yanfei Lei
offline, and reached a consensus on this FLIP. Based on the offline
discussions and suggestions from @Weihua Hu, the following changes
have been updated in the FLIP.

1. Changes in public interfaces.
- Updated the descriptions of the newly added config to describe the
option more clearly.
- The new config will be marked as experimental in the first release,
and we will revisit this in the next release based on the user feedback.
- In the long run, with the new config, we think the original two configs
can be deprecated. At this stage, since the new config is still
experimental,
we will not immediately deprecate them.
- Modify the config key name as
taskmanager.memory.network.read-buffer.required-per-gate.max for
more clarity.
2. Modify the floating buffer calculation method.
- When the memory used reaches the threshold, the number of exclusive
buffers is gradually reduced in a fine-grained manner, rather than directly
reducing the number of exclusive buffers to 0.

Best,
Yuxin


Yuxin Tan  于2022年12月29日周四 14:48写道：

> Hi, Roman
>
> Sorry about that I missed one question just now.
>
> >  if the two configuration options are still in use, why does the FLIP
> propose to deprecate them?
> These two configs are usually used to avoid the memory issue, but
> after introducing the improvement, generally, I think it is no longer
> necessary to adjust these two configurations to avoid the issue. So
> I propose to deprecate them in the future when the @Experimental
> annotation of the newly added config is removed.
>
> Best,
> Yuxin
>
>
> Roman Khachatryan  于2022年12月28日周三 20:10写道：
>
>> Thanks for your reply Yuxin,
>>
>> > ExclusiveBuffersPerChannel and FloatingBuffersPerGate are obtained from
>> > configurations, which are not calculated. I have described them in the
>> FLIP
>> > motivation section.
>>
>> The motivation section says about floating buffers:
>> > FloatingBuffersPerGate is within the range of
>> [numFloatingBufferThreashold, ExclusiveBuffersPerChannel * numChannels +
>> DefaultFloatingBuffersPerGate] ...
>> So my question is what value exactly in this range will it have and how
>> and
>> where will it be computed?
>>
>> As for the ExclusiveBuffersPerChannel, there was a proposal in the thread
>> to calculate it dynamically (by linear search
>> from taskmanager.network.memory.buffers-per-channel down to 0).
>>
>> Also, if the two configuration options are still in use, why does the FLIP
>> propose to deprecate them?
>>
>> Besides that, wouldn't it be more clear to separate motivation from the
>> proposed changes?
>>
>> Regards,
>> Roman
>>
>>
>> On Wed, Dec 28, 2022 at 12:19 PM JasonLee <17610775...@163.com> wrote:
>>
>> > Hi Yuxin
>> >
>> >
>> > Thanks for the proposal, big + 1 for this FLIP.
>> >
>> >
>> >
>> > It is difficult for users to calculate the size of network memory. If
>> the
>> > setting is too small, the task cannot be started. If the setting is too
>> > large, there may be a waste of resources. As far as possible, Flink
>> > framework can automatically set a reasonable value, but I have a small
>> > problem. network memory is not only related to the parallelism of the
>> task,
>> > but also to the complexity of the task DAG. The more complex a DAG is,
>> > shuffle write and shuffle read require larger buffers. How can we
>> determine
>> > how many RS and IG a DAG has?
>> >
>> >
>> >
>> > Best
>> > JasonLee
>> >
>> >
>> >  Replied Message 
>> > | From | Yuxin Tan |
>> > | Date | 12/28/2022 18:29 |
>> > | To |  |
>> > | Subject | Re: [DISCUSS] FLIP-266: Simplify network memory
>> configurations
>> > for TaskManager |
>> > Hi, Roman
>> >
>> > Thanks for the replay.
>> >
>> > ExclusiveBuffersPerChannel and FloatingBuffersPerGate are obtained from
>> > configurations, which are not calculated. I have described them in the
>> FLIP
>> > motivation section.
>> >
>> > 3. Each gate requires at least one buffer...
>> > The timeout exception occurs when the ExclusiveBuffersPerChannel
>> > can not be requested from NetworkBufferPool, which is not caused by the
>> > change of this Flip. In addition, if  we have set the
>> > ExclusiveBuffersPerChannel
>> > to 0 when using floating buffers, which can also decrease the

Re: [DISCUSS] FLIP-266: Simplify network memory configurations for TaskManager

[Yuxin Tan]

Hi all,

Thanks for all the feedback so far.

The discussion has been going on for some time. If there are no
more new comments, we will start a vote today.

Best,
Yuxin


Yuxin Tan  于2022年12月29日周四 17:37写道：

> Hi, everyone
>
> Thanks for the reply and the discussion.
>
> We discussed this with @Guowei Ma, @Dong Lin, and @Yanfei Lei
> offline, and reached a consensus on this FLIP. Based on the offline
> discussions and suggestions from @Weihua Hu, the following changes
> have been updated in the FLIP.
>
> 1. Changes in public interfaces.
> - Updated the descriptions of the newly added config to describe the
> option more clearly.
> - The new config will be marked as experimental in the first release,
> and we will revisit this in the next release based on the user feedback.
> - In the long run, with the new config, we think the original two configs
> can be deprecated. At this stage, since the new config is still
> experimental,
> we will not immediately deprecate them.
> - Modify the config key name as
> taskmanager.memory.network.read-buffer.required-per-gate.max for
> more clarity.
> 2. Modify the floating buffer calculation method.
> - When the memory used reaches the threshold, the number of exclusive
> buffers is gradually reduced in a fine-grained manner, rather than
> directly
> reducing the number of exclusive buffers to 0.
>
> Best,
> Yuxin
>
>
> Yuxin Tan  于2022年12月29日周四 14:48写道：
>
>> Hi, Roman
>>
>> Sorry about that I missed one question just now.
>>
>> >  if the two configuration options are still in use, why does the FLIP
>> propose to deprecate them?
>> These two configs are usually used to avoid the memory issue, but
>> after introducing the improvement, generally, I think it is no longer
>> necessary to adjust these two configurations to avoid the issue. So
>> I propose to deprecate them in the future when the @Experimental
>> annotation of the newly added config is removed.
>>
>> Best,
>> Yuxin
>>
>>
>> Roman Khachatryan  于2022年12月28日周三 20:10写道：
>>
>>> Thanks for your reply Yuxin,
>>>
>>> > ExclusiveBuffersPerChannel and FloatingBuffersPerGate are obtained from
>>> > configurations, which are not calculated. I have described them in the
>>> FLIP
>>> > motivation section.
>>>
>>> The motivation section says about floating buffers:
>>> > FloatingBuffersPerGate is within the range of
>>> [numFloatingBufferThreashold, ExclusiveBuffersPerChannel * numChannels +
>>> DefaultFloatingBuffersPerGate] ...
>>> So my question is what value exactly in this range will it have and how
>>> and
>>> where will it be computed?
>>>
>>> As for the ExclusiveBuffersPerChannel, there was a proposal in the thread
>>> to calculate it dynamically (by linear search
>>> from taskmanager.network.memory.buffers-per-channel down to 0).
>>>
>>> Also, if the two configuration options are still in use, why does the
>>> FLIP
>>> propose to deprecate them?
>>>
>>> Besides that, wouldn't it be more clear to separate motivation from the
>>> proposed changes?
>>>
>>> Regards,
>>> Roman
>>>
>>>
>>> On Wed, Dec 28, 2022 at 12:19 PM JasonLee <17610775...@163.com> wrote:
>>>
>>> > Hi Yuxin
>>> >
>>> >
>>> > Thanks for the proposal, big + 1 for this FLIP.
>>> >
>>> >
>>> >
>>> > It is difficult for users to calculate the size of network memory. If
>>> the
>>> > setting is too small, the task cannot be started. If the setting is too
>>> > large, there may be a waste of resources. As far as possible, Flink
>>> > framework can automatically set a reasonable value, but I have a small
>>> > problem. network memory is not only related to the parallelism of the
>>> task,
>>> > but also to the complexity of the task DAG. The more complex a DAG is,
>>> > shuffle write and shuffle read require larger buffers. How can we
>>> determine
>>> > how many RS and IG a DAG has?
>>> >
>>> >
>>> >
>>> > Best
>>> > JasonLee
>>> >
>>> >
>>> >  Replied Message 
>>> > | From | Yuxin Tan |
>>> > | Date | 12/28/2022 18:29 |
>>> > | To |  |
>>> > | Subject | Re: [DISCUSS] FLIP-266: Simplify network memory
>>> configurations
>>> > for TaskManager |
>>> > H

[VOTE]FLIP-266: Simplify network memory configurations for TaskManager

[Yuxin Tan]

Hi all,

Thanks for all the feedback so far.
Based on the discussion[1], we have come to a consensus,
so I would like to start a vote on FLIP-266: Simplify network
memory configurations for TaskManager[2].

The vote will last for at least 72 hours (Jan 6th at 11:00 GMT)
unless there is an objection or insufficient votes.

[1]https://lists.apache.org/thread/yzfn5yf2tf8o165ns337bvfmb7t8h7mf
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-266%3A+Simplify+network+memory+configurations+for+TaskManager

Best,
Yuxin

[RESULT][VOTE]FLIP-266: Simplify network memory configurations for TaskManager

[Yuxin Tan]

Hi all,

FLIP-266: Simplify network memory configurations for TaskManager[1]
has been accepted. The FLIP was voted in this thread[2].

There are 3 bindings, and 1 non-bindings as follows:

Xintong Song (binding)
Zhu Zhu (binding)
JasonLee (no-binding)
Lijie Wang (binding)

There are no disapproving votes.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-266:+Simplify+network+memory+configurations+for+TaskManager
[2]https://lists.apache.org/thread/flv4w4tn5r8xbhzdqngx8o8o8t0gv3bt

Best,
Yuxin

Re: [VOTE]FLIP-266: Simplify network memory configurations for TaskManager

[Yuxin Tan]

Hi all,

There are 3 bindings, and 1 non-bindings. Based on the result, I have
closed the vote with the thread[1].
Thanks again.

[1] https://lists.apache.org/thread/m0nt12pvoczlf924bry464ltwdm75lok

Best,
Yuxin


Lijie Wang  于2023年1月4日周三 22:46写道：

> +1 (binding)
>
> Best,
> Lijie
>
> 17610775726 <17610775...@163.com> 于2023年1月4日周三 13:03写道：
>
> >
> >
> > +1 (no binding)
> >
> >
> > Best
> > JasonLee
> >
> >
> >  Replied Message 
> > | From | Yuxin Tan |
> > | Date | 01/3/2023 17:56 |
> > | To |  |
> > | Subject | [VOTE]FLIP-266: Simplify network memory configurations for
> > TaskManager |
> > Hi all,
> >
> > Thanks for all the feedback so far.
> > Based on the discussion[1], we have come to a consensus,
> > so I would like to start a vote on FLIP-266: Simplify network
> > memory configurations for TaskManager[2].
> >
> > The vote will last for at least 72 hours (Jan 6th at 11:00 GMT)
> > unless there is an objection or insufficient votes.
> >
> > [1]https://lists.apache.org/thread/yzfn5yf2tf8o165ns337bvfmb7t8h7mf
> > [2]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-266%3A+Simplify+network+memory+configurations+for+TaskManager
> >
> > Best,
> > Yuxin
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Lincoln Lee

[Yuxin Tan]

Congratulations, Lincoln!

Best,
Yuxin


Qingsheng Ren  于2023年1月10日周二 14:11写道：

> Congratulations, Lincoln!
>
> Best,
> Qingsheng
>
> On Tue, Jan 10, 2023 at 2:06 PM yuxia  wrote:
>
> > Congratulations, Lincoln!
> >
> > Best regards,
> > Yuxia
> >
> > - 原始邮件 -
> > 发件人: "Shuiqiang Chen" 
> > 收件人: "dev" 
> > 发送时间: 星期二, 2023年 1 月 10日 下午 1:50:50
> > 主题: Re: [ANNOUNCE] New Apache Flink Committer - Lincoln Lee
> >
> > Congratulations, Lincoln!
> >
> > Best,
> > Shuiqiang
> >
> > Benchao Li 于2023年1月10日 周二13:48写道：
> >
> > > Congratulations, Lincoln!
> > >
> > > Hangxiang Yu  于2023年1月10日周二 13:46写道：
> > >
> > > > Congratulations, Lincoln!
> > > >
> > > > On Tue, Jan 10, 2023 at 1:34 PM Dian Fu 
> wrote:
> > > >
> > > > > Congratulations, Lincoln!
> > > > >
> > > > > Regards,
> > > > > Dian
> > > > >
> > > > > On Tue, Jan 10, 2023 at 1:31 PM weijie guo <
> > guoweijieres...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Congratulations, Lincoln!
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Weijie
> > > > > >
> > > > > >
> > > > > > Lijie Wang  于2023年1月10日周二 12:24写道：
> > > > > >
> > > > > > > Congratulations, Lincoln!
> > > > > > >
> > > > > > > Best,
> > > > > > > Lijie
> > > > > > >
> > > > > > > Jingsong Li  于2023年1月10日周二 12:07写道：
> > > > > > >
> > > > > > > > Congratulations, Lincoln!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Jingsong
> > > > > > > >
> > > > > > > > On Tue, Jan 10, 2023 at 11:56 AM Leonard Xu <
> xbjt...@gmail.com
> > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > Congratulations, Lincoln!
> > > > > > > > >
> > > > > > > > > Impressive work in streaming semantics, well deserved!
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Leonard
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > On Jan 10, 2023, at 11:52 AM, Jark Wu 
> > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi everyone,
> > > > > > > > > >
> > > > > > > > > > On behalf of the PMC, I'm very happy to announce Lincoln
> > Lee
> > > > as a
> > > > > > new
> > > > > > > > Flink
> > > > > > > > > > committer.
> > > > > > > > > >
> > > > > > > > > > Lincoln Lee has been a long-term Flink contributor since
> > > 2017.
> > > > He
> > > > > > > > mainly
> > > > > > > > > > works on Flink
> > > > > > > > > > SQL parts and drives several important FLIPs, e.g.,
> > FLIP-232
> > > > > (Retry
> > > > > > > > Async
> > > > > > > > > > I/O), FLIP-234 (
> > > > > > > > > > Retryable Lookup Join), FLIP-260 (TableFunction Finish).
> > > > Besides,
> > > > > > He
> > > > > > > > also
> > > > > > > > > > contributed
> > > > > > > > > > much to Streaming Semantics, including the
> non-determinism
> > > > > problem
> > > > > > > and
> > > > > > > > the
> > > > > > > > > > message
> > > > > > > > > > ordering problem.
> > > > > > > > > >
> > > > > > > > > > Please join me in congratulating Lincoln for becoming a
> > Flink
> > > > > > > > committer!
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Jark Wu
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Best,
> > > > Hangxiang.
> > > >
> > >
> > >
> > > --
> > >
> > > Best,
> > > Benchao Li
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Weijie Guo

[Yuxin Tan]

Congratulations, Weijie!

Best,
Yuxin


Yanfei Lei  于2023年2月13日周一 12:16写道：

> Congratulations, Weijie!
>
> Best regards,
> Yanfei
>
> Junrui Lee  于2023年2月13日周一 12:12写道：
> >
> > Congratulations, Weijie!
> >
> > Best,
> > Junrui
>

Re: [ANNOUNCE] New Apache Flink Committer - Jing Ge

[Yuxin Tan]

Congratulations, Jing Ge!

Best,
Yuxin


Shengkai Fang  于2023年2月15日周三 10:52写道：

> Congratulations!
>
> Best,
> Shengkai
>
> Yanfei Lei  于2023年2月15日周三 10:08写道：
>
> > Congratulations, Jing Ge !
> >
> > Best regards,
> > Yanfei
> >
> > yuxia  于2023年2月15日周三 09:34写道：
> > >
> > > Congratulations, Jing Ge !
> > >
> > > Best regards,
> > > Yuxia
> > >
> > > - 原始邮件 -
> > > 发件人: "Jark Wu" 
> > > 收件人: "dev" 
> > > 发送时间: 星期二, 2023年 2 月 14日 下午 11:32:30
> > > 主题: Re: [ANNOUNCE] New Apache Flink Committer - Jing Ge
> > >
> > > Welcome and congratulations, Jing!
> > >
> > > Best,
> > > Jark
> > >
> > > > 2023年2月14日 23:14，Lijie Wang  写道：
> > > >
> > > > Congratulations, Jing Ge !
> > > >
> > > > Best,
> > > > Lijie
> > > >
> > > > Sergey Nuyanzin  于2023年2月14日周二 21:47写道：
> > > >
> > > >> Congratulations, Jing Ge!
> > > >>
> > > >> On Tue, Feb 14, 2023 at 2:47 PM Rui Fan  wrote:
> > > >>
> > > >>> Congratulations, Jing!
> > > >>>
> > > >>> Best,
> > > >>> Rui Fan
> > > >>>
> > > >>> On Tue, Feb 14, 2023 at 19:36 Yuepeng Pan  wrote:
> > > >>>
> > >  Congratulations, Jing Ge !
> > > 
> > >  Best,Yuepeng Pan
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > >  At 2023-02-14 15:49:24, "godfrey he"  wrote:
> > > > Hi everyone,
> > > >
> > > > On behalf of the PMC, I'm very happy to announce Jing Ge as a new
> > > >> Flink
> > > > committer.
> > > >
> > > > Jing has been consistently contributing to the project for over 1
> > > >> year.
> > > > He authored more than 50 PRs and reviewed more than 40 PRs
> > > > with mainly focus on connector, test, and document modules.
> > > > He was very active on the mailing list (more than 90 threads)
> last
> > > >> year,
> > > > which includes participating in a lot of dev discussions (30+),
> > > > providing many effective suggestions for FLIPs and answering
> > > > many user questions. He was the Flink Forward 2022 keynote
> speaker
> > > > to help promote Flink and  a trainer for Flink troubleshooting
> and
> > >  performance
> > > > tuning of Flink Forward 2022 training program.
> > > >
> > > > Please join me in congratulating Jing for becoming a Flink
> > committer!
> > > >
> > > > Best,
> > > > Godfrey
> > > 
> > > >>>
> > > >>
> > > >>
> > > >> --
> > > >> Best regards,
> > > >> Sergey
> > > >>
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Dong Lin

[Yuxin Tan]

Congratulations, Dong!

Best,
Yuxin


ConradJam  于2023年2月17日周五 09:59写道：

> Congratulations Dong
>
> Dong Lin  于2023年2月16日周四 21:32写道：
>
> > Thank you all!
> >
> > I am looking forward to making more contributions to Apache Flink!
> >
> > On Thu, Feb 16, 2023 at 8:34 PM Jark Wu  wrote:
> >
> > > Congratulations, Dong!
> > >
> > > Best,
> > > Jark
> > >
> > > > 2023年2月16日 20:13，Yanfei Lei  写道：
> > > >
> > > > Congratulations, Dong!
> > > >
> > > > Best regards,
> > > > Yanfei
> > > >
> > > > Sergey Nuyanzin  于2023年2月16日周四 17:21写道：
> > > >>
> > > >> Congratulations, Dong!
> > > >>
> > > >> On Thu, Feb 16, 2023 at 10:01 AM Wencong Liu 
> > > wrote:
> > > >>
> > > >>> Congratulations Dong!
> > > >>>
> > > >>>
> > > >>> Bast,
> > > >>> Wencong Liu
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> At 2023-02-16 14:19:08, "Guowei Ma"  wrote:
> > >  Hi, everyone
> > > 
> > >    On behalf of the PMC, I'm very happy to announce Dong Lin as a
> new
> > >  Flink PMC.
> > > 
> > >    Dong is currently the main driver of Flink ML. He reviewed a
> large
> > >  number of Flink ML related PRs and also participated in many Flink
> > ML
> > >  improvements, such as "FLIP-173","FLIP-174" etc. At the same time,
> > he
> > > made
> > >  a lot of evangelism events contributions for the Flink ML
> ecosystem.
> > >    In fact, in addition to the Flink machine learning field, Dong
> has
> > > >>> also
> > >  participated in many other improvements in Flink, such as
> > "FLIP-205",
> > >  "FLIP-266","FLIP-269","FLIP-274" etc.
> > >    Please join me in congratulating Dong Lin for becoming a Flink
> > PMC!
> > > 
> > >  Best,
> > >  Guowei(on behalf of the Flink PMC)
> > > >>>
> > > >>
> > > >>
> > > >> --
> > > >> Best regards,
> > > >> Sergey
> > >
> > >
> >
>
>
> --
> Best
>
> ConradJam
>

[DISCUSS] FLIP-301: Hybrid Shuffle supports Remote Storage

[Yuxin Tan]

Hi everyone,

I would like to start a discussion on FLIP-301: Hybrid Shuffle supports
Remote Storage[1].

In the cloud-native environment, it is difficult to determine the
appropriate
disk space for Batch shuffle, which will affect job stability.

This FLIP is to support Remote Storage for Hybrid Shuffle to improve the
Batch job stability in the cloud-native environment.

The goals of this FLIP are as follows.
1. By default, use the local memory and disk to ensure high shuffle
performance if the local storage space is sufficient.
2. When the local storage space is insufficient, use remote storage as
a supplement to avoid large-scale Batch job failure.

Looking forward to hearing from you.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-301%3A+Hybrid+Shuffle+supports+Remote+Storage

Best,
Yuxin

Re: [DISCUSS] FLIP-301: Hybrid Shuffle supports Remote Storage

[Yuxin Tan]

Thanks for joining the discussion.

@weijie guo
> 1. How to optimize the broadcast result partition?
For the partitions with multi-consumers, e.g., broadcast result partition,
partition reuse,
speculative, etc, the processing logic is the same as the original Hybrid
Shuffle, that is,
using the full spilling strategy. It indeed may reduce the opportunity to
consume from
memory, but the PoC shows that it has no effect on the performance
basically.

> 2. Can the new proposal completely avoid this problem of inaccurate
backlog
calculation?
Yes, this can avoid the problem completely. About the read buffers, the N
is to reserve
one exclusive buffer per channel, which is to avoid the deadlock because
the buffers
are acquired by some channels and other channels can not request any
buffers. But
the buffers except for the N can be floating (competing to request the
buffers) by all
channels.

@Wencong Liu
> Deciding the Segment size dynamically will be helpful.
I agree that it may be better if the segment size is dynamically decided,
but for simplifying
the implementation of the first version, we want to make this a fixed value
for each tier.
In the future, this can be a good improvement if necessary. In the first
version, we will mainly
focus on the more important features, such as the tiered storage
architecture, dynamic
switching tiers, supporting remote storage, memory management, etc.

Best,
Yuxin


Wencong Liu  于2023年3月7日周二 16:48写道：

> Hello Yuxin,
>
>
> Thanks for your proposal! Adding remote storage capability to Flink's
> Hybrid Shuffle is a significant improvement that addresses the issue of
> local disk storage limitations. This enhancement not only ensures
> uninterrupted Shuffle, but also enables Flink to handle larger workloads
> and more complex data processing tasks. With the ability to seamlessly
> shift between local and remote storage, Flink's Hybrid Shuffle will be more
> versatile and scalable, making it an ideal choice for organizations looking
> to build distributed data processing applications with ease.
> Besides, I've a small question about the size of Segment in different
> storages. According to the FLIP, the size of Segment may be fixed for each
> Storage Tier, but I think the fixed size may affect the shuffle
> performance. For example, smaller segment size will improve the utilization
> rate of Memory Storage Tier, but it may brings extra cost to Disk Storage
> Tier or Remote Storage Tier. Deciding the size of Segment dynamicly will be
> helpful.
>
> Best,
>
>
> Wencong Liu
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> At 2023-03-06 13:51:21, "Yuxin Tan"  wrote:
> >Hi everyone,
> >
> >I would like to start a discussion on FLIP-301: Hybrid Shuffle supports
> >Remote Storage[1].
> >
> >In the cloud-native environment, it is difficult to determine the
> >appropriate
> >disk space for Batch shuffle, which will affect job stability.
> >
> >This FLIP is to support Remote Storage for Hybrid Shuffle to improve the
> >Batch job stability in the cloud-native environment.
> >
> >The goals of this FLIP are as follows.
> >1. By default, use the local memory and disk to ensure high shuffle
> >performance if the local storage space is sufficient.
> >2. When the local storage space is insufficient, use remote storage as
> >a supplement to avoid large-scale Batch job failure.
> >
> >Looking forward to hearing from you.
> >
> >[1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-301%3A+Hybrid+Shuffle+supports+Remote+Storage
> >
> >Best,
> >Yuxin
>

Re: [DISCUSS] FLIP-301: Hybrid Shuffle supports Remote Storage

[Yuxin Tan]

Hi, Junrui,
Thanks for the suggestions and ideas.

> If they are fixed, I suggest that FLIP could provide clearer explanations.
I have updated the FLIP and described the segment size more clearly.

> can we provide configuration options for users to manually adjust the
sizes?
The segment size can be configured if necessary. But considering that if we
exposed these parameters prematurely, it may be difficult to modify the
implementation later because the user has already used the configs. We
can make these internal configs or fixed values when implementing the first
version, I think we can use either of these two ways, because they are
internal and do not affect the public APIs.

Best,
Yuxin


Junrui Lee  于2023年3月8日周三 00:24写道：

> Hi Yuxin,
>
> This FLIP looks quite reasonable. Flink can solve the problem of Batch
> shuffle by
> combining local and remote storage, and can use fixed local disks for
> better performance
>  in most scenarios, while using remote storage as a supplement when local
> disks are not
>  sufficient, avoiding wasteful costs and poor job stability. Moreover, the
> solution also
> considers the issue of dynamic switching, which can automatically switch to
> remote
> storage when the local disk is full, saving costs, and automatically switch
> back when
> there is available space on the local disk.
>
> As Wencong Liu stated, an appropriate segment size is essential, as it can
> significantly
> affect shuffle performance. I also agree that the first version should
> focus mainly on the
> design and implementation. However, I have a small question about FLIP. I
> did not see
> any information regarding the segment size of memory, local disk, and
> remote storage
> in this FLIP. Are these three values fixed at present? If they are fixed, I
> suggest that FLIP
> could provide clearer explanations. Moreover, although a dynamic segment
> size
> mechanism is not necessary at the moment, can we provide configuration
> options for users
>  to manually adjust these sizes? I think it might be useful.
>
> Best,
> Junrui.
>
> Yuxin Tan  于2023年3月7日周二 20:14写道：
>
> > Thanks for joining the discussion.
> >
> > @weijie guo
> > > 1. How to optimize the broadcast result partition?
> > For the partitions with multi-consumers, e.g., broadcast result
> partition,
> > partition reuse,
> > speculative, etc, the processing logic is the same as the original Hybrid
> > Shuffle, that is,
> > using the full spilling strategy. It indeed may reduce the opportunity to
> > consume from
> > memory, but the PoC shows that it has no effect on the performance
> > basically.
> >
> > > 2. Can the new proposal completely avoid this problem of inaccurate
> > backlog
> > calculation?
> > Yes, this can avoid the problem completely. About the read buffers, the N
> > is to reserve
> > one exclusive buffer per channel, which is to avoid the deadlock because
> > the buffers
> > are acquired by some channels and other channels can not request any
> > buffers. But
> > the buffers except for the N can be floating (competing to request the
> > buffers) by all
> > channels.
> >
> > @Wencong Liu
> > > Deciding the Segment size dynamically will be helpful.
> > I agree that it may be better if the segment size is dynamically decided,
> > but for simplifying
> > the implementation of the first version, we want to make this a fixed
> value
> > for each tier.
> > In the future, this can be a good improvement if necessary. In the first
> > version, we will mainly
> > focus on the more important features, such as the tiered storage
> > architecture, dynamic
> > switching tiers, supporting remote storage, memory management, etc.
> >
> > Best,
> > Yuxin
> >
> >
> > Wencong Liu  于2023年3月7日周二 16:48写道：
> >
> > > Hello Yuxin,
> > >
> > >
> > > Thanks for your proposal! Adding remote storage capability to
> Flink's
> > > Hybrid Shuffle is a significant improvement that addresses the issue of
> > > local disk storage limitations. This enhancement not only ensures
> > > uninterrupted Shuffle, but also enables Flink to handle larger
> workloads
> > > and more complex data processing tasks. With the ability to seamlessly
> > > shift between local and remote storage, Flink's Hybrid Shuffle will be
> > more
> > > versatile and scalable, making it an ideal choice for organizations
> > looking
> > > to build distributed data processing applications with ease.
> > > Besides, I&#x

Re: [DISCUSS] FLIP-301: Hybrid Shuffle supports Remote Storage

[Yuxin Tan]

Hi, Weihua,
Thanks for the questions and the ideas.

> 1. How many performance regressions would there be if we only
used remote storage?

The new architecture can support to use remote storage only, but this
FLIP target is to improve job stability. And the change in the FLIP has
been significantly complex and the goal of the first version is to update
Hybrid Shuffle to the new architecture and support remote storage as
a supplement. The performance of this version is not the first priority,
so we haven’t tested the performance of using only remote storage.
If there are indeed regressions, we will keep optimizing the performance
of the remote storages and improve it until only remote storage is
available in the production environment.

> 2. Shall we move the local data to remote storage if the producer is
finished for a long time?

I agree that it is a good idea, which can release task manager resources
more timely. But moving data from TM local disk to remote storage needs
more detailed discussion and design, and it is easier to implement it based
on the new architecture. Considering the complexity, the target focus, and
the iteration cycle of the FLIP, we decide that the details are not
included
in the first version. We will extend and implement them in the subsequent
versions.

Best,
Yuxin


Weihua Hu  于2023年3月9日周四 11:22写道：

> Hi, Yuxin
>
> Thanks for driving this FLIP.
>
> The remote storage shuffle could improve the stability of Batch jobs.
>
> In our internal scenario, we use a hybrid cluster to run both
> Streaming(high priority)
> and Batch jobs(low priority). When there is not enough resources(such as
> cpu usage
> reaches a threshold), the batch containers will be evicted. So this will
> cause some re-run
> of batch tasks.
>
> It would be a great help if the remote storage could address this. So I
> have a few questions.
>
> 1. How many performance regressions would there be if we only used remote
> storage?
>
> 2. In current design, the shuffle data segment will write to one kind of
> storage tier.
> Shall we move the local data to remote storage if the producer is finished
> for a long time?
> So we can release the idle task manager with no shuffle data on it. This
> may help to reduce
> the resource usage when producer parallelism is larger than consume.
>
> Best,
> Weihua
>
>
> On Thu, Mar 9, 2023 at 10:38 AM Yuxin Tan  wrote:
>
> > Hi, Junrui,
> > Thanks for the suggestions and ideas.
> >
> > > If they are fixed, I suggest that FLIP could provide clearer
> > explanations.
> > I have updated the FLIP and described the segment size more clearly.
> >
> > > can we provide configuration options for users to manually adjust the
> > sizes?
> > The segment size can be configured if necessary. But considering that if
> we
> > exposed these parameters prematurely, it may be difficult to modify the
> > implementation later because the user has already used the configs. We
> > can make these internal configs or fixed values when implementing the
> first
> > version, I think we can use either of these two ways, because they are
> > internal and do not affect the public APIs.
> >
> > Best,
> > Yuxin
> >
> >
> > Junrui Lee  于2023年3月8日周三 00:24写道：
> >
> > > Hi Yuxin,
> > >
> > > This FLIP looks quite reasonable. Flink can solve the problem of Batch
> > > shuffle by
> > > combining local and remote storage, and can use fixed local disks for
> > > better performance
> > >  in most scenarios, while using remote storage as a supplement when
> local
> > > disks are not
> > >  sufficient, avoiding wasteful costs and poor job stability. Moreover,
> > the
> > > solution also
> > > considers the issue of dynamic switching, which can automatically
> switch
> > to
> > > remote
> > > storage when the local disk is full, saving costs, and automatically
> > switch
> > > back when
> > > there is available space on the local disk.
> > >
> > > As Wencong Liu stated, an appropriate segment size is essential, as it
> > can
> > > significantly
> > > affect shuffle performance. I also agree that the first version should
> > > focus mainly on the
> > > design and implementation. However, I have a small question about
> FLIP. I
> > > did not see
> > > any information regarding the segment size of memory, local disk, and
> > > remote storage
> > > in this FLIP. Are these three values fixed at present? If they are
> > fixed, I
> > > suggest that FLIP
> > > could provide clearer explanations. Moreover, a

Re: [Vote] FLIP-298: Unifying the Implementation of SlotManager

[Yuxin Tan]

Thanks, Weihua!
+1 (non-binding)

Best,
Yuxin


weijie guo  于2023年3月10日周五 11:29写道：

> +1 (binding)
>
> Best regards,
>
> Weijie
>
>
> Shammon FY  于2023年3月10日周五 11:02写道：
>
> > Thanks weihua, +1 (non-binding)
> >
> > Best,
> > Shammon
> >
> > On Fri, Mar 10, 2023 at 10:32 AM Xintong Song 
> > wrote:
> >
> > > +1 (binding)
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > >
> > > On Thu, Mar 9, 2023 at 1:28 PM Weihua Hu 
> wrote:
> > >
> > > > Hi Everyone,
> > > >
> > > > I would like to start the vote on FLIP-298: Unifying the
> Implementation
> > > > of SlotManager [1]. The FLIP was discussed in this thread [2].
> > > >
> > > > This FLIP aims to unify the implementation of SlotManager in
> > > > order to reduce maintenance costs.
> > > >
> > > > The vote will last for at least 72 hours (03/14, 15:00 UTC+8)
> > > > unless there is an objection or insufficient votes. Thank you all.
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-298%3A+Unifying+the+Implementation+of+SlotManager
> > > > [2]https://lists.apache.org/thread/ocssfxglpc8z7cto3k8p44mrjxwr67r9
> > > >
> > > > Best,
> > > > Weihua
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo

[Yuxin Tan]

Congratulations, Yuxia!

Best,
Yuxin


Jark Wu  于2023年3月13日周一 10:26写道：

> Hi, everyone
>
> On behalf of the PMC, I'm very happy to announce Yuxia Luo as a new Flink
> Committer.
>
> Yuxia has been continuously contributing to the Flink project for almost
> two
> years, authored and reviewed hundreds of PRs over this time. He is
> currently
> the core maintainer of the Hive component, where he contributed many
> valuable
> features, including the Hive dialect with 95% compatibility and small file
> compaction.
> In addition, Yuxia driven FLIP-282 (DELETE & UPDATE API) to better
> integrate
> Flink with data lakes. He actively participated in dev discussions and
> answered
> many questions on the user mailing list.
>
> Please join me in congratulating Yuxia Luo for becoming a Flink Committer!
>
> Best,
> Jark Wu (on behalf of the Flink PMC)
>

Re: 退订

[Yuxin Tan]

退订请发任意内容到 dev-unsubscr...@flink.apache.org [1]

[1] https://flink.apache.org/community/

Best,
Yuxin


MuChen <9329...@qq.com.invalid> 于2023年3月13日周一 08:48写道：

> 退订
>
>
>
>
> MuChen
> 9329...@qq.com
>
>
>
>  

Re: [DISCUSS] FLIP-301: Hybrid Shuffle supports Remote Storage

[Yuxin Tan]

e an extension on hybrid shuffle which
> introduces a lots of changes, will it affect current design of pluggable
> remote shuffle service, such as Apache Celeborn [1].
>
> Thirdly, based on my previous experiences on implementing a tiered based
> state-backend, the condition of min-reserve-space-fraction to kick local
> data to remote storage might not be a good idea in all cases, we still need
> to consider the absolute reserved disk storage. Take a 20GB local data disk
> as example, it might be a bit too late to kick the local data when only 1GB
> (20GB*5%) space left.
>
> Last but not least, will we meet a concurrency problem when different
> subtasks within one process/node start to check the left disk space before
> deciding to write to local or remote?
>
>
> [1] https://celeborn.apache.org/
>
> Best
> Yun Tang
>
> 
> From: Xia Sun 
> Sent: Sunday, March 12, 2023 17:16
> To: dev@flink.apache.org 
> Subject: Re: [DISCUSS] FLIP-301: Hybrid Shuffle supports Remote Storage
>
> Hi Yuxin,
>
> Thanks for creating this FLIP!
> I'm a flink user, and in our internal scenario we use the colocation
> technology to run flink jobs and online service on the same machine
> together. We found that flink jobs are occasionally affected by other
> non-flink jobs (i.e. if the host disk space is full, that will result in
> 'No space left on device' error on flink jobs). This flip will really help
> us to benefit from hybrid shuffle without being worried about insufficient
> disk space problem.
>
> And I also have a few questions.
> 1. If the same subpartition spans multiple different tiers, how to keep the
> order of segments between different storage tiers (if necessary)?
> 2. In the process of writing to the local disk for a subpartition, what
> will happen if the disk space is found to be full? Will it report an error
> or automatically transfer to remote storage?
> 3. For remote storage, I noticed that it uses direct reading, which is
> different from the other two, does the switching between different tiers
> will bring overhead or waiting? In addition, compared to flink rss, which
> optimizes data compression and small file merging to improve throughput and
> relieve file system pressure, does the object storage system can meet the
> performance requirements and concurrent access challenges of large-scale
> batch jobs(parallelism > 1)?
>
> Thanks,
> Xia
>
> Zhu Zhu  于2023年3月10日周五 16:44写道：
>
> > Hi Yuxin,
> >
> > Thanks for creating this FLIP!
> > The idea of tiered storage looks good. Instead of choosing one from
> > multiple storages, it can help to balance between performance, cost and
> > stability. It also has the potential to adaptively select proper tiers
> > according to more runtime information, to achieve better performance
> > and ease of use.
> >
> > I have a question about the tier finding of data reading. In the FLIP
> > it proposes that the Read Client asks each storage tier whether a
> > given segment exists in it, from higher priority tiers to lower priority
> > ones. I'm a bit concerned about the cost of it, especially when data
> > are written to low priority tiers. Do you have any evaluation of it?
> > Is it possible to let the Reader Client know the location of the next
> > segment when it has finished reading one segment? Or maybe just let it
> > know whether the next segment is located in the same tier, if we can
> > have the assumption that tier changing would not be very frequent.
> >
> > Thanks,
> > Zhu
> >
> > Weihua Hu  于2023年3月10日周五 11:52写道：
> > >
> > > Thanks Yuxin for your explanation.
> > >
> > > That sounds reasonable. Looking forward to the new shuffle.
> > >
> > >
> > > Best,
> > > Weihua
> > >
> > >
> > > On Fri, Mar 10, 2023 at 11:48 AM Yuxin Tan 
> > wrote:
> > >
> > > > Hi, Weihua,
> > > > Thanks for the questions and the ideas.
> > > >
> > > > > 1. How many performance regressions would there be if we only
> > > > used remote storage?
> > > >
> > > > The new architecture can support to use remote storage only, but this
> > > > FLIP target is to improve job stability. And the change in the FLIP
> has
> > > > been significantly complex and the goal of the first version is to
> > update
> > > > Hybrid Shuffle to the new architecture and support remote storage as
> > > > a supplement. The performance of this version is not the first
> > priority,
> > > 

Re: [DISCUSS] FLIP-301: Hybrid Shuffle supports Remote Storage

[Yuxin Tan]

Hi, Yun,

Thanks for sharing the ideas.

> 1. We should trigger to kick the shuffle data to remote storage
once either condition is reached

I believe that configuring two options in this manner is a pragmatic
approach that can fulfill a wider range of usage scenarios. However,
if we present two options, it may become difficult to remove them
in the future once users have started relying on them. On the other
hand, if we introduce a single option, we can easily incorporate
additional options based on your recommendations if required.
Thus, we recommend adopting a one-option solution in the first
version to address the issue.

> 2. Perhaps we could switch to kick shuffle data to remote storage
once no space left exception is met.

Thanks for your valuable feedback. While the suggestion is a viable
solution to address the no space left exception issue, we are
concerned that implementing it could create interdependence between
the disk tier and remote storage tier, which would contradict our goal
of achieving independence between tiers in the new architecture.
Moreover, we believe that it is better to prevent encountering the
exception in the first place by reserving adequate disk space. This is
because other processes on the same machine may also be impacted
when the exception occurs. If the exception does still arise, we can
explore other potential solutions through detailed design and
discussions, such as the one you proposed, optimizing the reserved
space with a global counter of TM, etc. Although the current implementation
only partially addresses the exception issue, we expect to improve it
in subsequent versions due to the complexity of this FLIP. We would
appreciate hearing your thoughts on this matter.

Best,
Yuxin


Yun Tang  于2023年3月14日周二 14:48写道：

> Hi Yuxin
>
> Thanks for your reply.
> I am not saying that we should use an absolute reserved value to replace
> the current plan of the reserved fraction. We should trigger to kick the
> shuffle data to remote storage once either condition is reached.
> Maybe we could get some idea from the configuration of tiered based state
> backend [1].
>
> The concern about concurrent writing is due to the local disk being shared
> by all the instances running on that node, we cannot ensure other
> components would not flush data during shuffle writing. Perhaps we could
> switch to kick shuffle data to remote storage once no space left exception
> is met.
>
>
> [1]
> https://www.alibabacloud.com/help/en/realtime-compute-for-apache-flink/latest/geministatebackend-configurations#section-u0y-on0-owo
>
>
> Best
> Yun Tang
>
> 
> From: Yuxin Tan 
> Sent: Monday, March 13, 2023 15:06
> To: dev@flink.apache.org 
> Subject: Re: [DISCUSS] FLIP-301: Hybrid Shuffle supports Remote Storage
>
> Hi,
> Thanks for the feedback and questions from Zhu Zhu, Xia Sun and Yun Tang.
>
> @Zhu Zhu
> > 1. I'm a bit concerned about the cost of it.
>
> The Read Client's request to check the existence of a segment in each
> storage tier is the netty message, which is similar to the credit updating
> messages. It is observed that the netty message cost for these operations
> is relatively low and the number of messages is also relatively low
> compared to credit updates which are sent every few buffers. Moreover,
> since this request involves only memory operations, the message cost
> is significantly low during the total reading data process.
>
> And we will further optimize the message cost later, particularly for
> segments that remain in the same tier without switching tiers. That
> is, for consecutive segments in the same tier, we can continue to send
> the next segment without waiting for the downstream to ask whether the
> segment exists in this tier.
>
>
> @Xia Sun
> > 1. how to keep the order of segments between different storage tiers
>
> To indicate the sequential order of upstream segments, we rely on segmentId
> downstream. Once segment n has been fully consumed by the downstream,
> the subsequent segment n + 1 will be asked in its natural order. As for the
> order of the buffers within each segment, they follow the default ordering
> mechanisms of Flink.
>
> > 2. what will happen if the disk space is found to be full?
>
> By introducing a way of reserving some space in advance, we will try to
> avoid this situation of reporting the disk space error as much as possible.
> The size of reserved space can be configured through the option introduced
> in the public API.
>
> > 3. For remote storage, does the switching between different tiers
> will bring overhead or waiting?
>
> Our primary focus for the first version is to address the problem of job
> stability
> and implement a new architecture because the chan

Re: [DISCUSS] FLIP-301: Hybrid Shuffle supports Remote Storage

[Yuxin Tan]

Hi, Yun

Thanks for your suggestions.

> I think you could describe it explicitly in the original FLIP's goals or
design principles.

I have updated the FLIP and given a more detailed description of the
tier decoupling design.

Best,
Yuxin


Yun Tang  于2023年3月15日周三 20:46写道：

> Hi Yuxin,
>
> Thanks for your explanations.
> I think the kernel idea is that you prefer the simple and decoupling
> design for the 1st version of hybrid shuffle with remote storage. If
> following this idea, perhaps I could accept your current explanations and I
> think you could describe it explicitly in the original FLIP's goals or
> design principles.
>
>
> Best
> Yun Tang
> 
> From: Yuxin Tan 
> Sent: Wednesday, March 15, 2023 12:41
> To: dev@flink.apache.org 
> Subject: Re: [DISCUSS] FLIP-301: Hybrid Shuffle supports Remote Storage
>
> Hi, Yun,
>
> Thanks for sharing the ideas.
>
> > 1. We should trigger to kick the shuffle data to remote storage
> once either condition is reached
>
> I believe that configuring two options in this manner is a pragmatic
> approach that can fulfill a wider range of usage scenarios. However,
> if we present two options, it may become difficult to remove them
> in the future once users have started relying on them. On the other
> hand, if we introduce a single option, we can easily incorporate
> additional options based on your recommendations if required.
> Thus, we recommend adopting a one-option solution in the first
> version to address the issue.
>
> > 2. Perhaps we could switch to kick shuffle data to remote storage
> once no space left exception is met.
>
> Thanks for your valuable feedback. While the suggestion is a viable
> solution to address the no space left exception issue, we are
> concerned that implementing it could create interdependence between
> the disk tier and remote storage tier, which would contradict our goal
> of achieving independence between tiers in the new architecture.
> Moreover, we believe that it is better to prevent encountering the
> exception in the first place by reserving adequate disk space. This is
> because other processes on the same machine may also be impacted
> when the exception occurs. If the exception does still arise, we can
> explore other potential solutions through detailed design and
> discussions, such as the one you proposed, optimizing the reserved
> space with a global counter of TM, etc. Although the current implementation
> only partially addresses the exception issue, we expect to improve it
> in subsequent versions due to the complexity of this FLIP. We would
> appreciate hearing your thoughts on this matter.
>
> Best,
> Yuxin
>
>
> Yun Tang  于2023年3月14日周二 14:48写道：
>
> > Hi Yuxin
> >
> > Thanks for your reply.
> > I am not saying that we should use an absolute reserved value to replace
> > the current plan of the reserved fraction. We should trigger to kick the
> > shuffle data to remote storage once either condition is reached.
> > Maybe we could get some idea from the configuration of tiered based state
> > backend [1].
> >
> > The concern about concurrent writing is due to the local disk being
> shared
> > by all the instances running on that node, we cannot ensure other
> > components would not flush data during shuffle writing. Perhaps we could
> > switch to kick shuffle data to remote storage once no space left
> exception
> > is met.
> >
> >
> > [1]
> >
> https://www.alibabacloud.com/help/en/realtime-compute-for-apache-flink/latest/geministatebackend-configurations#section-u0y-on0-owo
> >
> >
> > Best
> > Yun Tang
> >
> > 
> > From: Yuxin Tan 
> > Sent: Monday, March 13, 2023 15:06
> > To: dev@flink.apache.org 
> > Subject: Re: [DISCUSS] FLIP-301: Hybrid Shuffle supports Remote Storage
> >
> > Hi,
> > Thanks for the feedback and questions from Zhu Zhu, Xia Sun and Yun Tang.
> >
> > @Zhu Zhu
> > > 1. I'm a bit concerned about the cost of it.
> >
> > The Read Client's request to check the existence of a segment in each
> > storage tier is the netty message, which is similar to the credit
> updating
> > messages. It is observed that the netty message cost for these operations
> > is relatively low and the number of messages is also relatively low
> > compared to credit updates which are sent every few buffers. Moreover,
> > since this request involves only memory operations, the message cost
> > is significantly low during the total reading data process.
> >
> > And we will further o

Re: [DISCUSS] FLIP-301: Hybrid Shuffle supports Remote Storage

[Yuxin Tan]

Hi,
Thanks for joining the discussion and giving the ideas.

@ron
> can the Hybrid Shuffle replace the RSS in the future?

The hybrid shuffle and RSS offer distinct solutions to address
the shuffle operation challenge. To optimize performance, we
store shuffle data in different tiers of memory and disk, enabling
greater flexibility and ease of use. Specifically, we cache
intermediate data in memory to reduce disk I/O overhead.
In contrast, RSS is a standalone service that can operate across
multiple servers within a cluster, parallelizing shuffle operations
to enhance performance. However, this introduces additional
deployment and maintenance costs. Each approach has its own
benefits and drawbacks, and users should be able to select the
method that best suits their needs. So I think we cannot replace
RSS in the future.

@ConradJam
> Should we define a data acceleration layer like Alluxio in remote storage?

I'm not entirely clear on the detailed plan you've proposed, but I
understand that you want to use Alluxio to serve as a cache layer for
the remote stoarge tier. It's designed to provide low-latency data
access to applications through a distributed caching layer. However,
implementing Alluxio introduces additional dependencies and
deployment/maintenance costs for users. While our design approach
is to supplement local storage with remote storage, as local storage
is generally sufficient. Given the limited usage scenarios, introducing
such costs for optimization may not be worthwhile or meaningful.
Additionally, for users, added dependencies imply increased complexity.

Best,
Yuxin


ConradJam  于2023年3月17日周五 11:11写道：

> Thanks for your start this discuss
>
>
> Here I am a bit confused about the memory layer definition. This refers to
> local memory. Should we define a data acceleration layer like Alluxio [1]
> in remote storage?
>
>
> Let me cite a scenario: If I use Fluid [2] to mount an AlluxioRuntime [3]
> on K8S, it looks like a local disk (but it is actually a remote memory
> storage), Have we specified this behavior or optimized it for this
> scenario?
>
>
> [1]  What is alluxio :
> https://docs.alluxio.io/os/user/stable/en/Overview.html
>
> [2]  Fluid: https://fluid-cloudnative.github.io/
>
> [3]  Fluid Alluxio Runtime:
>
> https://fluid-cloudnative.github.io/samples/tieredstore_config.html#prerequisites
>
> liu ron  于2023年3月17日周五 10:39写道：
>
> > Hi, Yuxin,
> >
> > Thanks for creating this FLIP. Adding remote storage capability to
> Flink's
> > Hybrid Shuffle is a significant improvement that addresses the issue of
> > local disk storage limitations, this also can improve the stability of
> > Flink Batch Job.
> > I just have one question: can the Hybrid Shuffle replace the RSS in the
> > future? Due to the Hybrid Shuffle having remote storage ability, I think
> > maybe we don't need to maintain a standalone RSS, it will simplify our
> > operation work.
> >
>
>
> --
> Best
>
> ConradJam
>

Re: [DISCUSS] FLIP-301: Hybrid Shuffle supports Remote Storage

[Yuxin Tan]

Hi all,

Thanks for all the feedback and suggestions so far.

The discussion has been going on for some time. If there are no
further comments, we will start voting today.

Best,
Yuxin


Yuxin Tan  于2023年3月17日周五 12:52写道：

> Hi,
> Thanks for joining the discussion and giving the ideas.
>
> @ron
> > can the Hybrid Shuffle replace the RSS in the future?
>
> The hybrid shuffle and RSS offer distinct solutions to address
> the shuffle operation challenge. To optimize performance, we
> store shuffle data in different tiers of memory and disk, enabling
> greater flexibility and ease of use. Specifically, we cache
> intermediate data in memory to reduce disk I/O overhead.
> In contrast, RSS is a standalone service that can operate across
> multiple servers within a cluster, parallelizing shuffle operations
> to enhance performance. However, this introduces additional
> deployment and maintenance costs. Each approach has its own
> benefits and drawbacks, and users should be able to select the
> method that best suits their needs. So I think we cannot replace
> RSS in the future.
>
> @ConradJam
> > Should we define a data acceleration layer like Alluxio in remote
> storage?
>
> I'm not entirely clear on the detailed plan you've proposed, but I
> understand that you want to use Alluxio to serve as a cache layer for
> the remote stoarge tier. It's designed to provide low-latency data
> access to applications through a distributed caching layer. However,
> implementing Alluxio introduces additional dependencies and
> deployment/maintenance costs for users. While our design approach
> is to supplement local storage with remote storage, as local storage
> is generally sufficient. Given the limited usage scenarios, introducing
> such costs for optimization may not be worthwhile or meaningful.
> Additionally, for users, added dependencies imply increased complexity.
>
> Best,
> Yuxin
>
>
> ConradJam  于2023年3月17日周五 11:11写道：
>
>> Thanks for your start this discuss
>>
>>
>> Here I am a bit confused about the memory layer definition. This refers to
>> local memory. Should we define a data acceleration layer like Alluxio [1]
>> in remote storage?
>>
>>
>> Let me cite a scenario: If I use Fluid [2] to mount an AlluxioRuntime [3]
>> on K8S, it looks like a local disk (but it is actually a remote memory
>> storage), Have we specified this behavior or optimized it for this
>> scenario?
>>
>>
>> [1]  What is alluxio :
>> https://docs.alluxio.io/os/user/stable/en/Overview.html
>>
>> [2]  Fluid: https://fluid-cloudnative.github.io/
>>
>> [3]  Fluid Alluxio Runtime:
>>
>> https://fluid-cloudnative.github.io/samples/tieredstore_config.html#prerequisites
>>
>> liu ron  于2023年3月17日周五 10:39写道：
>>
>> > Hi, Yuxin,
>> >
>> > Thanks for creating this FLIP. Adding remote storage capability to
>> Flink's
>> > Hybrid Shuffle is a significant improvement that addresses the issue of
>> > local disk storage limitations, this also can improve the stability of
>> > Flink Batch Job.
>> > I just have one question: can the Hybrid Shuffle replace the RSS in the
>> > future? Due to the Hybrid Shuffle having remote storage ability, I think
>> > maybe we don't need to maintain a standalone RSS, it will simplify our
>> > operation work.
>> >
>>
>>
>> --
>> Best
>>
>> ConradJam
>>
>

[VOTE] FLIP-301: Hybrid Shuffle supports Remote Storage

[Yuxin Tan]

Hi, everyone,

Thanks for all your feedback for FLIP-301: Hybrid Shuffle
supports Remote Storage[1] on the discussion thread[2].

I'd like to start a vote for it. The vote will be open for at
least 72 hours (03/23, 13:00 UTC+8) unless there is an
objection or not enough votes.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-301%3A+Hybrid+Shuffle+supports+Remote+Storage
[2] https://lists.apache.org/thread/nwrqd5jtqwks89tbxpcrgto6r2bhdhno

Best,
Yuxin

Re: [VOTE] FLIP-301: Hybrid Shuffle supports Remote Storage

[Yuxin Tan]

Hi, everyone

I'm closing this vote now. I will follow up with the result in another
email. Thanks

Best,
Yuxin


Yun Tang  于2023年3月21日周二 10:58写道：

> +1 (binding)
>
> Best
> Yun Tang
> 
> From: Zhu Zhu 
> Sent: Tuesday, March 21, 2023 10:07
> To: dev@flink.apache.org 
> Subject: Re: [VOTE] FLIP-301: Hybrid Shuffle supports Remote Storage
>
> +1 (binding)
>
> Thanks,
> Zhu
>
> Xintong Song  于2023年3月20日周一 16:26写道：
> >
> > +1 (binding)
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Mon, Mar 20, 2023 at 4:07 PM weijie guo 
> > wrote:
> >
> > > Thanks Yuxin for driving this.
> > >
> > > +1 (binding)
> > >
> > > Best regards,
> > >
> > > Weijie
> > >
> > >
> > > Junrui Lee  于2023年3月20日周一 14:57写道：
> > >
> > > > +1 (non-binding)
> > > >
> > > > Best regards,
> > > > Junrui
> > > >
> > > > Weihua Hu  于2023年3月20日周一 14:24写道：
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Best,
> > > > > Weihua
> > > > >
> > > > >
> > > > > On Mon, Mar 20, 2023 at 12:39 PM Wencong Liu  >
> > > > wrote:
> > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Wencong Liu
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > At 2023-03-20 12:05:47, "Yuxin Tan" 
> wrote:
> > > > > > >Hi, everyone,
> > > > > > >
> > > > > > >Thanks for all your feedback for FLIP-301: Hybrid Shuffle
> > > > > > >supports Remote Storage[1] on the discussion thread[2].
> > > > > > >
> > > > > > >I'd like to start a vote for it. The vote will be open for at
> > > > > > >least 72 hours (03/23, 13:00 UTC+8) unless there is an
> > > > > > >objection or not enough votes.
> > > > > > >
> > > > > > >[1]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-301%3A+Hybrid+Shuffle+supports+Remote+Storage
> > > > > > >[2]
> > > https://lists.apache.org/thread/nwrqd5jtqwks89tbxpcrgto6r2bhdhno
> > > > > > >
> > > > > > >Best,
> > > > > > >Yuxin
> > > > > >
> > > > >
> > > >
> > >
>

[RESULT][VOTE] FLIP-301: Hybrid Shuffle supports Remote Storage

[Yuxin Tan]

Hi all,

I am happy to announce that FLIP-301: Hybrid Shuffle supports Remote
Storage[1] has been accepted. The FLIP was voted in this thread[2].

There are 4 binding votes and 3 non-binding votes:

- Wencong Liu (non-binding)
- Weihua Hu (non-binding)
- Junrui Lee (non-binding)
- weijie guo (binding)
- Xintong Song (binding)
- Zhu Zhu (binding)
- Yun Tang (binding)

There is no disapproving vote. Thanks.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-301%3A+Hybrid+Shuffle+supports+Remote+Storage
[2] https://lists.apache.org/thread/xpmhpmodzlwo03n6zzovy36gox84l6zl

Best,
Yuxin

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

[Yuxin Tan]

Congratulations！

Best,
Yuxin


Guanghui Zhang  于2023年3月28日周二 11:06写道：

> Congratulations！
>
> Best,
> Zhang Guanghui
>
> Hang Ruan  于2023年3月28日周二 10:29写道：
>
> > Congratulations!
> >
> > Best,
> > Hang
> >
> > yu zelin  于2023年3月28日周二 10:27写道：
> >
> >> Congratulations!
> >>
> >> Best,
> >> Yu Zelin
> >>
> >> 2023年3月27日 17:23，Yu Li  写道：
> >>
> >> Dear Flinkers,
> >>
> >>
> >>
> >> As you may have noticed, we are pleased to announce that Flink Table
> Store has joined the Apache Incubator as a separate project called Apache
> Paimon(incubating) [1] [2] [3]. The new project still aims at building a
> streaming data lake platform for high-speed data ingestion, change data
> tracking and efficient real-time analytics, with the vision of supporting a
> larger ecosystem and establishing a vibrant and neutral open source
> community.
> >>
> >>
> >>
> >> We would like to thank everyone for their great support and efforts for
> the Flink Table Store project, and warmly welcome everyone to join the
> development and activities of the new project. Apache Flink will continue
> to be one of the first-class citizens supported by Paimon, and we believe
> that the Flink and Paimon communities will maintain close cooperation.
> >>
> >>
> >> 亲爱的Flinkers，
> >>
> >>
> >> 正如您可能已经注意到的，我们很高兴地宣布，Flink Table Store 已经正式加入 Apache
> >> 孵化器独立孵化 [1] [2] [3]。新项目的名字是
> >> Apache
> Paimon(incubating)，仍致力于打造一个支持高速数据摄入、流式数据订阅和高效实时分析的新一代流式湖仓平台。此外，新项目将支持更加丰富的生态，并建立一个充满活力和中立的开源社区。
> >>
> >>
> >> 在这里我们要感谢大家对 Flink Table Store
> >> 项目的大力支持和投入，并热烈欢迎大家加入新项目的开发和社区活动。Apache Flink 将继续作为 Paimon
> 支持的主力计算引擎之一，我们也相信
> >> Flink 和 Paimon 社区将继续保持密切合作。
> >>
> >>
> >> Best Regards,
> >> Yu (on behalf of the Apache Flink PMC and Apache Paimon PPMC)
> >>
> >> 致礼，
> >> 李钰（谨代表 Apache Flink PMC 和 Apache Paimon PPMC）
> >>
> >> [1] https://paimon.apache.org/
> >> [2] https://github.com/apache/incubator-paimon
> >> [3]
> https://cwiki.apache.org/confluence/display/INCUBATOR/PaimonProposal
> >>
> >>
> >>
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Qingsheng Ren

[Yuxin Tan]

Congratulations, Qingsheng!

Best,
Yuxin


Ahmed Hamdy  于2023年4月22日周六 02:20写道：

> Congratulations Qingsheng.
> Best regards
> Ahmed
>
> On Fri, 21 Apr 2023 at 17:22, Samrat Deb  wrote:
>
> > congratulations !
> >
> > On Fri, 21 Apr 2023 at 9:45 PM, David Morávek  wrote:
> >
> > > Congratulations, Qingsheng, well deserved!
> > >
> > > Best,
> > > D.
> > >
> > > On Fri 21. 4. 2023 at 16:41, Feng Jin  wrote:
> > >
> > > > Congratulations, Qingsheng
> > > >
> > > >
> > > > 
> > > > Best,
> > > > Feng Jin
> > > >
> > > > On Fri, Apr 21, 2023 at 8:39 PM Mang Zhang 
> wrote:
> > > >
> > > > > Congratulations, Qingsheng.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Best regards,
> > > > > Mang Zhang
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > At 2023-04-21 19:50:02, "Jark Wu"  wrote:
> > > > > >Hi everyone,
> > > > > >
> > > > > >We are thrilled to announce that Qingsheng Ren has joined the
> Flink
> > > PMC!
> > > > > >
> > > > > >Qingsheng has been contributing to Apache Flink for a long time.
> He
> > is
> > > > the
> > > > > >core contributor and maintainer of the Kafka connector and
> > > > > >flink-cdc-connectors, bringing users stability and ease of use in
> > both
> > > > > >projects. He drove discussions and implementations in FLIP-221,
> > > > FLIP-288,
> > > > > >and the connector testing framework. He is continuously helping
> with
> > > the
> > > > > >expansion of the Flink community and has given several talks about
> > > Flink
> > > > > >connectors at many conferences, such as Flink Forward Global and
> > Flink
> > > > > >Forward Asia. Besides that, he is willing to help a lot in the
> > > community
> > > > > >work, such as being the release manager for both 1.17 and 1.18,
> > > > verifying
> > > > > >releases, and answering questions on the mailing list.
> > > > > >
> > > > > >Congratulations and welcome Qingsheng!
> > > > > >
> > > > > >Best,
> > > > > >Jark (on behalf of the Flink PMC)
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Leonard Xu

[Yuxin Tan]

Congratulations, Leonard!

Best,
Yuxin


Panagiotis Garefalakis  于2023年4月22日周六 08:08写道：

> Congrats Leonard!
>
> On Fri, Apr 21, 2023 at 11:19 AM Ahmed Hamdy  wrote:
>
> > Congratulations Leonard.
> > Best Regards
> > Ahmed
> >
> > On Fri, 21 Apr 2023 at 17:23, Samrat Deb  wrote:
> >
> > > congratulations
> > >
> > > On Fri, 21 Apr 2023 at 9:44 PM, David Morávek  wrote:
> > >
> > > > Congratulations, Leonard, well deserved!
> > > >
> > > > Best,
> > > > D.
> > > >
> > > > On Fri 21. 4. 2023 at 16:40, Feng Jin  wrote:
> > > >
> > > > > Congratulations, Leonard
> > > > >
> > > > >
> > > > > 
> > > > > Best,
> > > > > Feng Jin
> > > > >
> > > > > On Fri, Apr 21, 2023 at 8:38 PM Mang Zhang 
> > wrote:
> > > > >
> > > > > > Congratulations, Leonard.
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Best regards,
> > > > > > Mang Zhang
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > At 2023-04-21 19:47:52, "Jark Wu"  wrote:
> > > > > > >Hi everyone,
> > > > > > >
> > > > > > >We are thrilled to announce that Leonard Xu has joined the Flink
> > > PMC!
> > > > > > >
> > > > > > >Leonard has been an active member of the Apache Flink community
> > for
> > > > many
> > > > > > >years and became a committer in Nov 2021. He has been involved
> in
> > > > > various
> > > > > > >areas of the project, from code contributions to community
> > building.
> > > > His
> > > > > > >contributions are mainly focused on Flink SQL and connectors,
> > > > especially
> > > > > > >leading the flink-cdc-connectors project to receive 3.8+K GitHub
> > > > stars.
> > > > > He
> > > > > > >authored 150+ PRs, and reviewed 250+ PRs, and drove several
> FLIPs
> > > > (e.g.,
> > > > > > >FLIP-132, FLIP-162). He has participated in plenty of
> discussions
> > in
> > > > the
> > > > > > >dev mailing list, answering questions about 500+ threads in the
> > > > > > >user/user-zh mailing list. Besides that, he is community minded,
> > > such
> > > > as
> > > > > > >being the release manager of 1.17, verifying releases, managing
> > > > release
> > > > > > >syncs, etc.
> > > > > > >
> > > > > > >Congratulations and welcome Leonard!
> > > > > > >
> > > > > > >Best,
> > > > > > >Jark (on behalf of the Flink PMC)
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Release 1.16.2, release candidate #1

[Yuxin Tan]

+1 (non-binding)

- Verified signature
- Verified hashes
- Build form source with mac
- Verify that the source archives do not contain any binaries
- Run streaming and batch job in sql-client successfully.

Thanks weijie for driving this release candidate.

Best,
Yuxin


weijie guo  于2023年5月19日周五 16:19写道：

> Hi everyone,
>
>
> Please review and vote on the release candidate #1 for the version 1.16.2,
>
> as follows:
>
>
> [ ] +1, Approve the release
>
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
> The complete staging area is available for your review, which includes:
>
> * JIRA release notes [1],
>
> * the official Apache source release and binary convenience releases to be
>
> deployed to dist.apache.org [2], which are signed with the key with
>
> fingerprint 8D56AE6E7082699A4870750EA4E8C4C05EE6861F [3],
>
> * all artifacts to be deployed to the Maven Central Repository [4],
>
> * source code tag "release-1.16.2-rc1" [5],
>
> * website pull request listing the new release and adding announcement blog
>
> post [6].
>
>
> The vote will be open for at least 72 hours. It is adopted by majority
>
> approval, with at least 3 PMC affirmative votes.
>
>
> Thanks,
>
> Release Manager
>
>
> [1]
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352765
>
> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.16.2-rc1/
>
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>
> [4]
> https://repository.apache.org/content/repositories/orgapacheflink-1634/
>
> [5] https://github.com/apache/flink/releases/tag/release-1.16.2-rc1
>
> [6] https://github.com/apache/flink-web/pull/649
>

Re: [VOTE] Release 1.17.1, release candidate #1

[Yuxin Tan]

+1 (non-binding)

- verified the signatures
- verified the checksums
- built from source
- start a standalone cluster, run a simple batch and a streaming job
successfully
- review release announcement PR

Best,
Yuxin


Xintong Song  于2023年5月22日周一 18:24写道：

> +1 (binding)
>
> - verified signatures and checksums
> - built from source
> - tried example jobs with a standalone cluster, everything seems fine
> - review release announcement PR
>
> Best,
>
> Xintong
>
>
>
> On Mon, May 22, 2023 at 2:18 PM weijie guo 
> wrote:
>
> > Hi everyone,
> >
> >
> > Please review and vote on the release candidate #1 for the version
> 1.17.1,
> >
> > as follows:
> >
> > [ ] +1, Approve the release
> >
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> >
> > The complete staging area is available for your review, which includes:
> >
> >
> > * JIRA release notes [1],
> >
> > * the official Apache source release and binary convenience releases to
> be
> >
> > deployed to dist.apache.org [2], which are signed with the key with
> >
> > fingerprint 8D56AE6E7082699A4870750EA4E8C4C05EE6861F [3],
> >
> > * all artifacts to be deployed to the Maven Central Repository [4],
> >
> > * source code tag "release-1.17.1-rc1" [5],
> >
> > * website pull request listing the new release and adding announcement
> blog
> >
> > post [6].
> >
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> >
> > approval, with at least 3 PMC affirmative votes.
> >
> >
> > Thanks,
> >
> > Release Manager
> >
> >
> > [1]
> >
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352886
> >
> > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.17.1-rc1/
> >
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> >
> > [4]
> > https://repository.apache.org/content/repositories/orgapacheflink-1635/
> >
> > [5] https://github.com/apache/flink/releases/tag/release-1.17.1-rc1
> >
> > [6] https://github.com/apache/flink-web/pull/650
> >
>

Re: [VOTE] Release flink-connector-jdbc v3.1.1, release candidate #1

[Yuxin Tan]

+1 (non-binding)

- Checked sign
- Checked the hash
- Checked tag
- Build from source

Best,
Yuxin


weijie guo  于2023年5月29日周一 14:14写道：

> +1 (non-binding)
>
> - checked sign and checksum
> - checked tag in github repository
> - compiled from source
> - checked the web PR
>
> BTW, please remember to update docs/jdbc.yaml for the v3.1 branch after the
> release is completed.
>
> Best regards,
>
> Weijie
>
>
> Jing Ge  于2023年5月29日周一 04:26写道：
>
> > +1 (non-binding)
> >
> > - checked sign
> > - checked hash
> > - checked repos
> > - checked tag
> > - compiled from source
> > - check the web PR
> >
> > Best regards,
> > Jing
> >
> >
> > On Sun, May 28, 2023 at 4:00 PM Benchao Li  wrote:
> >
> > > Thanks Martijn,
> > >
> > > - checked signature/checksum [OK]
> > > - downloaded src, compiled from source [OK]
> > > - diffed src and tag, no binary files [OK]
> > > - gone through nexus staging area, looks good [OK]
> > > - run with flink 1.7.1 [OK]
> > >
> > > One thing I spotted is that the version in `docs/data/jdbc.yml` is
> still
> > > 3.1.0, I'm not sure whether this should be a blocker.
> > >
> > >
> > > Martijn Visser  于2023年5月25日周四 02:55写道：
> > >
> > > > Hi everyone,
> > > > Please review and vote on the release candidate #1 for the version
> > 3.1.1,
> > > > as follows:
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > >
> > > > The complete staging area is available for your review, which
> includes:
> > > > * JIRA release notes [1],
> > > > * the official Apache source release to be deployed to
> dist.apache.org
> > > > [2],
> > > > which are signed with the key with fingerprint
> > > > A5F3BCE4CBE993573EC5966A65321B8382B219AF [3],
> > > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > > * source code tag v3.1.1-rc1 [5],
> > > > * website pull request listing the new release [6].
> > > >
> > > > The vote will be open for at least 72 hours. It is adopted by
> majority
> > > > approval, with at least 3 PMC affirmative votes.
> > > >
> > > > Thanks,
> > > > Release Manager
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12353281
> > > > [2]
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-jdbc-3.1.1-rc1
> > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > [4]
> > > >
> > https://repository.apache.org/content/repositories/orgapacheflink-1636/
> > > > [5]
> > > https://github.com/apache/flink-connector-jdbc/releases/tag/v3.1.1-rc1
> > > > [6] https://github.com/apache/flink-web/pull/654
> > > >
> > >
> > >
> > > --
> > >
> > > Best,
> > > Benchao Li
> > >
> >
>

Re: [VOTE] FLIP-366: Support standard YAML for FLINK configuration

[Yuxin Tan]

+1(non-binding)

Best,
Yuxin


Zhanghao Chen  于2023年10月13日周五 10:54写道：

> +1 (non-binding)
>
> Best,
> Zhanghao Chen
> 
> From: Junrui Lee 
> Sent: Friday, October 13, 2023 10:12
> To: dev@flink.apache.org 
> Subject: [VOTE] FLIP-366: Support standard YAML for FLINK configuration
>
> Hi all,
>
> Thank you to everyone for the feedback on FLIP-366[1]: Support standard
> YAML for FLINK configuration in the discussion thread [2].
> I would like to start a vote for it. The vote will be open for at least 72
> hours (excluding weekends, unless there is an objection or an insufficient
> number of votes).
>
> Thanks,
> Junrui
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration
> [2]https://lists.apache.org/thread/qfhcm7h8r5xkv38rtxwkghkrcxg0q7k5
>

Re: Flink and Flink shaded dependency

[Yuxin Tan]

Hi, devs,

I would like to revive the discussion again.

In our ARM environment, we encounter a compile error when
using Flink 1.17.

Flink 1.17 depends on flink-shaded 16.1, which uses netty 4.1.82.
However, flink-shaded 16.1 fails to compile in the ARM
environment. As a result, we are unable to compile Flink 1.17
due to this issue.

We have tested compiling flink-shaded using netty 4.1.83 or
a later version in ARM env, and it can compile successfully.

I believe this is a critical bug that needs to be addressed first.

Taking into consideration the previous discussions regarding
compatibility and the dependency of external connectors on
this version, I propose addressing the bug by only updating
flink-shaded's netty to a minor version (e.g., 4.1.83) rather than
backporting FLINK-32032. This approach can avoid introducing
other changes, such as updating to a higher netty version (4.1.91),
potential guava version alterations, Curator version modifications,
and so on.

WDYT?


Best,
Yuxin


Jing Ge  于2023年10月5日周四 14:39写道：

> Hi Chesnay,
>
> Thanks for joining this discussion and sharing your thoughts!
>
>
> > Connectors shouldn't depend on flink-shaded.
> >
>
> Perfect! We are on the same page. If you could read through the discussion,
> you would realize that, currently, there are many connectors depend on
> flink-shaded.
>
>
> > Connectors are small enough in scope that depending directly on
> > guava/jackson/etc. is a fine approach, and they have plenty of other
> > dependencies that they need to manage anyway; let's treat these the same
> > way.
> >
>
> It is even better, if we could do that. Jira tickest[1] are created.
>
>
> > As for class-loading, there has been a long-standing goal of each
> > connector being loaded in their own classloader. That still is the north
> > star and the only reasonable way to ensure that multiple connectors can
> > be safely used with SQL.
> >
>
> What is the current status? Do we have any Jira ticket for that?
>
> Best regards,
> Jing
>
> [1] https://issues.apache.org/jira/browse/FLINK-33190
>
> On Wed, Oct 4, 2023 at 4:43 PM Chesnay Schepler 
> wrote:
>
> > There is no "monolithic" flink-shaded dependency.
> > Connectors shouldn't depend on anything that Flink provides, but be
> > self-contained as Martijn pointed out.
> >
> >
>
> > Connectors shouldn't depend on flink-shaded.
> >
>
>
> > The overhead and/or risks of doing/supporting that right now far
> > outweigh the benefits.
> > ( Because we either have to encode the full version for all dependencies
> > into the package, or accept the risk of minor/patch dependency clashes)
> >
>
>
> > Connectors are small enough in scope that depending directly on
> > guava/jackson/etc. is a fine approach, and they have plenty of other
> > dependencies that they need to manage anyway; let's treat these the same
> > way.
> >
>
>
> > Naturally this is also an argument against flink-shaded-connectors; on
> > top of that we already experience repo creep and managing releases is
> > difficult enough as-is.
> >
> >
>
> > As for class-loading, there has been a long-standing goal of each
> > connector being loaded in their own classloader. That still is the north
> > star and the only reasonable way to ensure that multiple connectors can
> > be safely used with SQL.
> >
>
>
> > On 02/10/2023 18:32, Jing Ge wrote:
> > > Hi Sergey,
> > >
> > > Thanks for sharing your thoughts. It could somehow help but didn't get
> to
> > > the root of this issue.
> > >
> > > According to the documentation, Flink shaded is used to provide a
> single
> > > instance of a shaded dependency across sub-modules in Flink repo.
> Shaded
> > > namespaces should be used where shaded dependencies are configured.
> After
> > > connectors have been externalized, it ends up with more repos depending
> > on
> > > one shaded jar, e.g. guava. This is a "monolithic" dependency setup
> that
> > > makes it difficult to change the root(flink-shade), because any changes
> > of
> > > the root have to be propagated to all downstream repos. Even worse is
> > that
> > > not every downstream repo is known while modifying the root.
> > >
> > > Since all externalized connectors have their own repos and are not
> > > sub-modules of Flink anymore, I would suggest the following upgrade:
> > >
> > > 1. Connectors should use their own classloader instead of Flink's
> > > classloader. This will break the monolithic dependency. Connectors and
> > > Flink can use different versions of flink-shaded.
> > > 2. [optional] It would be even better that all connector repos depend
> on
> > > their own individual shaded repo, e.g. flink-connector-shaded.
> > flink-shaded
> > > should only be used by Flink.
> > >
> > > WDYT?
> > >
> > > Best regards,
> > > Jing
> > >
> > >
> > > On Thu, Sep 14, 2023 at 11:28 PM Sergey Nuyanzin 
> > > wrote:
> > >
> > >> Yes, that's a reasonable question, thanks for raising it.
> > >>
> > >> I think this is not only about flink-shaded, rather about dependencies
> > in
>

Re: [ANNOUNCE] New Apache Flink Committer - Jane Chan

[Yuxin Tan]

Congratulations Jane!

Best,
Yuxin


xiangyu feng  于2023年10月16日周一 10:27写道：

> Congratulations Jane!
>
> Best,
> Xiangyu
>
> Xuannan Su  于2023年10月16日周一 10:25写道：
>
> > Congratulations Jane!
> >
> > Best,
> > Xuannan
> >
> > On Mon, Oct 16, 2023 at 10:21 AM Yun Tang  wrote:
> > >
> > > Congratulations, Jane!
> > >
> > > Best
> > > Yun Tang
> > > 
> > > From: Rui Fan <1996fan...@gmail.com>
> > > Sent: Monday, October 16, 2023 10:16
> > > To: dev@flink.apache.org 
> > > Cc: qingyue@gmail.com 
> > > Subject: Re: [ANNOUNCE] New Apache Flink Committer - Jane Chan
> > >
> > > Congratulations Jane!
> > >
> > > Best,
> > > Rui
> > >
> > > On Mon, Oct 16, 2023 at 10:15 AM yu zelin 
> wrote:
> > >
> > > > Congratulations!
> > > >
> > > > Best,
> > > > Yu Zelin
> > > >
> > > > > 2023年10月16日 09:58，Jark Wu  写道：
> > > > >
> > > > > Hi, everyone
> > > > >
> > > > > On behalf of the PMC, I'm very happy to announce Jane Chan as a new
> > Flink
> > > > > Committer.
> > > > >
> > > > > Jane started code contribution in Jan 2021 and has been active in
> the
> > > > Flink
> > > > > community since. She authored more than 60 PRs and reviewed more
> > than 40
> > > > > PRs. Her contribution mainly revolves around Flink SQL, including
> > Plan
> > > > > Advice (FLIP-280), operator-level state TTL (FLIP-292), and ALTER
> > TABLE
> > > > > statements (FLINK-21634). Jane participated deeply in development
> > > > > discussions and also helped answer user question emails. Jane was
> > also a
> > > > > core contributor of Flink Table Store (now Paimon) when the project
> > was
> > > > in
> > > > > the early days.
> > > > >
> > > > > Please join me in congratulating Jane Chan for becoming a Flink
> > > > Committer!
> > > > >
> > > > > Best,
> > > > > Jark Wu (on behalf of the Flink PMC)
> > > >
> > > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Ron Liu

[Yuxin Tan]

Congratulations, Ron!

Best,
Yuxin


Junrui Lee  于2023年10月16日周一 10:24写道：

> Congratulations Ron !
>
> Best,
> Junrui
>
> Yun Tang  于2023年10月16日周一 10:22写道：
>
> > Congratulations, Ron!
> >
> > Best
> > Yun Tang
> > 
> > From: yu zelin 
> > Sent: Monday, October 16, 2023 10:16
> > To: dev@flink.apache.org 
> > Cc: ron9@gmail.com 
> > Subject: Re: [ANNOUNCE] New Apache Flink Committer - Ron Liu
> >
> > Congratulations!
> >
> > Best,
> > Yu Zelin
> >
> > > 2023年10月16日 09:56，Jark Wu  写道：
> > >
> > > Hi, everyone
> > >
> > > On behalf of the PMC, I'm very happy to announce Ron Liu as a new Flink
> > > Committer.
> > >
> > > Ron has been continuously contributing to the Flink project for many
> > years,
> > > authored and reviewed a lot of codes. He mainly works on Flink SQL
> parts
> > > and drove several important FLIPs, e.g., USING JAR (FLIP-214), Operator
> > > Fusion CodeGen (FLIP-315), Runtime Filter (FLIP-324). He has a great
> > > knowledge of the Batch SQL and improved a lot of batch performance in
> the
> > > past several releases. He is also quite active in mailing lists,
> > > participating in discussions and answering user questions.
> > >
> > > Please join me in congratulating Ron Liu for becoming a Flink
> Committer!
> > >
> > > Best,
> > > Jark Wu (on behalf of the Flink PMC)
> >
> >
>

Re: Flink and Flink shaded dependency

[Yuxin Tan]

Hi, Matthias

Thanks a lot for joining the discussion.

> Generally, netty upgrades are a source of instability based on past
experiences. But considering that we have newer versions (4.1.91.Final)
tested with flink-shaded 17.0 in Flink 1.18, it should be alright bumping
netty in flink-shaded 16.2 to netty 4.1.83.Final.

Agree with your consideration, this also is the reason why we chose
4.1.83.Final.

> If no other objections are shared, the next step would be to create a Jira
issue, do the change in flink-shaded, release flink-shaded 16.2 and update
the flink-shaded version in the 1.17 branch.

If there are no objections, I'm ready to create a Jira ticket and prepare
the
PR for the change. However, it may need assistance from a committer for
the subsequent steps. It would be very appreciated if someone could take the
releasing task. Thanks

Best,
Yuxin


Matthias Pohl  于2023年10月20日周五 20:30写道：

>  IHi Yuxin,
> That would be the way to go for flink-shaded and Flink 1.17.
>
> Generally, netty upgrades are a source of instability based on past
> experiences. But considering that we have newer versions (4.1.91.Final)
> tested with flink-shaded 17.0 in Flink 1.18, it should be alright bumping
> netty in flink-shaded 16.2 to netty 4.1.83.Final.
>
> One argument against such an upgrade would be that the Flink community is
> not supporting arm officially (i.e. there's no CI setup). In this sense I
> want to wait whether other's have objections against doing the upgrade.
>
> If no other objections are shared, the next step would be to create a Jira
> issue, do the change in flink-shaded, release flink-shaded 16.2 and update
> the flink-shaded version in the 1.17 branch.
>
> Matthias
>
> On Mon, Oct 16, 2023 at 5:07 AM Yuxin Tan  wrote:
>
> > Hi, devs,
> >
> > I would like to revive the discussion again.
> >
> > In our ARM environment, we encounter a compile error when
> > using Flink 1.17.
> >
> > Flink 1.17 depends on flink-shaded 16.1, which uses netty 4.1.82.
> > However, flink-shaded 16.1 fails to compile in the ARM
> > environment. As a result, we are unable to compile Flink 1.17
> > due to this issue.
> >
> > We have tested compiling flink-shaded using netty 4.1.83 or
> > a later version in ARM env, and it can compile successfully.
> >
> > I believe this is a critical bug that needs to be addressed first.
> >
> > Taking into consideration the previous discussions regarding
> > compatibility and the dependency of external connectors on
> > this version, I propose addressing the bug by only updating
> > flink-shaded's netty to a minor version (e.g., 4.1.83) rather than
> > backporting FLINK-32032. This approach can avoid introducing
> > other changes, such as updating to a higher netty version (4.1.91),
> > potential guava version alterations, Curator version modifications,
> > and so on.
> >
> > WDYT?
> >
> >
> > Best,
> > Yuxin
> >
> >
> > Jing Ge  于2023年10月5日周四 14:39写道：
> >
> > > Hi Chesnay,
> > >
> > > Thanks for joining this discussion and sharing your thoughts!
> > >
> > >
> > > > Connectors shouldn't depend on flink-shaded.
> > > >
> > >
> > > Perfect! We are on the same page. If you could read through the
> > discussion,
> > > you would realize that, currently, there are many connectors depend on
> > > flink-shaded.
> > >
> > >
> > > > Connectors are small enough in scope that depending directly on
> > > > guava/jackson/etc. is a fine approach, and they have plenty of other
> > > > dependencies that they need to manage anyway; let's treat these the
> > same
> > > > way.
> > > >
> > >
> > > It is even better, if we could do that. Jira tickest[1] are created.
> > >
> > >
> > > > As for class-loading, there has been a long-standing goal of each
> > > > connector being loaded in their own classloader. That still is the
> > north
> > > > star and the only reasonable way to ensure that multiple connectors
> can
> > > > be safely used with SQL.
> > > >
> > >
> > > What is the current status? Do we have any Jira ticket for that?
> > >
> > > Best regards,
> > > Jing
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-33190
> > >
> > > On Wed, Oct 4, 2023 at 4:43 PM Chesnay Schepler 
> > > wrote:
> > >
> > > > There is no "monolithic" flink-shaded dependency.
> > > > Connect

Re: [VOTE] Release 1.18.0, release candidate #3

[Yuxin Tan]

+1(non-binding)

- Verified checksum
- Build from source code
- Verified signature
- Started a local cluster and run Streaming & Batch wordcount job, the
result is expected
- Verified web PR

Best,
Yuxin


Qingsheng Ren  于2023年10月24日周二 11:19写道：

> +1 (binding)
>
> - Verified checksums and signatures
> - Built from source with Java 8
> - Started a standalone cluster and submitted a Flink SQL job that read and
> wrote with Kafka connector and CSV / JSON format
> - Reviewed web PR and release note
>
> Best,
> Qingsheng
>
> On Mon, Oct 23, 2023 at 10:40 PM Leonard Xu  wrote:
>
> > +1 (binding)
> >
> > - verified signatures
> > - verified hashsums
> > - built from source code succeeded
> > - checked all dependency artifacts are 1.18
> > - started SQL Client, used MySQL CDC connector to read changelog from
> > database , the result is expected
> > - reviewed the web PR, left minor comments
> > - reviewed the release notes PR, left minor comments
> >
> >
> > Best,
> > Leonard
> >
> > > 2023年10月21日 下午7:28，Rui Fan <1996fan...@gmail.com> 写道：
> > >
> > > +1(non-binding)
> > >
> > > - Downloaded artifacts from dist[1]
> > > - Verified SHA512 checksums
> > > - Verified GPG signatures
> > > - Build the source with java-1.8 and verified the licenses together
> > > - Verified web PR
> > >
> > > [1] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc3/
> > >
> > > Best,
> > > Rui
> > >
> > > On Fri, Oct 20, 2023 at 10:31 PM Martijn Visser <
> > martijnvis...@apache.org>
> > > wrote:
> > >
> > >> +1 (binding)
> > >>
> > >> - Validated hashes
> > >> - Verified signature
> > >> - Verified that no binaries exist in the source archive
> > >> - Build the source with Maven
> > >> - Verified licenses
> > >> - Verified web PR
> > >> - Started a cluster and the Flink SQL client, successfully read and
> > >> wrote with the Kafka connector to Confluent Cloud with AVRO and Schema
> > >> Registry enabled
> > >>
> > >> On Fri, Oct 20, 2023 at 2:55 PM Matthias Pohl
> > >>  wrote:
> > >>>
> > >>> +1 (binding)
> > >>>
> > >>> * Downloaded artifacts
> > >>> * Built Flink from sources
> > >>> * Verified SHA512 checksums GPG signatures
> > >>> * Compared checkout with provided sources
> > >>> * Verified pom file versions
> > >>> * Verified that there are no pom/NOTICE file changes since RC1
> > >>> * Deployed standalone session cluster and ran WordCount example in
> > batch
> > >>> and streaming: Nothing suspicious in log files found
> > >>>
> > >>> On Thu, Oct 19, 2023 at 3:00 PM Piotr Nowojski  >
> > >> wrote:
> > >>>
> >  +1 (binding)
> > 
> >  Best,
> >  Piotrek
> > 
> >  czw., 19 paź 2023 o 09:55 Yun Tang  napisał(a):
> > 
> > > +1 (non-binding)
> > >
> > >
> > >  *   Build from source code
> > >  *   Verify the pre-built jar packages were built with JDK8
> > >  *   Verify FLIP-291 with a standalone cluster, and it works fine
> > >> with
> > > StateMachine example.
> > >  *   Checked the signature
> > >  *   Viewed the PRs.
> > >
> > > Best
> > > Yun Tang
> > > 
> > > From: Cheng Pan 
> > > Sent: Thursday, October 19, 2023 14:38
> > > To: dev@flink.apache.org 
> > > Subject: RE: [VOTE] Release 1.18.0, release candidate #3
> > >
> > > +1 (non-binding)
> > >
> > > We(the Apache Kyuubi community), verified that the Kyuubi Flink
> > >> engine
> > > works well[1] with Flink 1.18.0 RC3.
> > >
> > > [1] https://github.com/apache/kyuubi/pull/5465
> > >
> > > Thanks,
> > > Cheng Pan
> > >
> > >
> > > On 2023/10/19 00:26:24 Jing Ge wrote:
> > >> Hi everyone,
> > >>
> > >> Please review and vote on the release candidate #3 for the version
> > >> 1.18.0, as follows:
> > >> [ ] +1, Approve the release
> > >> [ ] -1, Do not approve the release (please provide specific
> > >> comments)
> > >>
> > >> The complete staging area is available for your review, which
> > >> includes:
> > >>
> > >> * JIRA release notes [1], and the pull request adding release note
> > >> for
> > >> users [2]
> > >> * the official Apache source release and binary convenience
> > >> releases to
> > > be
> > >> deployed to dist.apache.org [3], which are signed with the key
> > >> with
> > >> fingerprint 96AE0E32CBE6E0753CE6 [4],
> > >> * all artifacts to be deployed to the Maven Central Repository
> [5],
> > >> * source code tag "release-1.18.0-rc3" [6],
> > >> * website pull request listing the new release and adding
> > >> announcement
> > > blog
> > >> post [7].
> > >>
> > >> The vote will be open for at least 72 hours. It is adopted by
> > >> majority
> > >> approval, with at least 3 PMC affirmative votes.
> > >>
> > >> Best regards,
> > >> Konstantin, Sergey, Qingsheng, and Jing
> > >>
> > >> [1]
> > >>
> > >
> > 
> > >>
> >
> https://issues.apache.org/jira/secure/Rele

Re: [DISCUSS] FLIP-381: Deprecate configuration getters/setters that return/set complex Java objects

[Yuxin Tan]

Thanks Junrui for driving the proposal.

+1 for this proposal. I believe this change will enhance the usability of
Flink configuration for both users and developers, while also ensuring
consistency across various types of configurations.

Best,
Yuxin


Lijie Wang  于2023年11月3日周五 10:59写道：

> Thanks Junrui for driving this.
>
> Making configurations simple and consistent has great benefits for both
> users and devs. +1 for the proposal.
>
> Best,
> Lijie
>
> weijie guo  于2023年11月2日周四 16:49写道：
>
> > Thanks Junrui for driving this proposal!
> >
> > I believe this is helpful for the new Process Function API. Because we
> > don't need to move some related class/components from flink-core to a
> pure
> > API module (maybe, called flink-core-api) after this. Even though the
> FLIP
> > related to new API is in preparation atm, I still want to emphasize our
> > goal is that user application should no longer depend on these stuff. So
> > I'm + 1 for this proposal.
> >
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > Zhu Zhu  于2023年11月2日周四 16:00写道：
> >
> > > Thanks Junrui for creating the FLIP and kicking off this discussion.
> > >
> > > The community has been constantly striving to unify and simplify the
> > > configuration layer of Flink. Some progress has already been made,
> > > such as FLINK-29379. However, the compatibility of public interfaces
> > > poses an obstacle to completing the task. The release of Flink 2.0
> > > presents a great opportunity to accomplish this goal.
> > >
> > > +1 for the proposal.
> > >
> > > Thanks,
> > > Zhu
> > >
> > > Rui Fan <1996fan...@gmail.com> 于2023年11月2日周四 10:27写道：
> > >
> > > > Thanks Junrui for driving this proposal!
> > > >
> > > > ConfigOption is easy to use for flink users, easy to manage options
> > > > for flink platform maintainers, and easy to maintain for flink
> > developers
> > > > and flink community.
> > > >
> > > > So big +1 for this proposal!
> > > >
> > > > Best,
> > > > Rui
> > > >
> > > > On Thu, Nov 2, 2023 at 10:10 AM Junrui Lee 
> > wrote:
> > > >
> > > > > Hi devs,
> > > > >
> > > > > I would like to start a discussion on FLIP-381: Deprecate
> > configuration
> > > > > getters/setters that return/set complex Java objects[1].
> > > > >
> > > > > Currently, the job configuration in FLINK is spread out across
> > > different
> > > > > components, which leads to inconsistencies and confusion. To
> address
> > > this
> > > > > issue, it is necessary to migrate non-ConfigOption complex Java
> > objects
> > > > to
> > > > > use ConfigOption and adopt a single Configuration object to host
> all
> > > the
> > > > > configuration.
> > > > > However, there is a significant blocker in implementing this
> > solution.
> > > > > These complex Java objects in StreamExecutionEnvironment,
> > > > CheckpointConfig,
> > > > > and ExecutionConfig have already been exposed through the public
> API,
> > > > > making it challenging to modify the existing implementation.
> > > > >
> > > > > Therefore, I propose to deprecate these Java objects and their
> > > > > corresponding getter/setter interfaces, ultimately removing them in
> > > > > FLINK-2.0.
> > > > >
> > > > > Your feedback and thoughts on this proposal are highly appreciated.
> > > > >
> > > > > Best regards,
> > > > > Junrui Lee
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278464992
> > > > >
> > > >
> > >
> >
>

[DISCUSS] Release flink-shaded 16.2

[Yuxin Tan]

Hi, all,

I would like to discuss creating a new 16.2 patch release for
flink-shaded[1].

In our ARM environment, we recently encountered a critical bug
(FLINK-33417[2])
while using flink-shaded 16.1 in flink 1.17. The issue has been discussed
in the
discussion[3]. Fortunately, we could resolve this bug by updating the netty
version.

To address this issue comprehensively, we need to release a new minor
version for
flink-shaded. As a result, Flink 1.17 should update its dependent
flink-shaded version
accordingly. However, there is no need for Flink 1.18+ to update the
flink-shaded
version since these newer versions already depend on flink-shaded 17.0+ and
have
updated their netty version, thereby eliminating the existence of this bug.

Considering the low probability of this issue occurring, I do not recommend
considering it as a blocker for the Flink 1.17 release. Furthermore, the
external connector has been decoupled from flink-shaded in FLINK-33190[4],
and the compatibility of the newer netty version 4.1.91 has been verified
in the new
Flink 1.18+(which is dependent on flink-shaded 17.0 and it is dependent on
netty 4.1.91). Hence, we believe this update will not cause any
compatibility issues.

If there are no objections, we would greatly appreciate it if a committer
could
volunteer as the release manager. Thank you.

[1] https://github.com/apache/flink-shaded
[2] https://issues.apache.org/jira/browse/FLINK-33417
[3] https://lists.apache.org/thread/y1c8545bcsx2836d9pgfdzj65knvw7kb
[4] https://issues.apache.org/jira/browse/FLINK-33190

Best,
Yuxin

Re: [DISCUSS] Release flink-shaded 16.2

[Yuxin Tan]

Hi, Weijie,

Thank you for volunteering to be a release manager.

Currently, no objections are received. If there are no objections, we
will release flink-shaded 16.2 next week.

Best,
Yuxin


weijie guo  于2023年11月8日周三 11:11写道：

> Thanks Yuxin for driving this!
>
> I am willing to help release this. :)
>
> Best regards,
>
> Weijie
>
>
> Yuxin Tan  于2023年11月6日周一 15:45写道：
>
>> Hi, all,
>>
>> I would like to discuss creating a new 16.2 patch release for
>> flink-shaded[1].
>>
>> In our ARM environment, we recently encountered a critical bug
>> (FLINK-33417[2])
>> while using flink-shaded 16.1 in flink 1.17. The issue has been discussed
>> in the
>> discussion[3]. Fortunately, we could resolve this bug by updating the
>> netty version.
>>
>> To address this issue comprehensively, we need to release a new minor
>> version for
>> flink-shaded. As a result, Flink 1.17 should update its dependent
>> flink-shaded version
>> accordingly. However, there is no need for Flink 1.18+ to update the
>> flink-shaded
>> version since these newer versions already depend on flink-shaded 17.0+
>> and have
>> updated their netty version, thereby eliminating the existence of this
>> bug.
>>
>> Considering the low probability of this issue occurring, I do not
>> recommend
>> considering it as a blocker for the Flink 1.17 release. Furthermore, the
>> external connector has been decoupled from flink-shaded in FLINK-33190[4],
>> and the compatibility of the newer netty version 4.1.91 has been verified
>> in the new
>> Flink 1.18+(which is dependent on flink-shaded 17.0 and it is dependent
>> on
>> netty 4.1.91). Hence, we believe this update will not cause any
>> compatibility issues.
>>
>> If there are no objections, we would greatly appreciate it if a committer
>> could
>> volunteer as the release manager. Thank you.
>>
>> [1] https://github.com/apache/flink-shaded
>> [2] https://issues.apache.org/jira/browse/FLINK-33417
>> [3] https://lists.apache.org/thread/y1c8545bcsx2836d9pgfdzj65knvw7kb
>> [4] https://issues.apache.org/jira/browse/FLINK-33190
>>
>> Best,
>> Yuxin
>>
>

Re: [VOTE] FLIP-381: Deprecate configuration getters/setters that return/set complex Java objects

[Yuxin Tan]

+1(non-binding)

Best,
Yuxin


Yuepeng Pan  于2023年11月10日周五 18:31写道：

> +1(non-binding)
>
> Best,
> Roc
>
> On 2023/11/10 03:58:10 Junrui Lee wrote:
> > Hi everyone,
> >
> > Thank you to everyone for the feedback on FLIP-381: Deprecate
> configuration
> > getters/setters that return/set complex Java objects[1] which has been
> > discussed in this thread [2].
> >
> > I would like to start a vote for it. The vote will be open for at least
> 72
> > hours (excluding weekends) unless there is an objection or not enough
> votes.
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278464992
> > [2]https://lists.apache.org/thread/y5owjkfxq3xs9lmpdbl6d6jmqdgbjqxo
> >
>

Re: [VOTE] Release flink-shaded 16.2, release candidate #1

[Yuxin Tan]

Thanks weijie for driving the new release!

+1 (non-binding)

- Built from source code succeeded
- Verified signatures
- Verified hashsums
- Reviewed the web PR

Best,
Yuxin


weijie guo  于2023年11月13日周一 15:57写道：

> Hi everyone,
>
> Please review and vote on the release candidate #1 for the version
> 16.2, as follows:
>
> [ ] +1, Approve the release
>
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
>
>
>
> The complete staging area is available for your review, which includes:
>
> * JIRA release notes [1],
>
> * the official Apache source release to be deployed to dist.apache.org
>  [2], which are signed with the key with fingerprint
> 8D56AE6E7082699A4870750EA4E8C4C05EE6861F [3],
>
> * all artifacts to be deployed to the Maven Central Repository [4],
>
> * source code tag "release-16.2-rc1" [5],
>
> * website pull request listing the new release [6].
>
>
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
>
>
> Thanks,
>
> Release Manager
>
>
>
> [1] https://issues.apache.org/jira/projects/FLINK/versions/12353810
>
> [2] https://dist.apache.org/repos/dist/dev/flink/flink-shaded-16.2-rc1
>
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>
> [4]
> https://repository.apache.org/content/repositories/orgapacheflink-1671/
>
> [5] https://github.com/apache/flink-shaded/releases/tag/release-16.2-rc1
>
> [6] https://github.com/apache/flink-web/pull/697
>