Re: [ANNOUNCE] New Apache Flink PMC Member - Weijie Guo

[Guowei Ma]

Congratulations!

Best,
Guowei


On Tue, Jun 4, 2024 at 4:55 PM gongzhongqiang 
wrote:

> Congratulations Weijie! Best,
> Zhongqiang Gong
>
> Xintong Song  于2024年6月4日周二 14:46写道：
>
> > Hi everyone,
> >
> > On behalf of the PMC, I'm very happy to announce that Weijie Guo has
> joined
> > the Flink PMC!
> >
> > Weijie has been an active member of the Apache Flink community for many
> > years. He has made significant contributions in many components,
> including
> > runtime, shuffle, sdk, connectors, etc. He has driven / participated in
> > many FLIPs, authored and reviewed hundreds of PRs, been consistently
> active
> > on mailing lists, and also helped with release management of 1.20 and
> > several other bugfix releases.
> >
> > Congratulations and welcome Weijie!
> >
> > Best,
> >
> > Xintong (on behalf of the Flink PMC)
> >
>

Re: Re: [ANNOUNCE] Apache Paimon is graduated to Top Level Project

[Guowei Ma]

Congratulations!
Best,
Guowei


On Mon, Apr 1, 2024 at 11:15 AM Feng Jin  wrote:

> Congratulations!
>
> Best,
> Feng Jin
>
> On Mon, Apr 1, 2024 at 10:51 AM weijie guo 
> wrote:
>
>> Congratulations!
>>
>> Best regards,
>>
>> Weijie
>>
>>
>> Hang Ruan  于2024年4月1日周一 09:49写道：
>>
>> > Congratulations!
>> >
>> > Best,
>> > Hang
>> >
>> > Lincoln Lee  于2024年3月31日周日 00:10写道：
>> >
>> > > Congratulations!
>> > >
>> > > Best,
>> > > Lincoln Lee
>> > >
>> > >
>> > > Jark Wu  于2024年3月30日周六 22:13写道：
>> > >
>> > > > Congratulations!
>> > > >
>> > > > Best,
>> > > > Jark
>> > > >
>> > > > On Fri, 29 Mar 2024 at 12:08, Yun Tang  wrote:
>> > > >
>> > > > > Congratulations to all Paimon guys!
>> > > > >
>> > > > > Glad to see a Flink sub-project has been graduated to an Apache
>> > > top-level
>> > > > > project.
>> > > > >
>> > > > > Best
>> > > > > Yun Tang
>> > > > >
>> > > > > 
>> > > > > From: Hangxiang Yu 
>> > > > > Sent: Friday, March 29, 2024 10:32
>> > > > > To: dev@flink.apache.org 
>> > > > > Subject: Re: Re: [ANNOUNCE] Apache Paimon is graduated to Top
>> Level
>> > > > Project
>> > > > >
>> > > > > Congratulations!
>> > > > >
>> > > > > On Fri, Mar 29, 2024 at 10:27 AM Benchao Li > >
>> > > > wrote:
>> > > > >
>> > > > > > Congratulations!
>> > > > > >
>> > > > > > Zakelly Lan  于2024年3月29日周五 10:25写道：
>> > > > > > >
>> > > > > > > Congratulations!
>> > > > > > >
>> > > > > > >
>> > > > > > > Best,
>> > > > > > > Zakelly
>> > > > > > >
>> > > > > > > On Thu, Mar 28, 2024 at 10:13 PM Jing Ge
>> > > > > > > >
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > > Congrats!
>> > > > > > > >
>> > > > > > > > Best regards,
>> > > > > > > > Jing
>> > > > > > > >
>> > > > > > > > On Thu, Mar 28, 2024 at 1:27 PM Feifan Wang <
>> > zoltar9...@163.com>
>> > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Congratulations!——
>> > > > > > > > >
>> > > > > > > > > Best regards,
>> > > > > > > > >
>> > > > > > > > > Feifan Wang
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > At 2024-03-28 20:02:43, "Yanfei Lei" > >
>> > > > wrote:
>> > > > > > > > > >Congratulations!
>> > > > > > > > > >
>> > > > > > > > > >Best,
>> > > > > > > > > >Yanfei
>> > > > > > > > > >
>> > > > > > > > > >Zhanghao Chen  于2024年3月28日周四
>> > > > 19:59写道：
>> > > > > > > > > >>
>> > > > > > > > > >> Congratulations!
>> > > > > > > > > >>
>> > > > > > > > > >> Best,
>> > > > > > > > > >> Zhanghao Chen
>> > > > > > > > > >> 
>> > > > > > > > > >> From: Yu Li 
>> > > > > > > > > >> Sent: Thursday, March 28, 2024 15:55
>> > > > > > > > > >> To: d...@paimon.apache.org 
>> > > > > > > > > >> Cc: dev ; user <
>> > u...@flink.apache.org
>> > > >
>> > > > > > > > > >> Subject: Re: [ANNOUNCE] Apache Paimon is graduated to
>> Top
>> > > > Level
>> > > > > > > > Project
>> > > > > > > > > >>
>> > > > > > > > > >> CC the Flink user and dev mailing list.
>> > > > > > > > > >>
>> > > > > > > > > >> Paimon originated within the Flink community, initially
>> > > known
>> > > > as
>> > > > > > Flink
>> > > > > > > > > >> Table Store, and all our incubating mentors are
>> members of
>> > > the
>> > > > > > Flink
>> > > > > > > > > >> Project Management Committee. I am confident that the
>> > bonds
>> > > of
>> > > > > > > > > >> enduring friendship and close collaboration will
>> continue
>> > to
>> > > > > > unite the
>> > > > > > > > > >> two communities.
>> > > > > > > > > >>
>> > > > > > > > > >> And congratulations all!
>> > > > > > > > > >>
>> > > > > > > > > >> Best Regards,
>> > > > > > > > > >> Yu
>> > > > > > > > > >>
>> > > > > > > > > >> On Wed, 27 Mar 2024 at 20:35, Guojun Li <
>> > > > > gjli.schna...@gmail.com>
>> > > > > > > > > wrote:
>> > > > > > > > > >> >
>> > > > > > > > > >> > Congratulations!
>> > > > > > > > > >> >
>> > > > > > > > > >> > Best,
>> > > > > > > > > >> > Guojun
>> > > > > > > > > >> >
>> > > > > > > > > >> > On Wed, Mar 27, 2024 at 5:24 PM wulin <
>> > > ouyangwu...@163.com>
>> > > > > > wrote:
>> > > > > > > > > >> >
>> > > > > > > > > >> > > Congratulations~
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > > 2024年3月27日 15:54，王刚 > > .INVALID>
>> > > > 写道：
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > Congratulations~
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > >> 2024年3月26日 10:25，Jingsong Li <
>> > jingsongl...@gmail.com
>> > > >
>> > > > > 写道：
>> > > > > > > > > >> > > >>
>> > > > > > > > > >> > > >> Hi Paimon community,
>> > > > > > > > > >> > > >>
>> > > > > > > > > >> > > >> I’m glad to announce that the ASF board has
>> > approved
>> > > a
>> > > > > > > > > resolution to
>> > > > > > > > > >> > > >> graduate Paimon into a full Top Level Project.
>> > Thanks
>> > > > to
>> > > > > > > > > everyone for
>> > > > > > > > > >> > > >> your help to get to this point.
>> > > > > > > > > >> > > >>
>> > > > > > > > > >> > > >> I just created an issue to track the things we
>> need
>> > > to
>> > 

Re: [VOTE] FLIP-409: DataStream V2 Building Blocks: DataStream, Partitioning and ProcessFunction

[Guowei Ma]

+1 (binding)
Best,
Guowei


On Tue, Feb 27, 2024 at 10:08 AM Rui Fan <1996fan...@gmail.com> wrote:

> +1(binding)
>
> Best,
> Rui
>
> On Tue, Feb 27, 2024 at 9:44 AM weijie guo 
> wrote:
>
> > +1(binding)
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > Xintong Song  于2024年2月27日周二 09:38写道：
> >
> > > +1 (binding)
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > >
> > > On Mon, Feb 26, 2024 at 6:09 PM weijie guo 
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > >
> > > > Thanks for all the feedback about the FLIP-409: DataStream V2
> Building
> > > > Blocks: DataStream, Partitioning and ProcessFunction [1]. The
> > > > discussion thread is here [2].
> > > >
> > > >
> > > > The vote will be open for at least 72 hours unless there is an
> > > > objection or insufficient votes.
> > > >
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-409%3A+DataStream+V2+Building+Blocks%3A+DataStream%2C+Partitioning+and+ProcessFunction
> > > >
> > > > [2] https://lists.apache.org/thread/cwds0bwbgy3lfdgnlqbfhm6lfvx2qbrv
> > > >
> > > >
> > > > Best regards,
> > > >
> > > > Weijie
> > > >
> > >
> >
>

Re: [VOTE] FLIP-410: Config, Context and Processing Timer Service of DataStream API V2

[Guowei Ma]

+1 (binding)
Best,
Guowei


On Tue, Feb 27, 2024 at 10:09 AM Xuannan Su  wrote:

> +1 (non-binding)
>
> Best,
> Xuannan
>
> On Tue, Feb 27, 2024 at 9:37 AM Xintong Song 
> wrote:
> >
> > +1 (binding)
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Mon, Feb 26, 2024 at 6:10 PM weijie guo 
> > wrote:
> >
> > > Hi everyone,
> > >
> > >
> > > Thanks for all the feedback about the FLIP-410: Config, Context and
> > > Processing Timer Service of DataStream API V2 [1]. The discussion
> > > thread is here [2].
> > >
> > >
> > > The vote will be open for at least 72 hours unless there is an
> > > objection or insufficient votes.
> > >
> > >
> > > [1]
> > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-410%3A++Config%2C+Context+and+Processing+Timer+Service+of+DataStream+API+V2
> > >
> > > [2] https://lists.apache.org/thread/70gf028c5gsdb9qhsgpht0chzyp9nogc
> > >
> > >
> > > Best regards,
> > >
> > > Weijie
> > >
>

Re: [VOTE] FLIP-408: [Umbrella] Introduce DataStream API V2

[Guowei Ma]

+1(binding)
Best,
Guowei


On Tue, Feb 27, 2024 at 10:06 AM Rui Fan <1996fan...@gmail.com> wrote:

> +1(binding)
>
> Best,
> Rui
>
> On Tue, Feb 27, 2024 at 9:43 AM weijie guo 
> wrote:
>
> > +1(binding)
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > Xintong Song  于2024年2月27日周二 09:36写道：
> >
> > > +1 (binding)
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > >
> > > On Mon, Feb 26, 2024 at 6:08 PM weijie guo 
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > >
> > > > Thanks for all the feedback about the FLIP-408: [Umbrella] Introduce
> > > > DataStream API V2 [1]. The discussion thread is here [2].
> > > >
> > > >
> > > > The vote will be open for at least 72 hours unless there is an
> > > > objection or insufficient votes.
> > > >
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-408%3A+%5BUmbrella%5D+Introduce+DataStream+API+V2
> > > >
> > > > [2] https://lists.apache.org/thread/w8olky9s7fo5h8fl3nj3qbym307zk2l0
> > > >
> > > > Best regards,
> > > >
> > > > Weijie
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-408: [Umbrella] Introduce DataStream API V2

[Guowei Ma]

Hi,

Thanks for your reply!
I have no other comments!

Best,
Guowei


On Mon, Feb 26, 2024 at 3:43 PM weijie guo 
wrote:

> Hi Guowei,
>
> thanks for your reply!
>
> > Do connectors and SQL currently have similar problems?
>
> - Connectors:
> The APIs for FLIP-27 based source and Sink-V2 are currently in flink-core,
> and we will gradually move them to flink-core-api. Anyway, connector should
> be free of this problem.
>
> - SQL/Table:
>
> For pure SQL job, the user does not need to have any dependencies when
> writing the SQL query string.
>
> But for the Table API, it relies on the original DataStream(i.e. DataStream
> V1), so the implementation is not decoupled from the API.
>
> In the future, we will consider building the Table API on top of DataStream
> API V2, which solves this problem also.
>
>
> Best regards,
>
> Weijie
>
>
> Guowei Ma  于2024年2月26日周一 14:37写道：
>
> > Hi,weijie
> >
> > Thank you very much to Weijie for proposing this series of improvements,
> > especially the complete decoupling of user interface and implementation.
> > This part is actually a very serious problem that disturbs downstream
> users
> > in the community. I hope this problem can be completely solved in the
> > future.
> >
> > However, regarding the API decoupling part, I have a question: Do
> > connectors and SQL currently have similar problems? If so, will similar
> > methods be used to solve them?
> >
> > Best,
> > Guowei
> >
> >
> > On Tue, Feb 20, 2024 at 3:10 PM weijie guo 
> > wrote:
> >
> > > Hi All,
> > >
> > > Thanks for all the feedback.
> > >
> > > If there are no more comments, I would like to start the vote thread,
> > > thanks again!
> > >
> > > Best regards,
> > >
> > > Weijie
> > >
> > >
> > > Xintong Song  于2024年1月30日周二 11:04写道：
> > >
> > > > Thanks for working on this, Weijie.
> > > >
> > > > The design flaws of the current DataStream API (i.e., V1) have been a
> > > pain
> > > > for a long time. It's great to see efforts going on trying to resolve
> > > them.
> > > >
> > > > Significant changes to such an important and comprehensive set of
> > public
> > > > APIs deserves caution. From that perspective, the ideas of
> introducing
> > a
> > > > new set of APIs that gradually replace the current one, splitting the
> > > > introducing of the new APIs into many separate FLIPs, and making
> > > > intermediate APIs @Experiemental until all of them are completed make
> > > > great sense to me.
> > > >
> > > > Besides, the ideas of generalized watermark, execution hints sound
> > quite
> > > > interesting. Looking forward to more detailed discussions in the
> > > > corresponding sub-FLIPs.
> > > >
> > > > +1 for the roadmap.
> > > >
> > > > Best,
> > > >
> > > > Xintong
> > > >
> > > >
> > > >
> > > > On Tue, Jan 30, 2024 at 11:00 AM weijie guo <
> guoweijieres...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Wencong:
> > > > >
> > > > > > The Processing TimerService is currently
> > > > > defined as one of the basic primitives, partly because it's
> > understood
> > > > that
> > > > > you have to choose between processing time and event time.
> > > > > The other part of the reason is that it needs to work based on the
> > > task's
> > > > > mailbox thread model to avoid concurrency issues. Could you clarify
> > the
> > > > > second
> > > > > part of the reason?
> > > > >
> > > > > Since the processing logic of the operators takes place in the
> > mailbox
> > > > > thread, the processing timer's callback function must also be
> > executed
> > > in
> > > > > the mailbox to ensure thread safety.
> > > > > If we do not define the Processing TimerService as primitive, there
> > is
> > > no
> > > > > way for the user to dispatch custom logic to the mailbox thread.
> > > > >
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Weijie
> > > > >
> > > > >
> > > > > Xuannan Su  于2

Re: [DISCUSS] FLIP-408: [Umbrella] Introduce DataStream API V2

[Guowei Ma]

Hi,weijie

Thank you very much to Weijie for proposing this series of improvements,
especially the complete decoupling of user interface and implementation.
This part is actually a very serious problem that disturbs downstream users
in the community. I hope this problem can be completely solved in the
future.

However, regarding the API decoupling part, I have a question: Do
connectors and SQL currently have similar problems? If so, will similar
methods be used to solve them?

Best,
Guowei


On Tue, Feb 20, 2024 at 3:10 PM weijie guo 
wrote:

> Hi All,
>
> Thanks for all the feedback.
>
> If there are no more comments, I would like to start the vote thread,
> thanks again!
>
> Best regards,
>
> Weijie
>
>
> Xintong Song  于2024年1月30日周二 11:04写道：
>
> > Thanks for working on this, Weijie.
> >
> > The design flaws of the current DataStream API (i.e., V1) have been a
> pain
> > for a long time. It's great to see efforts going on trying to resolve
> them.
> >
> > Significant changes to such an important and comprehensive set of public
> > APIs deserves caution. From that perspective, the ideas of introducing a
> > new set of APIs that gradually replace the current one, splitting the
> > introducing of the new APIs into many separate FLIPs, and making
> > intermediate APIs @Experiemental until all of them are completed make
> > great sense to me.
> >
> > Besides, the ideas of generalized watermark, execution hints sound quite
> > interesting. Looking forward to more detailed discussions in the
> > corresponding sub-FLIPs.
> >
> > +1 for the roadmap.
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Tue, Jan 30, 2024 at 11:00 AM weijie guo 
> > wrote:
> >
> > > Hi Wencong:
> > >
> > > > The Processing TimerService is currently
> > > defined as one of the basic primitives, partly because it's understood
> > that
> > > you have to choose between processing time and event time.
> > > The other part of the reason is that it needs to work based on the
> task's
> > > mailbox thread model to avoid concurrency issues. Could you clarify the
> > > second
> > > part of the reason?
> > >
> > > Since the processing logic of the operators takes place in the mailbox
> > > thread, the processing timer's callback function must also be executed
> in
> > > the mailbox to ensure thread safety.
> > > If we do not define the Processing TimerService as primitive, there is
> no
> > > way for the user to dispatch custom logic to the mailbox thread.
> > >
> > >
> > > Best regards,
> > >
> > > Weijie
> > >
> > >
> > > Xuannan Su  于2024年1月29日周一 17:12写道：
> > >
> > > > Hi Weijie,
> > > >
> > > > Thanks for driving the work! There are indeed many pain points in the
> > > > current DataStream API, which are challenging to resolve with its
> > > > existing design. It is a great opportunity to propose a new
> DataStream
> > > > API that tackles these issues. I like the way we've divided the FLIP
> > > > into multiple sub-FLIPs; the roadmap is clear and comprehensible. +1
> > > > for the umbrella FLIP. I am eager to see the sub-FLIPs!
> > > >
> > > > Best regards,
> > > > Xuannan
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Jan 24, 2024 at 8:55 PM Wencong Liu 
> > > wrote:
> > > > >
> > > > > Hi Weijie,
> > > > >
> > > > >
> > > > > Thank you for the effort you've put into the DataStream API ! By
> > > > reorganizing and
> > > > > redesigning the DataStream API, as well as addressing some of the
> > > > unreasonable
> > > > > designs within it, we can enhance the efficiency of job development
> > for
> > > > developers.
> > > > > It also allows developers to design more flexible Flink jobs to
> meet
> > > > business requirements.
> > > > >
> > > > >
> > > > > I have conducted a comprehensive review of the DataStream API
> design
> > in
> > > > versions
> > > > > 1.18 and 1.19. I found quite a few functional defects in the
> > DataStream
> > > > API, such as the
> > > > > lack of corresponding APIs in batch processing scenarios. In the
> > > > upcoming 1.20 version,
> > > > > I will further improve the DataStream API in batch computing
> > scenarios.
> > > > >
> > > > >
> > > > > The issues existing in the old DataStream API (which can be
> referred
> > to
> > > > as V1) can be
> > > > > addressed from a design perspective in the initial version of V2. I
> > > hope
> > > > to also have the
> > > > >  opportunity to participate in the development of DataStream V2 and
> > > make
> > > > my contribution.
> > > > >
> > > > >
> > > > > Regarding FLIP-408, I have a question: The Processing TimerService
> is
> > > > currently
> > > > > defined as one of the basic primitives, partly because it's
> > understood
> > > > that
> > > > > you have to choose between processing time and event time.
> > > > > The other part of the reason is that it needs to work based on the
> > > task's
> > > > > mailbox thread model to avoid concurrency issues. Could you clarify
> > the
> > > > second
> > > > > part of the reason?
> > > > >
> > > > > Best,
> > > > > Wencong Liu

Re: [VOTE] Accept Flink CDC into Apache Flink

[Guowei Ma]

+1 (binding)
Best,
Guowei


On Tue, Jan 9, 2024 at 4:49 PM Rui Fan <1996fan...@gmail.com> wrote:

> +1 (non-binding)
>
> Best,
> Rui
>
> On Tue, Jan 9, 2024 at 4:41 PM Hang Ruan  wrote:
>
> > +1 (non-binding)
> >
> > Best,
> > Hang
> >
> > gongzhongqiang  于2024年1月9日周二 16:25写道：
> >
> > > +1 non-binding
> > >
> > > Best,
> > > Zhongqiang
> > >
> > > Leonard Xu  于2024年1月9日周二 15:05写道：
> > >
> > > > Hello all,
> > > >
> > > > This is the official vote whether to accept the Flink CDC code
> > > contribution
> > > >  to Apache Flink.
> > > >
> > > > The current Flink CDC code, documentation, and website can be
> > > > found here:
> > > > code: https://github.com/ververica/flink-cdc-connectors <
> > > > https://github.com/ververica/flink-cdc-connectors>
> > > > docs: https://ververica.github.io/flink-cdc-connectors/ <
> > > > https://ververica.github.io/flink-cdc-connectors/>
> > > >
> > > > This vote should capture whether the Apache Flink community is
> > interested
> > > > in accepting, maintaining, and evolving Flink CDC.
> > > >
> > > > Regarding my original proposal[1] in the dev mailing list, I firmly
> > > believe
> > > > that this initiative aligns perfectly with Flink. For the Flink
> > > community,
> > > > it represents an opportunity to bolster Flink's competitive edge in
> > > > streaming
> > > > data integration, fostering the robust growth and prosperity of the
> > > Apache
> > > > Flink
> > > > ecosystem. For the Flink CDC project, becoming a sub-project of
> Apache
> > > > Flink
> > > > means becoming an integral part of a neutral open-source community,
> > > > capable of
> > > > attracting a more diverse pool of contributors.
> > > >
> > > > All Flink CDC maintainers are dedicated to continuously contributing
> to
> > > > achieve
> > > > seamless integration with Flink. Additionally, PMC members like Jark,
> > > > Qingsheng,
> > > > and I are willing to infacilitate the expansion of contributors and
> > > > committers to
> > > > effectively maintain this new sub-project.
> > > >
> > > > This is a "Adoption of a new Codebase" vote as per the Flink bylaws
> > [2].
> > > > Only PMC votes are binding. The vote will be open at least 7 days
> > > > (excluding weekends), meaning until Thursday January 18 12:00 UTC, or
> > > > until we
> > > > achieve the 2/3rd majority. We will follow the instructions in the
> > Flink
> > > > Bylaws
> > > > in the case of insufficient active binding voters:
> > > >
> > > > > 1. Wait until the minimum length of the voting passes.
> > > > > 2. Publicly reach out via personal email to the remaining binding
> > > voters
> > > > in the
> > > > voting mail thread for at least 2 attempts with at least 7 days
> between
> > > > two attempts.
> > > > > 3. If the binding voter being contacted still failed to respond
> after
> > > > all the attempts,
> > > > the binding voter will be considered as inactive for the purpose of
> > this
> > > > particular voting.
> > > >
> > > > Welcome voting !
> > > >
> > > > Best,
> > > > Leonard
> > > > [1] https://lists.apache.org/thread/o7klnbsotmmql999bnwmdgo56b6kxx9l
> > > > [2]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120731026
> > >
> >
>

Re: [DISCUSS] FLIP-383: Support Job Recovery for Batch Jobs

[Guowei Ma]

Hi,


This is a very good proposal, as far as I know, it can solve some very
critical production operations in certain scenarios. I have two minor
issues:

As far as I know, there are multiple job managers on standby in some
scenarios. In this case, is your design still effective? I'm unsure if you
have conducted any tests. For instance, standby job managers might take
over these failed jobs more quickly.
Regarding the part about the operator coordinator, how can you ensure that
the checkpoint mechanism can restore the state of the operator coordinator:
For example:
How do you rule out that there might still be some states in the memory of
the original operator coordinator? After all, the implementation was done
under the assumption of scenarios where the job manager doesn't fail.
Additionally, using NO_CHECKPOINT seems a bit odd. Why not use a normal
checkpoint ID greater than 0 and record it in the event store?
If the issues raised in point 2 cannot be resolved in the short term, would
it be possible to consider not supporting failover with a source job
manager?

Best,
Guowei


On Thu, Nov 2, 2023 at 6:01 PM Lijie Wang  wrote:

> Hi devs,
>
> Zhu Zhu and I would like to start a discussion about FLIP-383: Support Job
> Recovery for Batch Jobs[1]
>
> Currently, when Flink’s job manager crashes or gets killed, possibly due to
> unexpected errors or planned nodes decommission, it will cause the
> following two situations:
> 1. Failed, if the job does not enable HA.
> 2. Restart, if the job enable HA. If it’s a streaming job, the job will be
> resumed from the last successful checkpoint. If it’s a batch job, it has to
> run from beginning, all previous progress will be lost.
>
> In view of this, we think the JM crash may cause great regression for batch
> jobs, especially long running batch jobs. This FLIP is mainly to solve this
> problem so that batch jobs can recover most job progress after JM crashes.
> In this FLIP, our goal is to let most finished tasks not need to be re-run.
>
> You can find more details in the FLIP-383[1]. Looking forward to your
> feedback.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-383%3A+Support+Job+Recovery+for+Batch+Jobs
>
> Best,
> Lijie
>

Re: [VOTE] FLIP-309: Support using larger checkpointing interval when source is processing backlog

[Guowei Ma]

+1（binding）
Best,
Guowei


On Wed, Jul 19, 2023 at 11:18 AM Hang Ruan  wrote:

> +1 (non-binding)
>
> Thanks for driving.
>
> Best,
> Hang
>
> Leonard Xu  于2023年7月19日周三 10:42写道：
>
> > Thanks Dong for the continuous work.
> >
> > +1(binding)
> >
> > Best,
> > Leonard
> >
> > > On Jul 18, 2023, at 10:16 PM, Jingsong Li 
> > wrote:
> > >
> > > +1 binding
> > >
> > > Thanks Dong for continuous driving.
> > >
> > > Best,
> > > Jingsong
> > >
> > > On Tue, Jul 18, 2023 at 10:04 PM Jark Wu  wrote:
> > >>
> > >> +1 (binding)
> > >>
> > >> Best,
> > >> Jark
> > >>
> > >> On Tue, 18 Jul 2023 at 20:30, Piotr Nowojski 
> > wrote:
> > >>
> > >>> +1 (binding)
> > >>>
> > >>> Piotrek
> > >>>
> > >>> wt., 18 lip 2023 o 08:51 Jing Ge 
> > napisał(a):
> > >>>
> >  +1(binding)
> > 
> >  Best regards,
> >  Jing
> > 
> >  On Tue, Jul 18, 2023 at 8:31 AM Rui Fan <1996fan...@gmail.com>
> wrote:
> > 
> > > +1(binding)
> > >
> > > Best,
> > > Rui Fan
> > >
> > >
> > > On Tue, Jul 18, 2023 at 12:04 PM Dong Lin 
> > wrote:
> > >
> > >> Hi all,
> > >>
> > >> We would like to start the vote for FLIP-309: Support using larger
> > >> checkpointing interval when source is processing backlog [1]. This
> > >>> FLIP
> > > was
> > >> discussed in this thread [2].
> > >>
> > >> The vote will be open until at least July 21st (at least 72
> hours),
> > >> following
> > >> the consensus voting process.
> > >>
> > >> Cheers,
> > >> Yunfeng and Dong
> > >>
> > >> [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-309
> > >>
> > >>
> > >
> > 
> > >>>
> >
> %3A+Support+using+larger+checkpointing+interval+when+source+is+processing+backlog
> > >> [2]
> > https://lists.apache.org/thread/l1l7f30h7zldjp6ow97y70dcthx7tl37
> > >>
> > >
> > 
> > >>>
> >
> >
>

Re: [VOTE] Apache Flink ML Release 2.2.0, release candidate #2

[Guowei Ma]

Hi Dong,

Thanks for driving this release!

+1 (binding)

* Checked JIRA release notes
* Verified signature and checksum for the source
* Download the source code and build the code with JDK8
* Browsed through README.md files.

Best,
Guowei


On Fri, Apr 14, 2023 at 2:48 PM Zhipeng Zhang 
wrote:

> Hi Dong,
> Thanks for driving this release!
>
> +1 (non-binding)
>
> Here is what I have checked.
> - Verified that the checksums and GPG files.
> - Verified that the source distributions do not contain any binaries.
> - Built the source distribution and run all unit tests.
> - Verified that all POM files point to the same version.
> - Browsed through JIRA release notes files.
> - Browsed through README.md files.
> - Verified the source code tag.
>
>
> Dong Lin  于2023年4月13日周四 18:28写道：
>
> >
> > Hi everyone,
> >
> > Please review and vote on the release candidate #2 for version 2.2.0 of
> > Apache Flink ML as follows:
> >
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > **Testing Guideline**
> >
> > You can find here [1] a page in the project wiki on instructions for
> > testing.
> >
> > To cast a vote, it is not necessary to perform all listed checks, but
> > please mention which checks you have performed when voting.
> >
> > **Release Overview**
> >
> > As an overview, the release consists of the following:
> > a) Flink ML source release to be deployed to dist.apache.org
> > b) Flink ML Python source distributions to be deployed to PyPI
> > c) Maven artifacts to be deployed to the Maven Central Repository
> >
> > **Staging Areas to Review**
> >
> > The staging areas containing the above-mentioned artifacts are as
> follows, for
> > your review:
> >
> > - All artifacts for a) and b) can be found in the corresponding dev
> repository
> > at dist.apache.org [2], which are signed with the key with fingerprint
> AFAC
> > DB09 E6F0 FF28 C93D 64BC BEED 4F6C B9F7 7D0E [3]
> > - All artifacts for c) can be found at the Apache Nexus Repository [4]
> >
> > **Other links for your review**
> >
> > - JIRA release notes [5]
> > - Source code tag "release-2.2.0-rc2" [6]
> > - PR to update the website Downloads page to include Flink ML links [7]
> >
> > **Vote Duration**
> >
> > The voting time will run for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> >
> > Cheers,
> > Dong
> >
> >
> > [1]
> https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+ML+
> > Release
> > [2] https://dist.apache.org/repos/dist/dev/flink/flink-ml-2.2.0-rc2/
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [4]
> https://repository.apache.org/content/repositories/orgapacheflink-1605/
> > [5]
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351884
> > [6] https://github.com/apache/flink-ml/releases/tag/release-2.2.0-rc2
> > [7] https://github.com/apache/flink-web/pull/630
>
>
>
> --
> best,
> Zhipeng
>

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

[Guowei Ma]

Congratulations!

Best,
Guowei


On Tue, Mar 28, 2023 at 12:02 PM Yuxin Tan  wrote:

> Congratulations！
>
> Best,
> Yuxin
>
>
> Guanghui Zhang  于2023年3月28日周二 11:06写道：
>
>> Congratulations！
>>
>> Best,
>> Zhang Guanghui
>>
>> Hang Ruan  于2023年3月28日周二 10:29写道：
>>
>> > Congratulations!
>> >
>> > Best,
>> > Hang
>> >
>> > yu zelin  于2023年3月28日周二 10:27写道：
>> >
>> >> Congratulations!
>> >>
>> >> Best,
>> >> Yu Zelin
>> >>
>> >> 2023年3月27日 17:23，Yu Li  写道：
>> >>
>> >> Dear Flinkers,
>> >>
>> >>
>> >>
>> >> As you may have noticed, we are pleased to announce that Flink Table
>> Store has joined the Apache Incubator as a separate project called Apache
>> Paimon(incubating) [1] [2] [3]. The new project still aims at building a
>> streaming data lake platform for high-speed data ingestion, change data
>> tracking and efficient real-time analytics, with the vision of supporting a
>> larger ecosystem and establishing a vibrant and neutral open source
>> community.
>> >>
>> >>
>> >>
>> >> We would like to thank everyone for their great support and efforts
>> for the Flink Table Store project, and warmly welcome everyone to join the
>> development and activities of the new project. Apache Flink will continue
>> to be one of the first-class citizens supported by Paimon, and we believe
>> that the Flink and Paimon communities will maintain close cooperation.
>> >>
>> >>
>> >> 亲爱的Flinkers，
>> >>
>> >>
>> >> 正如您可能已经注意到的，我们很高兴地宣布，Flink Table Store 已经正式加入 Apache
>> >> 孵化器独立孵化 [1] [2] [3]。新项目的名字是
>> >> Apache
>> Paimon(incubating)，仍致力于打造一个支持高速数据摄入、流式数据订阅和高效实时分析的新一代流式湖仓平台。此外，新项目将支持更加丰富的生态，并建立一个充满活力和中立的开源社区。
>> >>
>> >>
>> >> 在这里我们要感谢大家对 Flink Table Store
>> >> 项目的大力支持和投入，并热烈欢迎大家加入新项目的开发和社区活动。Apache Flink 将继续作为 Paimon
>> 支持的主力计算引擎之一，我们也相信
>> >> Flink 和 Paimon 社区将继续保持密切合作。
>> >>
>> >>
>> >> Best Regards,
>> >> Yu (on behalf of the Apache Flink PMC and Apache Paimon PPMC)
>> >>
>> >> 致礼，
>> >> 李钰（谨代表 Apache Flink PMC 和 Apache Paimon PPMC）
>> >>
>> >> [1] https://paimon.apache.org/
>> >> [2] https://github.com/apache/incubator-paimon
>> >> [3]
>> https://cwiki.apache.org/confluence/display/INCUBATOR/PaimonProposal
>> >>
>> >>
>> >>
>>
>

Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo

[Guowei Ma]

congratulations Yuxia
Best,
Guowei


On Mon, Mar 13, 2023 at 10:43 AM Junrui Lee  wrote:

> Congratulations, Yuxia!
>
> Best,
> Junrui
>
> Yanfei Lei  于2023年3月13日周一 10:42写道：
>
> > Congratulations, Yuxia!
> >
> > Best,
> > Yanfei
> >
> >
> > Samrat Deb  于2023年3月13日周一 10:41写道：
> > >
> > > congratulations Yuxia
> > >
> > > Bests,
> > > Samrat
> > >
> > > On Mon, 13 Mar 2023 at 8:06 AM, Yuxin Tan 
> > wrote:
> > >
> > > > Congratulations, Yuxia!
> > > >
> > > > Best,
> > > > Yuxin
> > > >
> > > >
> > > > Jark Wu  于2023年3月13日周一 10:26写道：
> > > >
> > > > > Hi, everyone
> > > > >
> > > > > On behalf of the PMC, I'm very happy to announce Yuxia Luo as a new
> > Flink
> > > > > Committer.
> > > > >
> > > > > Yuxia has been continuously contributing to the Flink project for
> > almost
> > > > > two
> > > > > years, authored and reviewed hundreds of PRs over this time. He is
> > > > > currently
> > > > > the core maintainer of the Hive component, where he contributed
> many
> > > > > valuable
> > > > > features, including the Hive dialect with 95% compatibility and
> small
> > > > file
> > > > > compaction.
> > > > > In addition, Yuxia driven FLIP-282 (DELETE & UPDATE API) to better
> > > > > integrate
> > > > > Flink with data lakes. He actively participated in dev discussions
> > and
> > > > > answered
> > > > > many questions on the user mailing list.
> > > > >
> > > > > Please join me in congratulating Yuxia Luo for becoming a Flink
> > > > Committer!
> > > > >
> > > > > Best,
> > > > > Jark Wu (on behalf of the Flink PMC)
> > > > >
> > > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Anton Kalashnikov

[Guowei Ma]

Congratulations, Anton!

Best,
Guowei


On Tue, Feb 21, 2023 at 1:52 PM Shammon FY  wrote:

> Congratulations, Anton!
>
> Best,
> Shammon
>
> On Tue, Feb 21, 2023 at 1:41 PM Sergey Nuyanzin 
> wrote:
>
> > Congratulations, Anton!
> >
> > On Tue, Feb 21, 2023 at 4:53 AM Weihua Hu 
> wrote:
> >
> > > Congratulations, Anton!
> > >
> > > Best,
> > > Weihua
> > >
> > >
> > > On Tue, Feb 21, 2023 at 11:22 AM weijie guo  >
> > > wrote:
> > >
> > > > Congratulations, Anton!
> > > >
> > > > Best regards,
> > > >
> > > > Weijie
> > > >
> > > >
> > > > Leonard Xu  于2023年2月21日周二 11:02写道：
> > > >
> > > > > Congratulations, Anton!
> > > > >
> > > > > Best,
> > > > > Leonard
> > > > >
> > > > >
> > > > > > On Feb 21, 2023, at 10:02 AM, Rui Fan  wrote:
> > > > > >
> > > > > > Congratulations, Anton!
> > > > > >
> > > > > > Best,
> > > > > > Rui Fan
> > > > > >
> > > > > > On Tue, Feb 21, 2023 at 9:23 AM yuxia <
> luoyu...@alumni.sjtu.edu.cn
> > >
> > > > > wrote:
> > > > > >
> > > > > >> Congrats Anton!
> > > > > >>
> > > > > >> Best regards,
> > > > > >> Yuxia
> > > > > >>
> > > > > >> - 原始邮件 -
> > > > > >> 发件人: "Matthias Pohl" 
> > > > > >> 收件人: "dev" 
> > > > > >> 发送时间: 星期二, 2023年 2 月 21日 上午 12:52:40
> > > > > >> 主题: Re: [ANNOUNCE] New Apache Flink Committer - Anton
> Kalashnikov
> > > > > >>
> > > > > >> Congratulations, Anton! :-)
> > > > > >>
> > > > > >> On Mon, Feb 20, 2023 at 5:09 PM Jing Ge
> >  > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Congrats Anton!
> > > > > >>>
> > > > > >>> On Mon, Feb 20, 2023 at 5:02 PM Samrat Deb <
> > decordea...@gmail.com>
> > > > > >> wrote:
> > > > > >>>
> > > > >  congratulations Anton!
> > > > > 
> > > > >  Bests,
> > > > >  Samrat
> > > > > 
> > > > >  On Mon, 20 Feb 2023 at 9:29 PM, John Roesler <
> > vvcep...@apache.org
> > > >
> > > > > >>> wrote:
> > > > > 
> > > > > > Congratulations, Anton!
> > > > > > -John
> > > > > >
> > > > > > On Mon, Feb 20, 2023, at 08:18, Piotr Nowojski wrote:
> > > > > >> Hi, everyone
> > > > > >>
> > > > > >> On behalf of the PMC, I'm very happy to announce Anton
> > > Kalashnikov
> > > > > >>> as a
> > > > > > new
> > > > > >> Flink Committer.
> > > > > >>
> > > > > >> Anton has been very active for almost two years already,
> > > authored
> > > > > >> and
> > > > > >> reviewed many PRs over this time. He is active in the
> Flink's
> > > > > >>> runtime,
> > > > > >> being the main author of improvements like Buffer Debloating
> > > > > >>> (FLIP-183)
> > > > > >> [1], solved many bugs and fixed many test instabilities,
> > > generally
> > > > > > speaking
> > > > > >> helping with the maintenance of runtime components.
> > > > > >>
> > > > > >> Please join me in congratulating Anton Kalashnikov for
> > becoming
> > > a
> > > > > >>> Flink
> > > > > >> Committer!
> > > > > >>
> > > > > >> Best,
> > > > > >> Piotr Nowojski (on behalf of the Flink PMC)
> > > > > >>
> > > > > >> [1]
> > > > > >>
> > > > > >
> > > > > 
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-183%3A+Dynamic+buffer+size+adjustment
> > > > > >
> > > > > 
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Best regards,
> > Sergey
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Rui Fan

[Guowei Ma]

Congratulations, Rui!

Best,
Guowei


On Tue, Feb 21, 2023 at 2:57 PM Wei Zhong  wrote:

> Congratulations, Rui!
>
> Best,
> Wei
>
> > 2023年2月21日 下午1:52，Shammon FY  写道：
> >
> > Congratulations, Rui!
> >
> >
> > Best,
> > Shammon
> >
> > On Tue, Feb 21, 2023 at 1:40 PM Sergey Nuyanzin 
> wrote:
> >
> >> Congratulations, Rui!
> >>
> >> On Tue, Feb 21, 2023 at 4:53 AM Weihua Hu 
> wrote:
> >>
> >>> Congratulations, Rui!
> >>>
> >>> Best,
> >>> Weihua
> >>>
> >>>
> >>> On Tue, Feb 21, 2023 at 11:28 AM Biao Geng 
> wrote:
> >>>
>  Congrats, Rui!
>  Best,
>  Biao Geng
> 
>  weijie guo  于2023年2月21日周二 11:21写道：
> 
> > Congrats, Rui!
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > Leonard Xu  于2023年2月21日周二 11:03写道：
> >
> >> Congratulations, Rui!
> >>
> >> Best,
> >> Leonard
> >>
> >>> On Feb 21, 2023, at 9:50 AM, Matt Wang  wrote:
> >>>
> >>> Congrats Rui
> >>>
> >>>
> >>> --
> >>>
> >>> Best,
> >>> Matt Wang
> >>>
> >>>
> >>>  Replied Message 
> >>> | From | yuxia |
> >>> | Date | 02/21/2023 09:22 |
> >>> | To | dev |
> >>> | Subject | Re: [ANNOUNCE] New Apache Flink Committer - Rui Fan |
> >>> Congrats Rui
> >>>
> >>> Best regards,
> >>> Yuxia
> >>>
> >>> - 原始邮件 -
> >>> 发件人: "Samrat Deb" 
> >>> 收件人: "dev" 
> >>> 发送时间: 星期二, 2023年 2 月 21日 上午 1:09:25
> >>> 主题: Re: [ANNOUNCE] New Apache Flink Committer - Rui Fan
> >>>
> >>> Congrats Rui
> >>>
> >>> On Mon, 20 Feb 2023 at 10:28 PM, Anton Kalashnikov <
>  kaa@yandex.com
> >>
> >>> wrote:
> >>>
> >>> Congrats Rui!
> >>>
> >>> --
> >>> Best regards,
> >>> Anton Kalashnikov
> >>>
> >>> On 20.02.23 17:53, Matthias Pohl wrote:
> >>> Congratulations, Rui :)
> >>>
> >>> On Mon, Feb 20, 2023 at 5:10 PM Jing Ge
> >>  
> >>> wrote:
> >>>
> >>> Congrats Rui!
> >>>
> >>> On Mon, Feb 20, 2023 at 3:19 PM Piotr Nowojski <
> >>> pnowoj...@apache.org
> >
> >>> wrote:
> >>>
> >>> Hi, everyone
> >>>
> >>> On behalf of the PMC, I'm very happy to announce Rui Fan as a new
>  Flink
> >>> Committer.
> >>>
> >>> Rui Fan has been active on a small scale since August 2019, and
>  ramped
> >>> up
> >>> his contributions in the 2nd half of 2021. He was mostly involved
> >>> in
> >>> quite
> >>> demanding performance related work around the network stack and
> >>> checkpointing, like re-using TCP connections [1], and many
> >> crucial
> >>> improvements to the unaligned checkpoints. Among others:
> >> FLIP-227:
> >>> Support
> >>> overdraft buffer [2], Merge small ChannelState file for Unaligned
> >>> Checkpoint [3], Timeout aligned to unaligned checkpoint barrier
> >> in
>  the
> >>> output buffers [4].
> >>>
> >>> Please join me in congratulating Rui Fan for becoming a Flink
> >>> Committer!
> >>>
> >>> Best,
> >>> Piotr Nowojski (on behalf of the Flink PMC)
> >>>
> >>> [1] https://issues.apache.org/jira/browse/FLINK-22643
> >>> [2]
> >>>
> >>>
> >>>
> >>>
> >>
> >
> 
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-227%3A+Support+overdraft+buffer
> >>> [3] https://issues.apache.org/jira/browse/FLINK-26803
> >>> [4] https://issues.apache.org/jira/browse/FLINK-27251
> >>>
> >>>
> >>
> >>
> >
> 
> >>>
> >>
> >>
> >> --
> >> Best regards,
> >> Sergey
> >>
>
>

[ANNOUNCE] New Apache Flink PMC Member - Dong Lin

[Guowei Ma]

Hi, everyone

On behalf of the PMC, I'm very happy to announce Dong Lin as a new
Flink PMC.

Dong is currently the main driver of Flink ML. He reviewed a large
number of Flink ML related PRs and also participated in many Flink ML
improvements, such as "FLIP-173","FLIP-174" etc. At the same time, he made
a lot of evangelism events contributions for the Flink ML ecosystem.
In fact, in addition to the Flink machine learning field, Dong has also
participated in many other improvements in Flink, such as "FLIP-205",
"FLIP-266","FLIP-269","FLIP-274" etc.
Please join me in congratulating Dong Lin for becoming a Flink PMC!

Best,
Guowei(on behalf of the Flink PMC)

Re: [ANNOUNCE] New Apache Flink Committer - Jing Ge

[Guowei Ma]

Congratulations!
Best,
Guowei


On Tue, Feb 14, 2023 at 5:11 PM Leonard Xu  wrote:

> Congratulations!! Jing
>
> Best,
> Leonard
>
> > On Feb 14, 2023, at 4:33 PM, Hang Ruan  wrote:
> >
> > Congratulations Jing!
> >
> > Shammon FY  于2023年2月14日周二 16:04写道：
> >
> >> Congratulations Jing!
> >>
> >> Best,
> >> Shammon
> >>
> >> On Tue, Feb 14, 2023 at 4:00 PM weijie guo 
> >> wrote:
> >>
> >>> Congratulations Jing!
> >>>
> >>> Best regards,
> >>>
> >>> Weijie
> >>>
> >>>
> >>> Jingsong Li  于2023年2月14日周二 15:54写道：
> >>>
>  Congratulations Jing!
> 
>  Best,
>  Jingsong
> 
>  On Tue, Feb 14, 2023 at 3:50 PM godfrey he 
> >> wrote:
> >
> > Hi everyone,
> >
> > On behalf of the PMC, I'm very happy to announce Jing Ge as a new
> >> Flink
> > committer.
> >
> > Jing has been consistently contributing to the project for over 1
> >> year.
> > He authored more than 50 PRs and reviewed more than 40 PRs
> > with mainly focus on connector, test, and document modules.
> > He was very active on the mailing list (more than 90 threads) last
> >>> year,
> > which includes participating in a lot of dev discussions (30+),
> > providing many effective suggestions for FLIPs and answering
> > many user questions. He was the Flink Forward 2022 keynote speaker
> > to help promote Flink and  a trainer for Flink troubleshooting and
>  performance
> > tuning of Flink Forward 2022 training program.
> >
> > Please join me in congratulating Jing for becoming a Flink committer!
> >
> > Best,
> > Godfrey
> 
> >>>
> >>
>
>

Re: Re: [ANNOUNCE] New Apache Flink Committer - Weijie Guo

[Guowei Ma]

Congratulations!!! Weijie

Best,
Guowei


On Tue, Feb 14, 2023 at 1:49 PM Dian Fu  wrote:

> Congratulations, Weijie!
>
> Regards,
> Dian
>
> On Mon, Feb 13, 2023 at 10:55 PM Matthias Pohl
>  wrote:
>
> > Congrats, Weijie! :-)
> >
> > On Mon, Feb 13, 2023 at 10:50 AM Sergey Nuyanzin 
> > wrote:
> >
> > > Congratulations, Weijie!
> > >
> > > On Mon, Feb 13, 2023 at 10:32 AM Yun Tang  wrote:
> > >
> > > > Congratulations, Weijie!
> > > >
> > > > Best
> > > > Yun Tang
> > > >
> > > > On 2023/02/13 09:23:16 ramkrishna vasudevan wrote:
> > > > > Congratulations!!! Weijie
> > > > >
> > > > > Regards
> > > > > Ram
> > > > >
> > > > > On Mon, Feb 13, 2023 at 2:17 PM Yuepeng Pan 
> wrote:
> > > > >
> > > > > > Congratulations, Weijie!
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Best, Yuepeng Pan.
> > > > > >
> > > > > > 在 2023-02-13 16:26:32，"Lincoln Lee"  写道：
> > > > > > >Congratulations, Weijie!
> > > > > > >
> > > > > > >Best,
> > > > > > >Lincoln Lee
> > > > > > >
> > > > > > >
> > > > > > >Martijn Visser  于2023年2月13日周一
> 16:17写道：
> > > > > > >
> > > > > > >> Congrats Weijie!
> > > > > > >>
> > > > > > >> On Mon, Feb 13, 2023 at 7:19 AM Weihua Hu <
> > huweihua@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >>
> > > > > > >> > Congratulations, Weijie!
> > > > > > >> >
> > > > > > >> > Best,
> > > > > > >> > Weihua
> > > > > > >> >
> > > > > > >> > > On Feb 13, 2023, at 11:55, Lijie Wang <
> > > wangdachui9...@gmail.com
> > > > >
> > > > > > >> wrote:
> > > > > > >> > >
> > > > > > >> > > Congratulations, Weijie!
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best regards,
> > > Sergey
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Lincoln Lee

[Guowei Ma]

Congratulations, Lincoln!

Best,
Guowei


On Tue, Jan 10, 2023 at 2:57 PM Biao Geng  wrote:

> Congrats, Lincoln!
> Best,
> Biao Geng
>
> 获取 Outlook for iOS
> 
> 发件人: Wencong Liu 
> 发送时间: Tuesday, January 10, 2023 2:39:47 PM
> 收件人: dev@flink.apache.org 
> 主题: Re:Re: [ANNOUNCE] New Apache Flink Committer - Lincoln Lee
>
> Congratulations, Lincoln!
>
> Best regards,
> Wencong
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> 在 2023-01-10 13:25:09，"Yanfei Lei"  写道：
> >Congratulations, well deserved!
> >
> >Best,
> >Yanfei
> >
> >Yuan Mei  于2023年1月10日周二 13:16写道：
> >
> >> Congratulations, Lincoln!
> >>
> >> Best,
> >> Yuan
> >>
> >> On Tue, Jan 10, 2023 at 12:23 PM Lijie Wang 
> >> wrote:
> >>
> >> > Congratulations, Lincoln!
> >> >
> >> > Best,
> >> > Lijie
> >> >
> >> > Jingsong Li  于2023年1月10日周二 12:07写道：
> >> >
> >> > > Congratulations, Lincoln!
> >> > >
> >> > > Best,
> >> > > Jingsong
> >> > >
> >> > > On Tue, Jan 10, 2023 at 11:56 AM Leonard Xu 
> wrote:
> >> > > >
> >> > > > Congratulations, Lincoln!
> >> > > >
> >> > > > Impressive work in streaming semantics, well deserved!
> >> > > >
> >> > > >
> >> > > > Best,
> >> > > > Leonard
> >> > > >
> >> > > >
> >> > > > > On Jan 10, 2023, at 11:52 AM, Jark Wu  wrote:
> >> > > > >
> >> > > > > Hi everyone,
> >> > > > >
> >> > > > > On behalf of the PMC, I'm very happy to announce Lincoln Lee as
> a
> >> new
> >> > > Flink
> >> > > > > committer.
> >> > > > >
> >> > > > > Lincoln Lee has been a long-term Flink contributor since 2017.
> He
> >> > > mainly
> >> > > > > works on Flink
> >> > > > > SQL parts and drives several important FLIPs, e.g., FLIP-232
> (Retry
> >> > > Async
> >> > > > > I/O), FLIP-234 (
> >> > > > > Retryable Lookup Join), FLIP-260 (TableFunction Finish).
> Besides,
> >> He
> >> > > also
> >> > > > > contributed
> >> > > > > much to Streaming Semantics, including the non-determinism
> problem
> >> > and
> >> > > the
> >> > > > > message
> >> > > > > ordering problem.
> >> > > > >
> >> > > > > Please join me in congratulating Lincoln for becoming a Flink
> >> > > committer!
> >> > > > >
> >> > > > > Cheers,
> >> > > > > Jark Wu
> >> > > >
> >> > >
> >> >
> >>
>

Re: [DISCUSS] FLIP-266: Simplify network memory configurations for TaskManager

[Guowei Ma]

Hi,
Thank you very much for driving this FLIP in order to improve user
usability.

I understand that a key goal of this FLIP is to adjust the memory
requirements of shuffle to a more reasonable range. Through this adaptive
range adjustment, the memory efficiency can be improved under the premise
of ensuring the performance, thereby improving the user experience.

I have no problem with this goal, but I have a concern about the means of
implementation: should we introduce a _new_ non-orthogonal
option(`taskmanager.memory.network.required-buffer-per-gate.max`). That is
to say, the option will affect both streaming and batch shuffle behavior at
the same time.

>From the description in FLIP, we can see that we do not want this value to
be the same in streaming and batch scenarios. But we still let the user
configure this parameter, and once this parameter is configured, the
shuffle behavior of streaming and batch may be the same. In theory, there
may be a configuration that can meet the requirements of batch shuffle, but
it will affect the performance of streaming shuffle. (For example, we need
to reduce the memory overhead in batch scenarios, but it will affect the
performance of streaming shuffle). In other words, do we really want to add
a new option that exposes this possible risk problem?

  Personally, I think there might be two ways:
1. Modify the current implementation of streaming shuffle. Don't let
the streaming shuffle performance regression. In this way, this option will
not couple streaming shuffle and batch shuffle. This also avoids confusion
for the user.  But I am not sure how to do it. :-）
2. Introduce a pure batch read option, similar to the one introduced on
the batch write side.

BTW: It's better not to expose more implementation-related concepts to
users. For example, the "gate" is related to the internal implementation.
Relatively speaking, `shuffle.read/shuffle.client.read` may be more
general. After all, it can also avoid coupling with the topology structure
and scheduling units.

Best,
Guowei


On Fri, Dec 23, 2022 at 2:57 PM Lijie Wang  wrote:

> Hi,
>
> Thanks for driving this FLIP, +1 for the proposed changes.
>
> Limit the maximum value of shuffle read memory is very useful when using
> when using adaptive batch scheduler. Currently, the adaptive batch
> scheduler may cause a large number of input channels in a certain TM, so we
> generally recommend that users configure
> "taskmanager.network.memory.buffers-per-channel: 0" to decrease the the
> possibility of “Insufficient number of network buffers” error. After this
> FLIP, users no longer need to configure the
> "taskmanager.network.memory.buffers-per-channel".
>
> So +1 from my side.
>
> Best,
> Lijie
>
> Xintong Song  于2022年12月20日周二 10:04写道：
>
> > Thanks for the proposal, Yuxin.
> >
> > +1 for the proposed changes. I think these are indeed helpful usability
> > improvements.
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Mon, Dec 19, 2022 at 3:36 PM Yuxin Tan 
> wrote:
> >
> > > Hi, devs,
> > >
> > > I'd like to start a discussion about FLIP-266: Simplify network memory
> > > configurations for TaskManager[1].
> > >
> > > When using Flink, users may encounter the following issues that affect
> > > usability.
> > > 1. The job may fail with an "Insufficient number of network buffers"
> > > exception.
> > > 2. Flink network memory size adjustment is complex.
> > > When encountering these issues, users can solve some problems by adding
> > or
> > > adjusting parameters. However, multiple memory config options should be
> > > changed. The config option adjustment requires understanding the
> detailed
> > > internal implementation, which is impractical for most users.
> > >
> > > To simplify network memory configurations for TaskManager and improve
> > Flink
> > > usability, this FLIP proposed some optimization solutions for the
> issues.
> > >
> > > Looking forward to your feedback.
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-266%3A+Simplify+network+memory+configurations+for+TaskManager
> > >
> > > Best regards,
> > > Yuxin
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer

[Guowei Ma]

Congratulations Danny!
Best,
Guowei


On Tue, Nov 1, 2022 at 2:20 PM weijie guo  wrote:

> Congratulations Danny!
>
> Best regards,
>
> Weijie
>
>
> Maximilian Michels  于2022年10月13日周四 21:41写道：
>
> > Congratulations Danny! Well deserved :)
> >
> > -Max
> >
> > On Thu, Oct 13, 2022 at 2:40 PM Yang Wang  wrote:
> >
> > > Congratulations Danny!
> > >
> > > Best,
> > > Yang
> > >
> > > Hang Ruan  于2022年10月13日周四 10:58写道：
> > >
> > > > Congratulations Danny!
> > > >
> > > > Best,
> > > > Hang
> > > >
> > > > Yun Gao  于2022年10月13日周四 10:56写道：
> > > >
> > > > > Congratulations Danny!
> > > > > Best,
> > > > > Yun Gao
> > > > > --
> > > > > From:yuxia 
> > > > > Send Time:2022 Oct. 12 (Wed.) 09:49
> > > > > To:dev 
> > > > > Subject:Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer
> > > > > Congratulations Danny!
> > > > > Best regards,
> > > > > Yuxia
> > > > > - 原始邮件 -
> > > > > 发件人: "Xingbo Huang" 
> > > > > 收件人: "dev" 
> > > > > 发送时间: 星期三, 2022年 10 月 12日 上午 9:44:22
> > > > > 主题: Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer
> > > > > Congratulations Danny!
> > > > > Best,
> > > > > Xingbo
> > > > > Sergey Nuyanzin  于2022年10月12日周三 01:26写道：
> > > > > > Congratulations, Danny
> > > > > >
> > > > > > On Tue, Oct 11, 2022, 15:18 Lincoln Lee 
> > > > wrote:
> > > > > >
> > > > > > > Congratulations Danny!
> > > > > > >
> > > > > > > Best,
> > > > > > > Lincoln Lee
> > > > > > >
> > > > > > >
> > > > > > > Congxian Qiu  于2022年10月11日周二 19:42写道：
> > > > > > >
> > > > > > > > Congratulations Danny!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Congxian
> > > > > > > >
> > > > > > > >
> > > > > > > > Leonard Xu  于2022年10月11日周二 18:03写道：
> > > > > > > >
> > > > > > > > > Congratulations Danny!
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Leonard
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Junhan Yang

[Guowei Ma]

Congratulations, Junhan!
Best,
Guowei


On Fri, Aug 19, 2022 at 6:01 AM Jing Ge  wrote:

> Congrats Junhan!
>
> Best regards,
> Jing
>
> On Thu, Aug 18, 2022 at 12:05 PM Jark Wu  wrote:
>
> > Congrats and welcome Junhan!
> >
> > Cheers,
> > Jark
> >
> > > 2022年8月18日 17:59，Timo Walther  写道：
> > >
> > > Congratulations and welcome to the committer team :-)
> > >
> > > Regards,
> > > Timo
> > >
> > > On 18.08.22 07:19, Lijie Wang wrote:
> > >> Congratulations, Junhan!
> > >> Best,
> > >> Lijie
> > >> Leonard Xu  于2022年8月18日周四 11:31写道：
> > >>> Congratulations, Junhan!
> > >>>
> > >>> Best,
> > >>>
> >  2022年8月18日 上午11:27，Zhipeng Zhang  写道：
> > 
> >  Congratulations, Junhan!
> > 
> >  Xintong Song  于2022年8月18日周四 11:21写道：
> > >
> > > Hi everyone,
> > >
> > > On behalf of the PMC, I'm very happy to announce Junhan Yang as a
> new
> > >>> Flink
> > > committer.
> > >
> > > Junhan has been contributing to the Flink project for more than 1
> > year.
> > >>> His
> > > contributions are mostly identified in the web frontend, including
> > > FLIP-241, FLIP-249 and various maintenance efforts of Flink's
> > frontend
> > > frameworks.
> > >
> > > Please join me in congratulating Junhan for becoming a Flink
> > committer!
> > >
> > > Best,
> > > Xintong
> > 
> > 
> > 
> >  --
> >  best,
> >  Zhipeng
> > >>>
> > >>>
> > >
> >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Lijie Wang

[Guowei Ma]

Congratulations, Lijie. Welcome on board~!
Best,
Guowei


On Wed, Aug 17, 2022 at 6:25 PM Zhu Zhu  wrote:

> Hi everyone,
>
> On behalf of the PMC, I'm very happy to announce Lijie Wang as
> a new Flink committer.
>
> Lijie has been contributing to Flink project for more than 2 years.
> He mainly works on the runtime/coordination part, doing feature
> development, problem debugging and code reviews. He has also
> driven the work of FLIP-187(Adaptive Batch Scheduler) and
> FLIP-224(Blocklist for Speculative Execution), which are important
> to run batch jobs.
>
> Please join me in congratulating Lijie for becoming a Flink committer!
>
> Cheers,
> Zhu
>

Re: [VOTE] FLIP-245: Source Supports Speculative Execution For Batch Job

[Guowei Ma]

+1 (binding)
Best,
Guowei


On Tue, Jul 5, 2022 at 12:38 PM Jiangang Liu 
wrote:

> +1 for the feature.
>
> Jing Zhang  于2022年7月5日周二 11:43写道：
>
> > Hi all,
> >
> > I'd like to start a vote for FLIP-245: Source Supports Speculative
> > Execution For Batch Job[1] on the discussion thread [2].
> >
> > The vote will last for at least 72 hours unless there is an objection or
> > insufficient votes.
> >
> > Best,
> > Jing Zhang
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-245%3A+Source+Supports+Speculative+Execution+For+Batch+Job
> > [2] https://lists.apache.org/thread/zvc5no4yxvwkto7xxpw1vo7j1p6h0lso
> >
>

Re: [DISCUSS] FLIP-245: Source Supports Speculative Execution For Batch Job

[Guowei Ma]

Hi, Jing

Thanks a lot for writing this FLIP, which is very useful to Batch users.
Currently  I have only two small questions:

1. First of all, please complete the fault-tolerant processing flow in the
FLIP. (Maybe you've already considered it, but it's better to explicitly
give the specific solution in the FLIP.)
For example, how to handle Source `Reader` in case of error. As far as I
know, once the reader is unavailable, it will result in the inability to
allocate a new split, which may be unacceptable in the case of speculative
execution.

2. Secondly the FLIP only says that user-defined events are not supported,
but it does not explain how to deal with the existing
ReportedWatermarkEvent/ReaderRegistrationEvent. After all, in the case of
speculative execution, there may be two "same" tasks being executed at the
same time. If these events are repeated, whether they really have no effect
on the execution of the job, there is still a clear evaluation.

Best,
Guowei


On Fri, Jun 24, 2022 at 5:41 PM Jing Zhang  wrote:

> Hi all,
> One major problem of Flink batch jobs is slow tasks running on hot/bad
> nodes, resulting in very long execution time.
>
> In order to solve this problem, FLIP-168: Speculative Execution for Batch
> Job[1] is introduced and approved recently.
>
> Here, Zhu Zhu and I propose to support speculative execution of sources as
> one of follow up of FLIP-168. You could find more details in FLIP-245[2].
> Looking forward to your feedback.
>
> Best,
> Jing Zhang
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+Execution+for+Batch+Job#FLIP168:SpeculativeExecutionforBatchJob-NointegrationwithFlink'swebUI
>
> [2]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-245%3A+Source+Supports+Speculative+Execution+For+Batch+Job
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

[Guowei Ma]

Congrats, Jingsong!

Best,
Guowei


On Thu, Jun 16, 2022 at 9:49 AM Hangxiang Yu  wrote:

> Congrats, Jingsong!
>
> Best,
> Hangxiang
>
> On Thu, Jun 16, 2022 at 9:46 AM Aitozi  wrote:
>
> > Congrats, Jingsong!
> >
> > Best,
> > Aitozi
> >
> > Zhuoluo Yang  于2022年6月16日周四 09:26写道：
> >
> > > Many congratulations to teacher Lee!
> > >
> > > Thanks,
> > > Zhuoluo
> > >
> > >
> > > Dian Fu  于2022年6月16日周四 08:54写道：
> > >
> > > > Congratulations, Jingsong!
> > > >
> > > > Regards,
> > > > Dian
> > > >
> > > > On Thu, Jun 16, 2022 at 1:08 AM Yu Li  wrote:
> > > >
> > > > > Congrats, Jingsong!
> > > > >
> > > > > Best Regards,
> > > > > Yu
> > > > >
> > > > >
> > > > > On Wed, 15 Jun 2022 at 15:26, Sergey Nuyanzin  >
> > > > wrote:
> > > > >
> > > > > > Congratulations, Jingsong!
> > > > > >
> > > > > > On Wed, Jun 15, 2022 at 8:45 AM Jingsong Li <
> > jingsongl...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks everyone.
> > > > > > >
> > > > > > > It's great to be with you in the Flink community!
> > > > > > >
> > > > > > > Best,
> > > > > > > Jingsong
> > > > > > >
> > > > > > > On Wed, Jun 15, 2022 at 2:11 PM Yun Gao
> > >  > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Congratulations, Jingsong!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Yun Gao
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > --
> > > > > > > > From:Jing Zhang 
> > > > > > > > Send Time:2022 Jun. 14 (Tue.) 11:05
> > > > > > > > To:dev 
> > > > > > > > Subject:Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong
> > Lee
> > > > > > > >
> > > > > > > > Congratulations, Jingsong!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Jing Zhang
> > > > > > > >
> > > > > > > > Leonard Xu  于2022年6月14日周二 10:54写道：
> > > > > > > >
> > > > > > > > > Congratulations, Jingsong!
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Leonard
> > > > > > > > >
> > > > > > > > > > 2022年6月13日 下午6:52，刘首维  写道：
> > > > > > > > > >
> > > > > > > > > > Congratulations and well deserved, Jingsong!
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Shouwei
> > > > > > > > > > --原始邮件--
> > > > > > > > > > 发件人:
> > > > > > > > > "dev"
> > > > > > > > >   <
> > > > > > > luoyu...@alumni.sjtu.edu.cn
> > > > > > > > > ;
> > > > > > > > > > 发送时间:2022年6月13日(星期一) 晚上6:09
> > > > > > > > > > 收件人:"dev" > > > > dev@flink.apache.org
> > > > > > > >;
> > > > > > > > > >
> > > > > > > > > > 主题:Re: [ANNOUNCE] New Apache Flink PMC Member -
> > > Jingsong
> > > > > Lee
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Congratulations, Jingsong!
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Yuxia
> > > > > > > > > >
> > > > > > > > > > - 原始邮件 -
> > > > > > > > > > 发件人: "Yun Tang"  > > > > > > > > > 收件人: "dev"  > > > > > > > > > 发送时间: 星期一, 2022年 6 月 13日 下午 6:12:24
> > > > > > > > > > 主题: Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong
> > Lee
> > > > > > > > > >
> > > > > > > > > > Congratulations, Jingsong! Well deserved.
> > > > > > > > > >
> > > > > > > > > > Best
> > > > > > > > > > Yun Tang
> > > > > > > > > > 
> > > > > > > > > > From: Xingbo Huang  > > > > > > > > > Sent: Monday, June 13, 2022 17:39
> > > > > > > > > > To: dev  > > > > > > > > > Subject: Re: [ANNOUNCE] New Apache Flink PMC Member -
> > > Jingsong
> > > > > Lee
> > > > > > > > > >
> > > > > > > > > > Congratulations, Jingsong!
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Xingbo
> > > > > > > > > >
> > > > > > > > > > Jane Chan  > 17:23写道：
> > > > > > > > > >
> > > > > > > > > >  Congratulations, Jingsong!
> > > > > > > > > > 
> > > > > > > > > >  Best,
> > > > > > > > > >  Jane Chan
> > > > > > > > > > 
> > > > > > > > > >  On Mon, Jun 13, 2022 at 4:43 PM Shuo Cheng <
> > > > > > njucs...@gmail.com
> > > > > > > > >  wrote:
> > > > > > > > > > 
> > > > > > > > > >   Congratulations, Jingsong!
> > > > > > > > > >  
> > > > > > > > > >   On 6/13/22, Paul Lam  > >  > > > > > > > > paullin3...@gmail.com> wrote:
> > > > > > > > > >Congrats, Jingsong! Well deserved!
> > > > > > > > > >   
> > > > > > > > > >Best,
> > > > > > > > > >Paul Lam
> > > > > > > > > >   
> > > > > > > > > >2022年6月13日 16:31，Lincoln Lee <
> > > > > > > lincoln.8...@gmail.com
> > > > > > > > >  写道：
> > > > > > > > > >   
> > > > > > > > > >Congratulations, Jingsong!
> > > > > > > > > >   
> > > > > > > > > >Best,
> > > > > > > > > >Lincoln Lee
> > > > > > > > > >   
> > > > > > > > > >   
> > > > > > > > > >Jark Wu  > > > > > > imj...@gmail.com>
> > > > > > > > > 于2022年6月13日周一 

Re: [VOTE] FLIP-168: Speculative Execution for Batch Job

[Guowei Ma]

+1 (binding)
Best,
Guowei


On Fri, May 27, 2022 at 12:41 AM Shqiprim Bunjaku 
wrote:

> +1 (non-binding)
>
> Best Regards
> Shqiprim
>
> On Thu, May 26, 2022 at 6:22 PM rui fan <1996fan...@gmail.com> wrote:
>
> > Hi
> >
> > +1(non-binding), it’s very useful for batch job stability.
> >
> > Best wishes
> > fanrui
> >
> > On Thu, May 26, 2022 at 15:56 Zhu Zhu  wrote:
> >
> > > Hi everyone,
> > >
> > > Thanks for the feedback for FLIP-168: Blocklist Mechanism [1] on the
> > > discussion thread [2].
> > >
> > > I'd like to start a vote for it. The vote will last for at least 72
> hours
> > > unless there is an objection or insufficient votes.
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+execution+for+Batch+Job
> > > [2] https://lists.apache.org/thread/ot352tp8t7mclzx9zfv704gcm0fwrq58
> > >
> >
>

Re: [ANNOUNCE] New Flink PMC member: Yang Wang

[Guowei Ma]

Congratulations！
Best,
Guowei


On Thu, May 5, 2022 at 9:01 PM Jiangang Liu 
wrote:

> Congratulations!
>
> Best
> Liu Jiangang
>
> Marios Trivyzas  于2022年5月5日周四 20:47写道：
>
> > Congrats Yang!
> >
> > On Thu, May 5, 2022, 15:29 Yuan Mei  wrote:
> >
> > > Congrats and well Deserved, Yang!
> > >
> > > Best,
> > > Yuan
> > >
> > > On Thu, May 5, 2022 at 8:21 PM Nicholas Jiang <
> nicholasji...@apache.org>
> > > wrote:
> > >
> > > > Congrats Yang!
> > > >
> > > > Best regards,
> > > > Nicholas Jiang
> > > >
> > > > On 2022/05/05 11:18:10 Xintong Song wrote:
> > > > > Hi all,
> > > > >
> > > > > I'm very happy to announce that Yang Wang has joined the Flink PMC!
> > > > >
> > > > > Yang has been consistently contributing to our community, by
> > > contributing
> > > > > codes, participating in discussions, mentoring new contributors,
> > > > answering
> > > > > questions on mailing lists, and giving talks on Flink at
> > > > > various conferences and events. He is one of the main contributors
> > and
> > > > > maintainers in Flink's Native Kubernetes / Yarn integrations and
> the
> > > > Flink
> > > > > Kubernetes Operator.
> > > > >
> > > > > Congratulations and welcome, Yang!
> > > > >
> > > > > Thank you~
> > > > >
> > > > > Xintong Song (On behalf of the Apache Flink PMC)
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-217 Support watermark alignment of source splits

[Guowei Ma]

Hi,

We know that in the case of Bounded input Flink supports the Batch
execution mode. Currently in Batch execution mode, flink is executed on a
stage-by-stage basis. In this way, perhaps watermark alignment might not
gain much.

So my question is: Is watermark alignment the default behavior(for
implemented source only)? If so, have you considered evaluating the impact
of this behavior on the Batch execution mode? Or thinks it is not necessary.

Correct me if I miss something.

Best,
Guowei


On Thu, May 5, 2022 at 1:01 PM Piotr Nowojski 
wrote:

> Hi Becket and Dawid,
>
> > I feel that no matter which option we choose this can not be solved
> entirely in either of the options, because of the point above and because
> the signature of SplitReader#pauseOrResumeSplits and
> SourceReader#pauseOrResumeSplits are slightly different (one identifies
> splits with splitId the other one passes the splits directly).
>
> Yes, that's a good point in this case and for features that need to be
> implemented in more than one place.
>
> > Is there any reason for pausing reading from a split an optional feature,
> > other than that this was not included in the original interface?
>
> An additional argument in favor of making it optional is to simplify source
> implementation. But on its own I'm not sure if that would be enough to
> justify making this feature optional. Maybe.
>
> > I think it would be way simpler and clearer to just let end users and
> Flink
> > assume all the connectors will implement this feature.
>
> As I wrote above that would be an interesting choice to make (ease of
> implementation for new users, vs system consistency). Regardless of that,
> yes, for me the main argument is the API backward compatibility. But let's
> clear a couple of points:
> - The current proposal adding methods to the base interface with default
> implementations is an OPTIONAL feature. Same as the decorative version
> would be.
> - Decorative version could implement "throw UnsupportedOperationException"
> if user enabled watermark alignment just as well and I agree that's a
> better option compared to logging a warning.
>
> Best,
> Piotrek
>
>
> śr., 4 maj 2022 o 15:40 Becket Qin  napisał(a):
>
> > Thanks for the reply and patient discussion, Piotr and Dawid.
> >
> > Is there any reason for pausing reading from a split an optional feature,
> > other than that this was not included in the original interface?
> >
> > To be honest I am really worried about the complexity of the user story
> > here. Optional features like this have a high overhead. Imagine this
> > feature is optional, now a user enabled watermark alignment and defined a
> > few watermark groups. Would it work? Hmm, that depends on whether the
> > involved Source has implmemented this feature. If the Sources are well
> > documented, good luck. Otherwise end users may have to look into the code
> > of the Source to see whether the feature is supported. Which is something
> > they shouldn't have to do.
> >
> > I think it would be way simpler and clearer to just let end users and
> Flink
> > assume all the connectors will implement this feature. After all the
> > watermark group is not optinoal to the end users. If in some rare cases,
> > the feature cannot be supported, a clear UnsupportedOperationException
> will
> > be thrown to tell users to explicitly remove this Source from the
> watermark
> > group. I don't think we should have a warning message here, as they tend
> to
> > be ignored in many cases. If we do this, we don't even need the
> supportXXX
> > method in the Source for this feature. In fact this is exactly how many
> > interfaces works today. For example, SplitEnumerator#addSplitsBack() is
> not
> > supported by Pravega source because it does not support partial failover.
> > In that case, it simply throws an exception to trigger a global recovery.
> >
> > The reason we add a default implementation in this case would just for
> the
> > sake of backwards compatibility so the old source can still compile.
> Sure,
> > in short term, this feature might not be supported by many existing
> > sources. That is OK, and it is quite visible to the source developers
> that
> > they did not override the default impl which throws an
> > UnsupportedOperationException.
> >
> > @Dawid,
> >
> > the Java doc of the SupportXXX() method in the Source would be the single
> > >> source of truth regarding how to implement this feature.
> > >
> > >
> >
> > I also don't find it entirely true. Half of the classes are theoretically
> > > optional and are utility classes from the point of view how the
> > interfaces
> > > are organized. Theoretically users do not need to use any of
> > > SourceReaderBase & SplitReader. Would be weird to list their methods in
> > the
> > > Source interface.
> >
> > I think the ultimate goal of java docs is to guide users to implement the
> > Source. If SourceReaderBase is the preferred way to implement a
> > SourceReader, it seems worth mentioning 

Re: [ANNOUNCE] Apache Flink 1.15.0 released

[Guowei Ma]

Hi, Yun

Great job!
Thank you very much for your efforts to release Flink-1.15 during this time.
Thanks also to all the contributors who worked on this release!

Best,
Guowei


On Thu, May 5, 2022 at 3:24 PM Peter Schrott  wrote:

> Great!
>
> Will install it on the cluster asap! :)
>
> One thing I noticed: the linked release notes in the blog announcement
> under "Upgrade Notes" result in a 404
> (
>
> https://nightlies.apache.org/flink/flink-docs-release-1.15/release-notes/flink-1.15/
> )
>
> They are also not linked on the main page:
> https://nightlies.apache.org/flink/flink-docs-release-1.15/
>
> Keep it up!
> Peter
>
>
> On Thu, May 5, 2022 at 8:43 AM Martijn Visser 
> wrote:
>
> > Thank you Yun Gao, Till and Joe for driving this release. Your efforts
> are
> > greatly appreciated!
> >
> > To everyone who has opened Jira tickets, provided PRs, reviewed code,
> > written documentation or anything contributed in any other way, this
> > release was (once again) made possible by you! Thank you.
> >
> > Best regards,
> >
> > Martijn
> >
> > Op do 5 mei 2022 om 08:38 schreef Yun Gao 
> >
> >> The Apache Flink community is very happy to announce the release of
> >> Apache Flink 1.15.0, which is the first release for the Apache Flink
> >> 1.15 series.
> >>
> >> Apache Flink® is an open-source stream processing framework for
> >> distributed, high-performing, always-available, and accurate data
> >> streaming applications.
> >>
> >> The release is available for download at:
> >> https://flink.apache.org/downloads.html
> >>
> >> Please check out the release blog post for an overview of the
> >> improvements for this release:
> >> https://flink.apache.org/news/2022/05/05/1.15-announcement.html
> >>
> >> The full release notes are available in Jira:
> >>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350442
> >>
> >> We would like to thank all contributors of the Apache Flink community
> >> who made this release possible!
> >>
> >> Regards,
> >> Joe, Till, Yun Gao
> >>
> > --
> >
> > Martijn Visser | Product Manager
> >
> > mart...@ververica.com
> >
> > 
> >
> >
> > Follow us @VervericaData
> >
> > --
> >
> > Join Flink Forward  - The Apache Flink
> > Conference
> >
> > Stream Processing | Event Driven | Real Time
> >
> >
>

Re: Failed Unit Test on Master Branch

[Guowei Ma]

Hi Haizhou

I ran the test and there is no problem.
And commit is "d940af688be90c92ce4f8b9ca883f6753c94aa0f"

Best,
Guowei


On Fri, Apr 29, 2022 at 5:39 AM Haizhou Zhao 
wrote:

> Hello Flink Community,
>
> I was encountering some unit test failure in the flink-avro sub-module when
> I tried to pull down the master branch and build.
>
> Here is the command I ran:
>
> mvn clean package -pl flink-formats/flink-avro
>
> Here is the test that fails:
>
> https://github.com/apache/flink/blob/master/flink-formats/flink-avro/src/test/java/org/apache/flink/formats/avro/AvroRowDeSerializationSchemaTest.java#L178
>
> Here is the exception that was thrown:
> [ERROR]
>
> org.apache.flink.formats.avro.AvroRowDeSerializationSchemaTest.testGenericDeserializeSeveralTimes
>  Time elapsed: 0.008 s  <<< ERROR!
> java.io.IOException: Failed to deserialize Avro record.
> ...
>
> Here is the latest commit of the HEAD I pulled:
> commit c5430e2e5d4eeb0aba14ce3ea8401747afe0182d (HEAD -> master,
> oss/master)
>
> Can someone confirm this is indeed a problem on the master branch? If yes,
> any suggestions on fixing it?
>
> Thank you,
> Haizhou Zhao
>

Re: Re: [DISCUSS] FLIP-168: Speculative execution for Batch Job

[Guowei Ma]

Hi, zhu

Many thanks to zhuzhu for initiating the FLIP discussion. Overall I think
it's ok, I just have 3 small questions

1. How to judge whether the Execution Vertex belongs to a slow task.
The current calculation method is: the current timestamp minus the
timestamp of the execution deployment. If the execution time of this
execution exceeds the baseline, then it is judged as a slow task. Normally
this is no problem. But if an execution fails, the time may not be
accurate. For example, the baseline is 59 minutes, and a task fails after
56 minutes of execution. In the worst case, it may take an additional 59
minutes to discover that the task is a slow task.

2. Speculative Scheduler's fault tolerance strategy.
The strategy in FLIP is: if the Execution Vertex can be executed, even if
the execution fails, the fault tolerance strategy will not be adopted.
Although currently `ExecutionTimeBasedSlowTaskDetector` can restart an
execution. But isn't this dependency a bit too strong? To some extent, the
fault tolerance strategy and the Slow task detection strategy are coupled
together.


3. The value of the default configuration
IMHO, prediction execution should only be required for relatively
large-scale, very time-consuming and long-term jobs.
If `slow-task-detector.execution-time.baseline-lower-bound` is too small,
is it possible for the system to always start some additional tasks that
have little effect? In the end, the user needs to reset this default
configuration. Is it possible to consider a larger configuration. Of
course, this part is best to listen to the suggestions of other community
users.

Best,
Guowei


On Thu, Apr 28, 2022 at 3:54 PM Jiangang Liu 
wrote:

> +1 for the feature.
>
> Mang Zhang  于2022年4月28日周四 11:36写道：
>
> > Hi zhu:
> >
> >
> > This sounds like a great job! Thanks for your great job.
> > In our company, there are already some jobs using Flink Batch,
> > but everyone knows that the offline cluster has a lot more load than
> > the online cluster, and the failure rate of the machine is also much
> higher.
> > If this work is done, we'd love to use it, it's simply awesome for
> our
> > flink users.
> > thanks again!
> >
> >
> >
> >
> >
> >
> >
> > --
> >
> > Best regards,
> > Mang Zhang
> >
> >
> >
> >
> >
> > At 2022-04-27 10:46:06, "Zhu Zhu"  wrote:
> > >Hi everyone,
> > >
> > >More and more users are running their batch jobs on Flink nowadays.
> > >One major problem they encounter is slow tasks running on hot/bad
> > >nodes, resulting in very long and uncontrollable execution time of
> > >batch jobs. This problem is a pain or even unacceptable in
> > >production. Many users have been asking for a solution for it.
> > >
> > >Therefore, I'd like to revive the discussion of speculative
> > >execution to solve this problem.
> > >
> > >Weijun Wang, Jing Zhang, Lijie Wang and I had some offline
> > >discussions to refine the design[1]. We also implemented a PoC[2]
> > >and verified it using TPC-DS benchmarks and production jobs.
> > >
> > >Looking forward to your feedback!
> > >
> > >Thanks,
> > >Zhu
> > >
> > >[1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+execution+for+Batch+Job
> > >[2]
> > https://github.com/zhuzhurk/flink/commits/1.14-speculative-execution-poc
> > >
> > >
> > >刘建刚  于2021年12月13日周一 11:38写道：
> > >
> > >> Any progress on the feature? We have the same requirement in our
> > company.
> > >> Since the soft and hard environment can be complex, it is normal to
> see
> > a
> > >> slow task which determines the execution time of the flink job.
> > >>
> > >>  于2021年6月20日周日 22:35写道：
> > >>
> > >> > Hi everyone,
> > >> >
> > >> > I would like to kick off a discussion on speculative execution for
> > batch
> > >> > job.
> > >> > I have created FLIP-168 [1] that clarifies our motivation to do this
> > and
> > >> > some improvement proposals for the new design.
> > >> > It would be great to resolve the problem of long tail task in batch
> > job.
> > >> > Please let me know your thoughts. Thanks.
> > >> >   Regards,
> > >> > wangwj
> > >> > [1]
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+execution+for+Batch+Job
> > >> >
> > >>
> >
>

[jira] [Created] (FLINK-27406) After canceling the job, it is falsely reported as a job failure.

[Guowei Ma (Jira)]

Guowei Ma created FLINK-27406:
-

 Summary: After canceling the job, it is falsely reported as a job 
failure.
 Key: FLINK-27406
 URL: https://issues.apache.org/jira/browse/FLINK-27406
 Project: Flink
  Issue Type: Improvement
  Components: Client / Job Submission
Affects Versions: 1.15.0
Reporter: Guowei Ma


Submit a streaming job in application mode and I cancel it in the web ui. 

However the command give the job failed exception.  I think it would misleading 
the user. IMO we should tell the user the job is cancelled.
{code:java}
org.apache.flink.client.program.ProgramInvocationException: The main method 
caused an error: org.apache.flink.client.program.ProgramInvocationException: 
Job failed (JobID: c816efb84b476b67cc4934e79a0694cf)
    at 
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
    at 
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
    at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
    at 
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:836)
    at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:247)
    at 
org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1078)
    at 
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1156)
    at 
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
    at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1156)
Caused by: java.util.concurrent.ExecutionException: 
org.apache.flink.client.program.ProgramInvocationException: Job failed (JobID: 
c816efb84b476b67cc4934e79a0694cf)
    at 
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
    at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
    at 
org.apache.flink.client.program.StreamContextEnvironment.getJobExecutionResult(StreamContextEnvironment.java:173)
    at 
org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:123)
    at 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1969)
    at 
org.apache.flink.streaming.examples.windowing.TopSpeedWindowing.main(TopSpeedWindowing.java:154)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at 
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
    ... 8 more
Caused by: org.apache.flink.client.program.ProgramInvocationException: Job 
failed (JobID: c816efb84b476b67cc4934e79a0694cf)
    at 
org.apache.flink.client.deployment.ClusterClientJobClientAdapter.lambda$null$6(ClusterClientJobClientAdapter.java:130)
    at 
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
    at 
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
    at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
    at 
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
    at 
org.apache.flink.util.concurrent.FutureUtils.lambda$retryOperationWithDelay$9(FutureUtils.java:403)
    at 
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
    at 
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
    at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
    at 
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
    at 
org.apache.flink.client.program.rest.RestClusterClient.lambda$pollResourceAsync$26(RestClusterClient.java:708)
    at 
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
    at 
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
    at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
    at 
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
    at 
org.apache.flink.util.concurrent.FutureUtils.lambda$retryOperationWithDelay$9(FutureUtils.java:403)
    at 
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
    at 
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
    at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
    at 
java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
    at 
java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:943

Re: Re: [VOTE] Release 1.15.0, release candidate #4

[Guowei Ma]

+1 (binding)

1. Build from the source
2. Verify the checksums and signatures
3. Start a local standalone cluster
4. Submit a streaming example job and stop it on the ui.
5. Submit a job running with batch mode.

Best,
Guowei


On Tue, Apr 26, 2022 at 10:56 AM Zhu Zhu  wrote:

> +1 (binding)
> - Checked checksums and signatures
> - Built from source
> - Ran a batch job using AdaptiveBatchScheduler on yarn cluster
> - Ran an example job(parallelism=1000) on yarn cluster
> - Checked the website PR
>
> Thanks,
> Zhu
>
> Yun Gao  于2022年4月26日周二 10:37写道：
> >
> > Very thanks Matthias for tracking and double confirming this issue!
> >
> > Best,
> > Yun Gao
> >
> >
> >
> >  --Original Mail --
> > Sender:Matthias Pohl 
> > Send Date:Mon Apr 25 23:52:30 2022
> > Recipients:Yun Gao 
> > CC:dev , Yun Gao 
> > Subject:Re: [VOTE] Release 1.15.0, release candidate #4
> >
> > +1 (non-binding)
> >
> > The issue around the user mailing list thread [1] is identified. We
> couldn't come up with a reason for why some change in 1.15 would have
> caused the behavior. The bug itself is/was already present in 1.14. All the
> findings are collected in FLINK-27354 [2]. We're not considering this to be
> a blocker for 1.15.
> >
> > * Checked checksums
> > * Compiled Flink 1.15 from source
> > * Did a test run with ZooKeeper HA and standalone setup to verify the
> repeatable cleanup and JobResultStore working based on rc3; verified that
> the diff between rc3 and rc4 didn't add any significant changes to run
> >
> > [1] https://lists.apache.org/thread/5pm3crntmb1hl17h4txnlhjz34clghrg
> > [2] https://issues.apache.org/jira/browse/FLINK-27354
> > On Mon, Apr 25, 2022 at 4:38 AM Yun Gao  wrote:
> >
> > Very thanks Peter for the help! The vote time
> > would exclude the weekend, thus no worry for
> > that.
> >
> > Best,
> > Yun Gao
> >
> >
> > --
> > From:Peter Schrott 
> > Send Time:2022 Apr. 25 (Mon.) 09:17
> > To:Matthias Pohl 
> > Cc:dev ; Yun Gao 
> > Subject:Re: [VOTE] Release 1.15.0, release candidate #4
> >
> > Hi all,
> >
> > Unfortunately I could not extract the logs in time today - the ordering
> in
> > the csv was wrong and the csv was not parsable due to extra commas in the
> > logs... I discovered the issue in a pre-production cluster.
> >
> > I am gone for the weekend - I don't have access to the systems - but I
> will
> > retry to get the logs Monday morning, CEST.
> >
> > Sorry for the delay. I hope this heads up helps.
> >
> > Best, Peter
> >
> > Matthias Pohl  schrieb am Fr., 22. Apr. 2022,
> 17:44:
> >
> > > An issue was reported on the user ML [1] by Peter (CC) that was
> observed
> > > on rc4. We couldn't find any blocker based on the description so far.
> But
> > > it doesn't explain the entire problem the user is describing, though.
> We're
> > > still waiting for the logs to investigate the cause of the JobMaster
> not
> > > terminating.
> > >
> > > Considering that it's unclear when the logs are provided and that we
> > > haven't found anything similar during the release testing we might
> want to
> > > handle this as a non-blocking issue if Peter cannot provide the logs
> in a
> > > reasonable time frame. But we might want to give him maybe till Monday
> to
> > > provide the logs. WDYT?
> > >
> > > Matthias
> > >
> > > [1] https://lists.apache.org/thread/5pm3crntmb1hl17h4txnlhjz34clghrg
> > >
> > > On Fri, Apr 22, 2022 at 11:03 AM Xingbo Huang 
> wrote:
> > >
> > >> +1 (non-binding)
> > >>
> > >> - Verified checksums and signatures
> > >> - Verified Python wheel package contents
> > >> - Pip install apache-flink-libraries source package and apache-flink
> wheel
> > >> package in Mac/Linux
> > >> - Run the examples from Python Table API Tutorial in Python REPL
> > >> - Test the Python UDF jobs in Thread Mode.
> > >>
> > >> Best,
> > >> Xingbo
> > >>
> > >> Dawid Wysakowicz  于2022年4月21日周四 18:52写道：
> > >>
> > >> > +1 (binding),
> > >> >
> > >> >- checked licenses diff to my previous checks on rc2, this time
> > >> >everything seems ok
> > >> >- checked checksums, signatures, there are no binaries
> > >> >- compiled from sources
> > >> >- run a standalone cluster, clicked through the UI
> > >> >- run StateMachineExample took a savepoint in native format and
> > >> >stopped with a savepoint in native format
> > >> >
> > >> > Best,
> > >> >
> > >> > Dawid
> > >> > On 21/04/2022 05:49, Yun Gao wrote:
> > >> >
> > >> > Hi everyone,
> > >> >
> > >> > Please review and vote on the release candidate #4 for the version
> > >> 1.15.0, as follows:
> > >> > [ ] +1, Approve the release[ ] -1, Do not approve the release
> (please
> > >> provide specific comments)
> > >> >
> > >> > The complete staging area is available for your review, which
> includes:
> > >> > * JIRA release notes [1],* the official Apache source release and
> > >> binary convenience releases to be deployed to dist.apache.org [2],
> > >> 

Re: [VOTE] Release 1.15.0, release candidate #3

[Guowei Ma]

+1(binding)

- Verified the signature and checksum of the release binary
- Run the SqlClient example
- Run the WordCount example
- Compile from the source and success

Best,
Guowei


On Mon, Apr 18, 2022 at 11:13 AM Xintong Song  wrote:

> +1 (binding)
>
> - verified signature and checksum
> - build from source
> - run example jobs in a standalone cluster, everything looks expected
>
>
> Thank you~
>
> Xintong Song
>
>
>
> On Fri, Apr 15, 2022 at 12:56 PM Yun Gao 
> wrote:
>
> > Hi everyone,
> >
> > Please review and vote on the release candidate #3 for the version
> 1.15.0,
> > as follows:
> > [ ] +1, Approve the release[ ] -1, Do not approve the release (please
> > provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * JIRA release notes [1],* the official Apache source release and binary
> > convenience releases to be deployed to dist.apache.org [2],
> >which are signed with the key with fingerprint
> > CBE82BEFD827B08AFA843977EDBF922A7BC84897 [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag "release-1.15.0-rc3" [5],* website pull request listing
> > the new release and adding announcement blog post [6].
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Joe, Till and Yun Gao
> > [1]
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350442
> > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.15.0-rc3/
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [4]
> > https://repository.apache.org/content/repositories/orgapacheflink-1497/
> > [5] https://github.com/apache/flink/releases/tag/release-1.15.0-rc3/
> > [6] https://github.com/apache/flink-web/pull/526
> >
> >
> >
>

Re: [ANNOUNCE] New scalafmt formatter has been merged

[Guowei Ma]

Hi, Francesco
Thanks for your work!
Best,
Guowei


On Wed, Apr 13, 2022 at 5:35 PM Dian Fu  wrote:

> Thanks a lot for this great work Francesco!
>
> Regards,
> Dian
>
> On Wed, Apr 13, 2022 at 3:23 PM Marios Trivyzas  wrote:
>
> > Thank you for this Francesco!
> >
> > It will really improve the lives of everyone touching scala code!
> >
> > Best,
> > Marios
> >
> > On Wed, Apr 13, 2022 at 9:55 AM Timo Walther  wrote:
> >
> > > Thanks for the great work Francesco!
> > >
> > > This will improve the contributor productivity a lot and ease reviews.
> > > This change was long overdue.
> > >
> > > Regards,
> > > Timo
> > >
> > > Am 12.04.22 um 17:21 schrieb Francesco Guardiani:
> > > > Hi all,
> > > > The new scalafmt formatter has been merged. From now on, just using
> mvn
> > > > spotless:apply as usual will format both Java and Scala, and Intellij
> > > will
> > > > automatically pick up the scalafmt config for who has the Scala
> plugin
> > > > installed. If it doesn't, just go in Preferences > Editor > Code
> Style
> > >
> > > > Scala and change the Formatter to scalafmt. If you use the actions on
> > > save
> > > > plugin, make sure you have the reformat on save enabled for Scala.
> > > >
> > > > For more details on integration with IDEs, please refer to
> > > > https://scalameta.org/scalafmt/docs/installation.html
> > > >
> > > > If you have a pending PR with Scala changes, chances are you're going
> > to
> > > > have conflicts with upstream/master now. In order to fix it, here is
> > the
> > > > suggested procedure:
> > > >
> > > > - Do an interactive rebase on commit
> > > > 3ea3fee5ac996f6ae8836c3cba252f974d20bd2e, which is the commit
> > before
> > > the
> > > > refactoring of the whole codebase, fixing as usual the
> conflicting
> > > changes.
> > > > This will make sure you won't miss the changes between your
> branch
> > > and
> > > > master *before* the reformatting commit.
> > > > - Do a rebase on commit 91d81c427aa6312841ca868d54e8ce6ea721cd60
> > > > accepting all changes from your local branch. You can easily do
> > that
> > > via git
> > > > rebase -Xours 91d81c427aa6312841ca868d54e8ce6ea721cd60
> > > > - Run mvn spotless:apply and commit all the changes
> > > > - Do an interactive rebase on upstream/master. This will make
> sure
> > > you
> > > > won't miss the changes between your branch and master *after* the
> > > > reformatting commit.
> > > > - Force push your branch to update the PR
> > > >
> > > > Sorry for this noise!
> > > >
> > > > Thank you,
> > > > FG
> > > >
> > >
> > >
> >
> > --
> > Marios
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Martijn Visser

[Guowei Ma]

Congrats Martijn!
Best,
Guowei


On Fri, Mar 11, 2022 at 10:36 PM Marios Trivyzas  wrote:

> Congrats Martijn!!
>
> Best,
> Marios
>
> On Thu, Mar 10, 2022 at 4:50 PM yu'an huang  wrote:
>
> > Congrats, Martijn!
> >
> >
> > Best
> > Yuan
> >
> >
> > > On 3 Mar 2022, at 11:49 PM, Robert Metzger 
> wrote:
> > >
> > > Hi everyone,
> > >
> > > On behalf of the PMC, I'm very happy to announce Martijn Visser as a
> new
> > > Flink committer.
> > >
> > > Martijn is a very active Flink community member, driving a lot of
> efforts
> > > on the dev@flink mailing list. He also pushes projects such as
> replacing
> > > Google Analytics with Matomo, so that we can generate our web analytics
> > > within the Apache Software Foundation.
> > >
> > > Please join me in congratulating Martijn for becoming a Flink
> committer!
> > >
> > > Cheers,
> > > Robert
> >
> >
>
> --
> Marios
>

Re: Re: Change of focus

[Guowei Ma]

Thanks Till for everything you've done for the community!
 It has been a great honor working with you.

Best,
Guowei


On Tue, Mar 1, 2022 at 9:26 PM Marios Trivyzas  wrote:

> Good luck in your future endeavours Till !!
>
> Best regards,
> Marios
>
> On Tue, Mar 1, 2022 at 8:10 AM Steven Wu  wrote:
>
> > Till, thank you for your immense contributions to the project and the
> > community.
> >
> > On Mon, Feb 28, 2022 at 9:16 PM Xintong Song 
> > wrote:
> >
> > > Thanks for everything, Till. It has been a great honor working with
> you.
> > > Good luck with your new chapter~!
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Tue, Mar 1, 2022 at 12:33 PM Zhilong Hong 
> > wrote:
> > >
> > > > Thank you for everything, Till! I've learned a lot from you.
> > > >
> > > > Good luck with your new adventure and the next chapter!
> > > >
> > > > Best,
> > > > Zhilong
> > > >
> > > > On Tue, Mar 1, 2022 at 12:08 PM Yun Tang  wrote:
> > > >
> > > > > Thanks a lot for your efforts and kindness of mentoring
> contributors
> > in
> > > > > Apache Flink community, Till!
> > > > >
> > > > > Good luck with your new adventure in your new life.
> > > > >
> > > > >
> > > > > Best
> > > > > Yun Tang
> > > > >
> > > > > 
> > > > > From: Yuan Mei 
> > > > > Sent: Tuesday, March 1, 2022 11:00
> > > > > To: dev 
> > > > > Subject: Re: Re: Change of focus
> > > > >
> > > > > Thanks Till for everything you've done for the community!
> > > > > Good luck with your new adventure and best wishes to your new life!
> > > > >
> > > > > Best Regards,
> > > > > Yuan
> > > > >
> > > > > On Tue, Mar 1, 2022 at 10:35 AM Zhu Zhu  wrote:
> > > > >
> > > > > > Thank you for all the efforts and good luck for the new
> adventure,
> > > > Till!
> > > > > >
> > > > > > Thanks,
> > > > > > Zhu
> > > > > >
> > > > > > Terry  于2022年3月1日周二 10:26写道：
> > > > > >
> > > > > > > Thanks a lot for your efforts! Good Luck!
> > > > > > >
> > > > > > > Jiangang Liu  于2022年3月1日周二 10:18写道：
> > > > > > >
> > > > > > > > Thanks for the efforts and help in flink, Till. Good luck!
> > > > > > > >
> > > > > > > > Best
> > > > > > > > Liu Jiangang
> > > > > > > >
> > > > > > > > Lijie Wang  于2022年3月1日周二 09:53写道：
> > > > > > > >
> > > > > > > > > Thanks for all your efforts Till. Good luck !
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Lijie
> > > > > > > > >
> > > > > > > > > Yun Gao  于2022年3月1日周二
> 01:15写道：
> > > > > > > > >
> > > > > > > > > > Very thanks Till for all the efforts! Good luck for the
> > next
> > > > > > chapter~
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Yun
> > > > > > > > > >
> > > > > > > > > >
> > > > > --
> > > > > > > > > > Sender:Piotr Nowojski
> > > > > > > > > > Date:2022/02/28 22:10:46
> > > > > > > > > > Recipient:dev
> > > > > > > > > > Theme:Re: Change of focus
> > > > > > > > > >
> > > > > > > > > > Good luck Till and thanks for all of your efforts.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Piotrek
> > > > > > > > > >
> > > > > > > > > > pon., 28 lut 2022 o 15:06 Aitozi 
> > > > > > napisał(a):
> > > > > > > > > >
> > > > > > > > > > > Good luck with the next chapter, will miss you :)
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Aitozi
> > > > > > > > > > >
> > > > > > > > > > > Jark Wu  于2022年2月28日周一 21:28写道：
> > > > > > > > > > >
> > > > > > > > > > > > Thank you Till for every things. It's great to work
> > with
> > > > you.
> > > > > > > Good
> > > > > > > > > > luck!
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Jark
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, 28 Feb 2022 at 21:26, Márton Balassi <
> > > > > > > > > balassi.mar...@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Thank you, Till. Good luck with the next chapter.
> :-)
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Feb 28, 2022 at 1:49 PM Flavio Pompermaier
> <
> > > > > > > > > > > pomperma...@okkam.it
> > > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Good luck for your new adventure Till!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Feb 28, 2022 at 12:00 PM Till Rohrmann <
> > > > > > > > > > trohrm...@apache.org
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I wanted to let you know that I will be less
> > active
> > > > in
> > > > > > the
> > > > > > > > > > > community
> > > > > > > > > > > > > > > because I’ve decided to start a new chapter in
> my
> > > > life.
> > > > > > > > Hence,
> > > > > > > > > > > please
> > > > > > > > > > > > > > don’t
> > > > > > > > > > > > > > > wonder if I might no longer be very 

Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin Knauf and Yun Gao

[Guowei Ma]

Congratulations

Best,
Guowei


On Thu, Feb 17, 2022 at 3:29 PM Yang Wang  wrote:

> Congratulations to all of you!
>
> Best,
> Yang
>
> Lijie Wang  于2022年2月17日周四 14:20写道：
>
> > Congratulations to all of you!
> >
> > Best,
> > Lijie
> >
> > Leonard Xu  于2022年2月17日周四 12:13写道：
> >
> > > Congratulations !
> > >
> > > Best,
> > > Leonard
> > >
> > > > 2022年2月17日 上午11:09，Yun Tang  写道：
> > > >
> > > > Congratulations to all of you!
> > > >
> > > > Best,
> > > > Yun Tang
> > > > 
> > > > From: Yangze Guo 
> > > > Sent: Thursday, February 17, 2022 10:44
> > > > To: dev 
> > > > Cc: ro...@apache.org 
> > > > Subject: Re: [ANNOUNCE] New Flink PMC members: Igal Shilman,
> Konstantin
> > > Knauf and Yun Gao
> > > >
> > > > Congratulations! Well deserved.
> > > >
> > > > Best,
> > > > Yangze Guo
> > > >
> > > > On Thu, Feb 17, 2022 at 10:15 AM Paul Lam 
> > wrote:
> > > >>
> > > >> Congrats to all of you!
> > > >>
> > > >> Best,
> > > >> Paul Lam
> > > >>
> > > >>> 2022年2月17日 01:24，Roman Khachatryan  写道：
> > > >>>
> > > >>> Congratulations!
> > > >>>
> > > >>> Regards,
> > > >>> Roman
> > > >>>
> > > >>> On Wed, Feb 16, 2022 at 5:22 PM Matthias Pohl <
> > matth...@ververica.com>
> > > wrote:
> > > 
> > >  Congratulations to all of you :)
> > > 
> > >  On Wed, Feb 16, 2022 at 4:04 PM Till Rohrmann <
> trohrm...@apache.org
> > >
> > > wrote:
> > > 
> > > > Congratulations! Well deserved!
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Wed, Feb 16, 2022 at 2:51 PM Martijn Visser <
> > > mart...@ververica.com>
> > > > wrote:
> > > >
> > > >> Congratulations to all of you!
> > > >>
> > > >> Best regards,
> > > >>
> > > >> Martijn Visser
> > > >> https://twitter.com/MartijnVisser82
> > > >>
> > > >>
> > > >> On Wed, 16 Feb 2022 at 14:29, Fabian Paul 
> > wrote:
> > > >>
> > > >>> Congrats to all three of you, well deserved.
> > > >>>
> > > >>> Best,
> > > >>> Fabian
> > > >>>
> > > >>> On Wed, Feb 16, 2022 at 2:23 PM Robert Metzger <
> > > rmetz...@apache.org>
> > > >>> wrote:
> > > 
> > >  Hi all,
> > > 
> > >  I would like to formally announce a few new Flink PMC members
> on
> > > the
> > > >> dev@
> > >  list. The PMC has not done a good job of always announcing new
> > PMC
> > > >>> members
> > >  (and committers) recently. I'll try to keep an eye on this in
> > the
> > > >> future
> > > >>> to
> > >  improve the situation.
> > > 
> > >  Nevertheless, I'm very happy to announce some very active
> > > community
> > > >>> members
> > >  as new PMC members:
> > > 
> > >  - Igal Shilman, added to the PMC in October 2021
> > >  - Konstantin Knauf, added to the PMC in January 2022
> > >  - Yun Gao, added to the PMC in February 2022
> > > 
> > >  Please join me in welcoming them to the Flink PMC!
> > > 
> > >  Best,
> > >  Robert
> > > >>>
> > > >>
> > > >>
> > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committers: Feng Wang, Zhipeng Zhang

[Guowei Ma]

Congratulations to Feng and Zhipeng!
Best,
Guowei


On Thu, Feb 17, 2022 at 3:30 PM Yang Wang  wrote:

> Congratulations!
>
> Best,
> Yang
>
> Lijie Wang  于2022年2月17日周四 14:21写道：
>
> > Congratulations to all of you!
> >
> > Best,
> > Lijie
> >
> > Zakelly Lan  于2022年2月17日周四 12:21写道：
> >
> > > Congratulations :-D
> > >
> > > On Thu, Feb 17, 2022 at 12:15 PM Leonard Xu  wrote:
> > >
> > > > Congratulations !
> > > >
> > > >
> > > > Best,
> > > > Leonard
> > > >
> > > > > 2022年2月17日 上午11:17，Paul Lam  写道：
> > > > >
> > > > > Congrats! Well deserved!
> > > > >
> > > > > Best,
> > > > > Paul Lam
> > > > >
> > > > >> 2022年2月16日 21:30，Robert Metzger  写道：
> > > > >>
> > > > >> Hi everyone,
> > > > >>
> > > > >> On behalf of the PMC, I'm very happy to announce two new Flink
> > > > >> committers: Feng Wang and Zhipeng Zhang!
> > > > >>
> > > > >> Feng is one of the most active Flink evangelists in China, with
> > plenty
> > > > of
> > > > >> public talks, blog posts and other evangelization activities. The
> > PMC
> > > > wants
> > > > >> to recognize and value these efforts by making Feng a committer!
> > > > >>
> > > > >> Zhipeng Zhang has made significant contributions to flink-ml, like
> > > most
> > > > of
> > > > >> the FLIPs for our ML efforts.
> > > > >>
> > > > >> Please join me in welcoming them as committers!
> > > > >>
> > > > >>
> > > > >> Best,
> > > > >> Robert
> > > > >
> > > >
> > > >
> > >
> >
>

Re: Request for Flink 1.12.8 release

[Guowei Ma]

Hi,
Yes, that's right.
Best,
Guowei


On Thu, Jan 20, 2022 at 2:08 PM V N, Suchithra (Nokia - IN/Bangalore) <
suchithra@nokia.com> wrote:

> Hi Guowei,
>
> Flink 1.12.7 is already released. Request is for 1.12.8 release with log4j
> 2.17.1. You meant 1.12.8 here?
>
> Regards,
> Suchithra
>
> -----Original Message-
> From: Guowei Ma 
> Sent: Thursday, January 20, 2022 11:26 AM
> To: dev 
> Subject: Re: Request for Flink 1.12.8 release
>
> Hi Suchithra
>
> I don't think there is a plan to release 1.12.7 for this. But I think you
> could build it from the source.[1]
>
> [1]
>
> https://github.com/apache/flink/tree/release-1.12#building-apache-flink-from-source
>
> Best,
> Guowei
>
>
> On Wed, Jan 19, 2022 at 7:11 PM V N, Suchithra (Nokia - IN/Bangalore) <
> suchithra@nokia.com> wrote:
>
> > Hello,
> >
> > We are using Apache Flink 1.12 version. Due to log4j security
> > vulnerabilities(CVE-2021-44228) we have upgraded to Flink 1.12.7 which
> > contains the fix for CVE-2021-44228(Critical) and
> CVE-2021-45046(Critical).
> > Later two more vulnerabilities are reported CVE-2021-45105(Moderate)
> > and
> > CVE-2021-44832(Moderate) which is fixed with Apache log4j 2.17.1 and
> > we were expecting patch release(Flink 1.12.8) with it.
> >
> > As per the community, it supports current and previous minor versions
> > (1.13, 1.14) with bug fixes.
> >
> > Flink community officially only supports current and previous minor
> > versions [1] (1.13, 1.14) with bug fixes. Personally I wouldn't expect
> > there will be another patch release for 1.12.
> >
> > If you really need an extra release for the unsupported version, the
> > most straightforward approach would be manually building the Flink
> > distribution from sources [2] with the patches you need.
> >
> > [1]
> > https://flink.apache.org/downloads.html#update-policy-for-old-releases
> > [2]
> >
> > https://github.com/apache/flink/tree/release-1.12#building-apache-flin
> > k-from-source
> >
> > Apache Flink 1.12.7 release with critical fix was really helpful. As
> > per the below ticket log4j 2.17.1 code changes are committed.
> > https://issues.apache.org/jira/browse/FLINK-25472
> > Since these are security fixes It will be helpful if Flink 1.12.8 will
> > be released. Could you please let us know if it is possible to plan
> > this release?
> >
> > Regards,
> > Suchithra
> >
> >
> >
> >
> >
>

Re: Request for Flink 1.12.8 release

[Guowei Ma]

Hi Suchithra

I don't think there is a plan to release 1.12.7 for this. But I think you
could build it from the source.[1]

[1]
https://github.com/apache/flink/tree/release-1.12#building-apache-flink-from-source

Best,
Guowei


On Wed, Jan 19, 2022 at 7:11 PM V N, Suchithra (Nokia - IN/Bangalore) <
suchithra@nokia.com> wrote:

> Hello,
>
> We are using Apache Flink 1.12 version. Due to log4j security
> vulnerabilities(CVE-2021-44228) we have upgraded to Flink 1.12.7 which
> contains the fix for CVE-2021-44228(Critical) and CVE-2021-45046(Critical).
> Later two more vulnerabilities are reported CVE-2021-45105(Moderate) and
> CVE-2021-44832(Moderate) which is fixed with Apache log4j 2.17.1 and we
> were expecting patch release(Flink 1.12.8) with it.
>
> As per the community, it supports current and previous minor versions
> (1.13, 1.14) with bug fixes.
>
> Flink community officially only supports current and previous minor
> versions [1] (1.13, 1.14) with bug fixes. Personally I wouldn't expect
> there will be another patch release for 1.12.
>
> If you really need an extra release for the unsupported version, the most
> straightforward approach would be manually building the Flink distribution
> from sources [2] with the patches you need.
>
> [1] https://flink.apache.org/downloads.html#update-policy-for-old-releases
> [2]
>
> https://github.com/apache/flink/tree/release-1.12#building-apache-flink-from-source
>
> Apache Flink 1.12.7 release with critical fix was really helpful. As per
> the below ticket log4j 2.17.1 code changes are committed.
> https://issues.apache.org/jira/browse/FLINK-25472
> Since these are security fixes It will be helpful if Flink 1.12.8 will be
> released. Could you please let us know if it is possible to plan this
> release?
>
> Regards,
> Suchithra
>
>
>
>
>

Re: [VOTE] FLIP-199: Change some default config values of blocking shuffle for better usability

[Guowei Ma]

+1（binding）
Best,
Guowei


On Wed, Jan 12, 2022 at 3:44 PM Jingsong Li  wrote:

> +1 Thanks Yingjie for driving.
>
> Best,
> Jingsong Lee
>
> On Wed, Jan 12, 2022 at 3:16 PM 刘建刚  wrote:
> >
> > +1 for the proposal. In fact, we have used these params in our inner
> flink
> > version for good performance.
> >
> > Yun Gao  于2022年1月12日周三 10:42写道：
> >
> > > +1 since it would highly improve the open-box experience for batch
> jobs.
> > >
> > > Thanks Yingjie for drafting the PR and initiating the discussion.
> > >
> > > Best,
> > > Yun
> > >
> > >
> > >
> > >  --Original Mail --
> > > Sender:Yingjie Cao 
> > > Send Date:Tue Jan 11 15:15:01 2022
> > > Recipients:dev 
> > > Subject:[VOTE] FLIP-199: Change some default config values of blocking
> > > shuffle for better usability
> > > Hi all,
> > >
> > > I'd like to start a vote on FLIP-199: Change some default config
> values of
> > > blocking shuffle for better usability [1] which has been discussed in
> this
> > > thread [2].
> > >
> > > The vote will be open for at least 72 hours unless there is an
> objection or
> > > not enough votes.
> > >
> > > [1]
> > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-199%3A+Change+some+default+config+values+of+blocking+shuffle+for+better+usability
> > > [2] https://lists.apache.org/thread/pt2b1f17x2l5rlvggwxs6m265lo4ly7p
> > >
> > > Best,
> > > Yingjie
> > >
>

Re: [VOTE] FLIP-191: Extend unified Sink interface to support small file compaction

[Guowei Ma]

+1(binding).
Thank you for driving this
Best,
Guowei


On Wed, Jan 5, 2022 at 5:15 AM Arvid Heise  wrote:

> +1 (binding).
>
> Thanks for driving!
>
> On Tue, Jan 4, 2022 at 10:31 AM Yun Gao 
> wrote:
>
> > +1 (binding).
> >
> > Very thanks for proposing the FLIP!
> >
> > Best,
> > Yun
> >
> >
> > --
> > From:Martijn Visser 
> > Send Time:2022 Jan. 4 (Tue.) 17:22
> > To:dev 
> > Subject:Re: [VOTE] FLIP-191: Extend unified Sink interface to support
> > small file compaction
> >
> > +1 (non-binding). Thanks for driving the FLIP!
> >
> > Best regards,
> >
> > Martijn
> >
> > On Tue, 4 Jan 2022 at 10:21, Fabian Paul  wrote:
> >
> > > Hi everyone,
> > >
> > > I'd like to start a vote on FLIP-191: Extend unified Sink interface to
> > > support small file compaction [1] that has been discussed in this
> > > thread [2].
> > >
> > > The vote will be open for at least 72 hours unless there is an
> objection
> > or
> > > not enough votes.
> > >
> > > Best,
> > > Fabian
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction
> > > [2] https://lists.apache.org/thread/97kwy315t9r4j02l8v0wotkll4tngb3m
> > >
> >
> >
>

[ANNOUNCE] New Apache Flink Committer - Yingjie Cao

[Guowei Ma]

Hi everyone,

On behalf of the PMC, I'm very happy to announce Yingjie Cao as a new Flink
committer.

Yingjie has submitted 88 PRs since he joined the Flink community for more
than 2 years. In general, his main contributions are concentrated in
Flink's Shuffle. Yingjie has done a lot of work in promoting the
performance and stability of TM Blocking Shuffle[1][2];In order to allow
Flink to use External/Remote Shuffle Service in batch production scenarios,
he also improves the Flink Shuffle architecture in 1.14 [3]. At the same
time, he has also done some related work in Streaming Shuffle, such as
Buffer Management[4] , Non-Blocking Network Output and some important bug
fixes.

Please join me in congratulating Yingjie for becoming a Flink committer!

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-148%3A+Introduce+Sort-Based+Blocking+Shuffle+to+Flink
[2] https://flink.apache.org/2021/10/26/sort-shuffle-part1.html
[3]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-184%3A+Refine+ShuffleMaster+lifecycle+management+for+pluggable+shuffle+service+framework
[4] https://issues.apache.org/jira/browse/FLINK-16428

Best,
Guowei

Re: Re: [DISCUSS] Improve the name and structure of job vertex and operator name for job

[Guowei Ma]

+1
This would be very helpful for our debugging online job.

Best,
Guowei


On Thu, Nov 11, 2021 at 8:03 PM Yuepeng Pan  wrote:

> +1. It's useful to understand the job topology.
> Looking forward to this feature.
> Best,
> Yuepeng Pan.
>
>
>
>
>
>
> At 2021-11-11 19:44:44, "Yangze Guo"  wrote:
> >+1. That's gonna help a lot for debugging.
> >
> >Best,
> >Yangze Guo
> >
> >On Thu, Nov 11, 2021 at 7:37 PM Till Rohrmann 
> wrote:
> >>
> >> This improvement looks like it makes the life of our users a lot easier
> >> when it comes to understanding logs and reading the UI. Hence +1.
> >>
> >> Cheers,
> >> Till
> >>
> >> On Thu, Nov 11, 2021 at 11:59 AM JING ZHANG 
> wrote:
> >>
> >> > Big +1.
> >> >
> >> > This is a problem frequently encountered in our production platform,
> look
> >> > forward to this improvement.
> >> >
> >> > Best,
> >> > Jing Zhang
> >> >
> >> > Martijn Visser  于2021年11月11日周四 下午6:26写道：
> >> >
> >> > > +1. Looks much better now
> >> > >
> >> > > On Thu, 11 Nov 2021 at 11:07, godfrey he 
> wrote:
> >> > >
> >> > > > Thanks for driving this, this improvement solves a long-complained
> >> > > > problem, +1
> >> > > >
> >> > > > Best,
> >> > > > Godfrey
> >> > > >
> >> > > > Jark Wu  于2021年11月11日周四 下午5:40写道：
> >> > > > >
> >> > > > > +1 for this. It looks much more clear and structured.
> >> > > > >
> >> > > > > Best,
> >> > > > > Jark
> >> > > > >
> >> > > > > On Thu, 11 Nov 2021 at 17:23, Chesnay Schepler <
> ches...@apache.org>
> >> > > > wrote:
> >> > > > >
> >> > > > > > I'm generally in favor of it, and there are already tickets
> that
> >> > > > > > proposed a dedicated operator/vertex description:
> >> > > > > >
> >> > > > > > https://issues.apache.org/jira/browse/FLINK-20388
> >> > > > > > https://issues.apache.org/jira/browse/FLINK-21858
> >> > > > > >
> >> > > > > > On 11/11/2021 10:02, wenlong.lwl wrote:
> >> > > > > > > Hi, all, I would like to start a discussion about an
> improvement
> >> > on
> >> > > > name
> >> > > > > > > and structure of job vertex name, mainly to improve
> experience of
> >> > > > > > debugging
> >> > > > > > > and analyzing sql job at runtime.
> >> > > > > > >
> >> > > > > > > the main proposed changes including:
> >> > > > > > > 1. separate description and name for operator, so that we
> can
> >> > have
> >> > > > > > detailed
> >> > > > > > > info at description and shorter name, which could be more
> >> > friendly
> >> > > > for
> >> > > > > > > external systems like logging/metrics without losing useful
> >> > > > information.
> >> > > > > > > 2. introduce a tree-mode vertex description which can make
> the
> >> > > > > > description
> >> > > > > > > more readable and easier to understand
> >> > > > > > > 3. clean up and improve description for sql operator
> >> > > > > > >
> >> > > > > > > here is an example with the changes for a sql job:
> >> > > > > > >
> >> > > > > > > vertex name:
> >> > > > > > > GlobalGroupAggregate[52] -> (Calc[53] ->
> NotNullEnforcer[54] ->
> >> > > Sink:
> >> > > > > > > tb_ads_dwi_pub_hbd_spm_dtr_002_003[54], Calc[55] ->
> >> > > > NotNullEnforcer[56]
> >> > > > > > ->
> >> > > > > > > Sink: tb_ads_dwi_pub_hbd_spm_dtr_002_004[56])
> >> > > > > > > vertex description:
> >> > > > > > > [52]:GlobalGroupAggregate(groupBy=[stat_date, spm_url_ab,
> >> > client],
> >> > > > > > > select=[stat_date, spm_url_ab, client, COUNT(count1$0) AS
> >> > > > > > > clk_cnt_app_mtr_001, COUNT(distinct$0 count$1) AS
> >> > > clk_uv_app_mtr_001,
> >> > > > > > > COUNT(count1$2) AS clk_cnt_app_mtr_002, COUNT(distinct$0
> count$3)
> >> > > AS
> >> > > > > > > clk_uv_app_mtr_002, COUNT(count1$4) AS clk_cnt_app_mtr_003,
> >> > > > > > > COUNT(distinct$0 count$5) AS clk_uv_app_mtr_003]) :-
> >> > > > > > > [53]:Calc(select=[CASE((client <> ''), CONCAT_WS('\u0004',
> >> > > > > > > CONCAT(SUBSTRING(MD5(CONCAT(spm_url_ab, '12345')), 1, 4),
> >> > ':md5'),
> >> > > > > > > CONCAT(spm_url_ab, ':spmab'), '12345:app', CONCAT(client,
> >> > > ':client'),
> >> > > > > > > CONCAT('ddd:', stat_date)), null:VARCHAR(2147483647)) AS
> rowkey,
> >> > > > > > > clk_cnt_app_mtr_001 AS clk_cnt_app_dtr_001,
> clk_uv_app_mtr_001 AS
> >> > > > > > > clk_uv_app_dtr_001, clk_cnt_app_mtr_002 AS
> clk_cnt_app_dtr_002,
> >> > > > > > > clk_uv_app_mtr_002 AS clk_uv_app_dtr_002,
> clk_cnt_app_mtr_003 AS
> >> > > > > > > clk_cnt_app_dtr_003, clk_uv_app_mtr_003 AS
> clk_uv_app_dtr_003]) :
> >> > > +-
> >> > > > > > > [54]:NotNullEnforcer(fields=[rowkey]) : +-
> >> > > > > > >
> >> > > > > >
> >> > > >
> >> > >
> >> >
> [54]:Sink(table=[default_catalog.default_database.tb_ads_dwi_pub_hbd_spm_dtr_002_003],
> >> > > > > > > fields=[rowkey, clk_cnt_app_dtr_001, clk_uv_app_dtr_001,
> >> > > > > > > clk_cnt_app_dtr_002, clk_uv_app_dtr_002,
> clk_cnt_app_dtr_003,
> >> > > > > > > clk_uv_app_dtr_003]) +- [55]:Calc(select=[CASE((client <>
> ''),
> >> > > > > > > CONCAT_WS('\u0004', CONCAT(SUBSTRING(MD5(CONCAT(spm_url_ab,
> >> > > > '12345')), 1,
> >> > > > > > > 4), ':md5'), 

Re: [VOTE] FLIP-187: Adaptive Batch Job Scheduler

[Guowei Ma]

Thanks for your excellent FLIP!
+1 binding

Lijie Wang 于2021年11月8日 周一下午2:53写道：

> Hi all,
>
> I would like to start the vote for FLIP-187[1], which proposes to introduce
> a new scheduler to Flink: adaptive batch job scheduler. The new scheduler
> can automatically decide parallelisms of job vertices for batch jobs,
> according to the size of data volume each vertex needs to process. This
> FLIP was discussed in [2].
>
> The vote will last at least 72 hours (Nov 11th 12:00 GMT) unless there is
> an objection or insufficient votes.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-187%3A+Adaptive+Batch+Job+Scheduler
>
> [2]
>
> https://lists.apache.org/thread.html/r32bd8f521daf4446fb70366bfabc4603bf3f56831c04e65bee6709aa%40%3Cdev.flink.apache.org%3E
>
> Best,
>
> Lijie
>
-- 
Best,
Guowei

Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction

[Guowei Ma]

Hi,

Very thanks Fabian for drafting this FLIP! It looks very good to me. I see
currently most of us agree with option 2, but I personally feel that option
3 may be better :-)
I have some small concerns for option 2

1. Developers understand that the cost is relatively high.
The merging of small files is very important, but from the perspective of
sink as a whole, this requirement is only a special case of sink. We expose
this requirement directly in the Sink API, which means that all developers
may need to understand this "special case". From my personal point of view,
for a developer who has no compaction requirements, seeing this API may
make him feel more complicated. In addition, from a higher level, the first
level of the Sink API should have a relatively general abstraction; the
second level might be different abstractions according to specific types.

2. Repeated implementation
The newly introduced `aggregate` can set the parallelism, thus perhaps
`setUid`, `slotSharingGroup (including resources)`, and `maxParallelism`
also need to be supported? If they are supported, additional workloads are
required, and at the same time I feel that these workloads are unnecessary;
if not, it will also increase the developers’ understanding cost: for
example, why these operators can not set these attributes? Also for another
example, `SinkWriter` has introduced many existing (or repeated)
implementations in the DataStream API in order to support ‘State’, ‘Timer’,
‘asynchronization’, ‘subindex’, etc.
In addition, if a new feature is introduced on the DataStream in the
future, I think it may be necessary to consider the special operators of
the sinks separately, which may also be a burden.

3. Might Blocking checkpoint
For option 2, we would need to actually merge files in the committer.
However, this might cause some problems in whether compaction blocks
checkpoint: suppose now our policy is to merge all the files produced in 10
checkpoints, if we allow users to construct the topology, they could merge
the file asynchronously in a dedicated executor, the merge could span
multiple checkpoints (as long as not exceed 10), and then we emit the files
to be renamed to the committer. But if we merge in a committer, it has to
be done before the next checkpoint since we do not support asynchronous
commits now. But with option3, this can be done at present.


In addition, The FLIP mentioned that option 3 has a certain burden on
developers from the perspective of stream batch unification. Could you
enlighten me a little about what the burden is? I think that the current
APIs of DataStream are all stream-batch unification, and the topology in
Opiont3 is also done with `DataStream`, so I understand that developers
will not have any additional burdens. Or what is missing

Best,
Guowei


On Thu, Nov 4, 2021 at 11:16 AM Jingsong Li  wrote:

> Hi Fabian,
>
> Thanks for drafting the FLIP!
>
> ## Few thoughts of user requirements
>
> 1.compact files from multiple checkpoints
>
> This is what users need very much.
>
> 2.The compaction block the checkpointing
>
> - Some scenarios are required. For example, the user expects the
> output data to be consistent with the input data. If the checkpoint
> succeeds, it needs to see how much data is output. Otherwise, the user
> restarts a job to consume the same source offset, and he may lose
> data.
> - But I think in most cases, users are concerned about this, and we
> can delay the data to be visible.
>
> 3.The end-to-end latency of data
>
> This also depends on the situation.
> - Some user delays are very important. We'd better compact the data at
> the current checkpoint, even if it affects the checkpoint delay.
> - Some users think that the delay doesn't matter (the delay is at the
> minute level). As long as you compact the file, it won't bring me
> trouble with small files.
>
> So I think flexibility is important.
>
> ## Few thoughts on the option 2
>
> Option1 and option2 seem to be just the difference between the
> aggregator in the middle, whether it is a separate operator or a
> coordinator.
>
> I would be prefer to option 2.
>
> If I understand correctly, the whole process should be as follows:
>
> 1.SinkWriter ->
> 2.Aggregator(I think 1 parallelism is OK, why is it multiple parallelism?)
> ->
> 3.LocalCommitter (Do compaction works) ->
> 4.GlobalCommitter
>
> The Aggregator can control single checkpoint compaction or cross
> checkpoint compaction. Controlling block or not block the current
> checkpoint. Controlling the end-to-end latency of data is how many
> checkpoints to wait for.
>
> Best,
> Jingsong
>
> On Wed, Nov 3, 2021 at 11:01 PM Fabian Paul 
> wrote:
> >
> >
> > Hi David and Till,
> >
> > Thanks for your great feedback. One definitely confusing point in the
> FLIP is who is doing the actual compaction. The compaction will not be done
> > by the CommittableAggregator operator but the committers so it should
> also not affect the checkpointing duration or have a 

Re: [Discuss] Planning Flink 1.15

[Guowei Ma]

Thanks for volunteering as our release managers Till, Yun and Joe!

+1 for feature freeze on 2.6 ,which is more convenient for us to plan 1.15
related work.

Best,
Guowei


On Thu, Oct 28, 2021 at 12:01 PM Xintong Song  wrote:

> Thanks for kicking this off, Joe.
>
> +1 for Till, Yun and Joe as the release managers.
>
> Also +1 for feature freeze on 2.6, which would make it easier for folks
> from China to dedicate to the release testing.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Wed, Oct 27, 2021 at 11:04 PM Yun Gao 
> wrote:
>
> > Hi all,
> >
> > Very thanks @Joe for starting the discussion! And it is really honorable
> > to be the release manager of 1.15~
> >
> > And for the releasing date, there might be some concerns since Jan. 17th
> > is closed to the Spring Festival Holiday (1.31 to 2.6) and many people
> > might also start the holiday further in advance, it might have some
> > influence to our releasing test / stabilization period. Thus perhaps
> > another option might be we slightly extend the code freeze date to bypass
> > the holiday till Feb. 6th ? WDYT ?
> >
> > Thank you~
> >
> > Best,
> > Yun
> >
> >
> >
> >
> > --
> > From:Israel Ekpo 
> > Send Time:2021 Oct. 26 (Tue.) 08:09
> > To:dev 
> > Subject:Re: [Discuss] Planning Flink 1.15
> >
> > Thanks for the update, Joe.
> >
> > Looking forward to this release.
> >
> > On Mon, Oct 25, 2021 at 11:13 AM Johannes Moser 
> wrote:
> >
> > > Hi all,
> > >
> > > As people have already started working on their 1.15.
> > > contribution, we'd like to start the discussion for
> > > the release setup.
> > >
> > > - Release managers: As a team of three seems to have
> > > worked perfectly fine, we'd like to suggest Till, Yun Gao
> > > & Joe as the release managers for 1.15.
> > >
> > > - Timeline: 1.14 was released at the end of September and
> > > aiming for a 4 months release cycle including one months
> > > of stabilisation would lead to a feature freeze date at the
> > > end of December, which would make the European holiday
> > > season a bit stressful. One option would have been to aim for early
> > > December, but we decided to go for the 17th of January.
> > > Such that we also have some buffer before the Chinese new
> > > year.
> > >
> > > - Bi-weekly sync: We'd also like to setup a bi-weekly sync again
> > > starting from the 9th of November at 9am CET/4pm CST.
> > >
> > > - Collecting features: As last time it would be helpful to have
> > > a rough overview of the efforts that will likely be included in
> > > this release. We have created a wiki page [1] for collecting such
> > > information. We'd like to kindly ask all committers to fill in the
> > > page with features that they intend to work on.
> > >
> > > Just copy pasting what we included into the planning email
> > > for 1.14, because it still applies:
> > >
> > > - Stability of master: This has been an issue during the 1.13 & 1.14
> > > feature freeze phase and it is still going on. We encourage every
> > > committer to not merge PRs through the Github button, but do this
> > > manually, with caution for the commits merged after the CI being
> > > triggered. It would be appreciated to always build the project before
> > > merging to master.
> > >
> > > - Documentation: Please try to see documentation as an integrated
> > > part of the engineering process and don't push it to the feature
> > > freeze phase or even after. You might even think about going
> > > documentation first. We, as the Flink community, are adding great
> > > stuff, that is pushing the limits of streaming data processors, with
> > > every release. We should also make this stuff usable for our users by
> > > documenting it well.
> > >
> > > - Promotion of 1.15: What applies to documentation also applies
> > > to all the activity around the release. We encourage every contributor
> > > to also think about, plan and prepare activities like blog posts and
> > talk,
> > > that will promote and spread the release once it is done.
> > >
> > > Please let us know what you think.
> > >
> > > Thank you~
> > > Till, Yun Gao & Joe
> > >
> > > [1] https://cwiki.apache.org/confluence/display/FLINK/1.15+Release
> > >
> >
>

Re: [VOTE] FLIP-176: Unified Iteration to Support Algorithms (Flink ML)

[Guowei Ma]

+1 (binding)

Best,
Guowei


On Thu, Oct 21, 2021 at 3:58 PM Yun Gao 
wrote:

>
> Hi all,
>
> We would like to start the vote for FLIP-176: Unified Iteration to Support
> Algorithms (Flink ML) [1].
> This FLIP was discussed in this thread [2][3]. The FLIP-176 targets at
> implementing the iteration
> API in flink-ml to support the implementation of the algorithms.
>
> The vote will be open for at least 72 hours till 26th Oct morning,
> including the weekend. Very thanks!
>
> Best,
> Yun
>
> [1] https://cwiki.apache.org/confluence/x/hAEBCw
> [2]
> https://lists.apache.org/thread.html/r72e87a71b14baac3d7d16268346f5fc7c65f1de989e00b4ab2aab9ab%40%3Cdev.flink.apache.org%3E
> [3]
> https://lists.apache.org/thread.html/r63914a616de05a91dbe8e1a3208eb2b7c7c840c5c366bbd224483754%40%3Cdev.flink.apache.org%3E

Re: [NOTICE] CiBot improvements

[Guowei Ma]

Thanks for your effort!

Best,
Guowei


On Mon, Oct 11, 2021 at 9:26 PM Stephan Ewen  wrote:

> Great initiative, thanks for doing this!
>
> On Mon, Oct 11, 2021 at 10:52 AM Till Rohrmann 
> wrote:
>
> > Thanks a lot for this effort Chesnay! The improvements sound really good.
> >
> > Cheers,
> > Till
> >
> > On Mon, Oct 11, 2021 at 8:46 AM David Morávek  wrote:
> >
> > > Nice! Thanks for the effort Chesnay, this is really a huge step
> forward!
> > >
> > > Best,
> > > D.
> > >
> > > On Mon, Oct 11, 2021 at 6:02 AM Xintong Song 
> > > wrote:
> > >
> > > > Thanks for the effort, @Chesnay. This is super helpful.
> > > >
> > > > @Jing,
> > > > Every push to the PR branch should automatically trigger an entire
> new
> > > > build. `@flinkbot run azure` should only be used when you want to
> > re-run
> > > > the failed stages without changing the PR. E.g., when running into
> > known
> > > > unstable cases that are unrelated to the PR.
> > > >
> > > > Thank you~
> > > >
> > > > Xintong Song
> > > >
> > > >
> > > >
> > > > On Mon, Oct 11, 2021 at 11:45 AM JING ZHANG 
> > > wrote:
> > > >
> > > > > Hi Chesnay,
> > > > > Thanks for the effort. It is a very useful improvement.
> > > > > I have a minor question. Please forgive me if the question is too
> > > naive.
> > > > > Since '@flinkbot run azure' now behaves like "Rerun failed jobs",
> is
> > > > there
> > > > > any way to trigger an entirely new build? Because some times I'm
> not
> > > sure
> > > > > the passed cases in the last build could still success in the new
> > build
> > > > > because of introduced updates in new commit.
> > > > >
> > > > > Best,
> > > > > JING ZHANG
> > > > >
> > > > >
> > > > > Yangze Guo  于2021年10月11日周一 上午10:31写道：
> > > > >
> > > > > > Thanks for that great job, Chesnay! "Rerun failed jobs" will
> help a
> > > > lot.
> > > > > >
> > > > > > Best,
> > > > > > Yangze Guo
> > > > > >
> > > > > > On Sun, Oct 10, 2021 at 4:56 PM Chesnay Schepler <
> > ches...@apache.org
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > I made a number of changes to the CiBot over the weekend.
> > > > > > >
> > > > > > > - '@flinkbot run azure' previously triggered an entirely new
> > build
> > > > > based
> > > > > > > on the last completed one. It now instead retries the last
> > > completed
> > > > > > > build, only running the jobs that actually failed. It basically
> > > > behaves
> > > > > > > like the "Rerun failed jobs" button in the Azure UI.
> > > > > > > - various optimizations to increase responsiveness (primarily
> by
> > > > doing
> > > > > > > significantly less unnecessary work / requests to GH)
> > > > > > > - removed TravisCI support (since we no longer support a
> release
> > > that
> > > > > > > used Travis)
> > > > > > >
> > > > > > > Please ping me if you spot anything weird.
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Create a public open GitHub org for Flink ecosystem projects.

[Guowei Ma]

Hi, Becket

Thank you very much for promoting this work.
After 1.14 is released, we have plans to open source Remote Shuffle Service
for Flink. If we have this, we can put it there.

Best,
Guowei


On Thu, Sep 23, 2021 at 4:07 PM Martijn Visser 
wrote:

> Hi all,
>
> Thanks Becket for the initiative. With the example projects mentioned, I
> agree that it makes a lot of sense to improve the ecosystem for projects
> that can be used to extend Flink.
>
> I would like to mention that Arvid and I have been working on a proposal
> with regards to the connectors and the connector ecosystem, which we want
> to share with the community probably next week or so.
>
> Best regards,
>
> Martijn
>
> On Thu, 23 Sept 2021 at 10:00, Becket Qin  wrote:
>
> > Thanks Leonard. Glad to know it helps.
> >
> > That said, I also want to mention that for connectors, another option is
> to
> > contribute them to Apache Bahir. I think letting the project stay in this
> > GitHub org for some time to ensure it is well maintained before
> > contributing that to an Apache project seems a good approach.
> >
> > Cheers,
> >
> > Jiangjie (Becket) Qin
> >
> > On Thu, Sep 23, 2021 at 3:26 PM Leonard Xu  wrote:
> >
> > > Thanks  Becket and PMC members for the effort.
> > >
> > > There’s one developer is willing to contribute MongoDB connector to
> > > community in user-zh mail list,
> > >  he has developed an internal version in their company, I think this
> > > project is pretty proper for such kind of contributions.
> > >
> > > Best,
> > > Leonard
> > > [1]
> > >
> >
> https://lists.apache.org/thread.html/r82c376fea440100f1cb3050026afa07cdc99b94454f608e05e4102f1%40%3Cuser-zh.flink.apache.org%3E
> > >
> > > > 在 2021年9月23日，14:59，Ingo Bürk  写道：
> > > >
> > > > Hi,
> > > >
> > > > thank you (and the PMC) for the initiative on such a community
> effort.
> > > > Are there already projects expected/known to move to such an
> > > organization?
> > > > I think it would make sense to have at least a couple projects lined
> up
> > > so
> > > > the org doesn't start out empty.
> > > >
> > > >
> > > > Best
> > > > Ingo
> > > >
> > > > On Thu, Sep 23, 2021 at 8:43 AM Becket Qin 
> > wrote:
> > > >
> > > >> Hi Flink devs,
> > > >>
> > > >> Recently we had some discussion in the Flink PMC about creating a
> > public
> > > >> open GitHub organization to host the code repo of some Flink
> ecosystem
> > > >> projects. Instead of the Flink PMC doing this, we found that it is
> > more
> > > >> suitable for someone in the Flink community to do this on their
> > personal
> > > >> behalf. So here I would love to see if people would be interested in
> > > coming
> > > >> together to help create and maintain this GitHub organization as a
> > > >> community effort.
> > > >>
> > > >> *** Motivation*
> > > >>
> > > >> Currently, usually an ecosystem project is hosted in a company's
> > GitHub
> > > >> repo. However, this does not always work well for those who want to
> > > >> collaborate on the projects..
> > > >>
> > > >>   1. Some employers may have concerns if their employees contribute
> > code
> > > >>   to another company's repo. Instead, they would rather fork and
> > develop
> > > >> in
> > > >>   their own repo. This results in split efforts instead of joint
> force
> > > to
> > > >>   develop the project.
> > > >>   2. Sometimes a company's policy disallows granting repo
> permissions
> > to
> > > >>   external contributors.
> > > >>   3. Sometimes a company does not have a GitHub repo and is also not
> > > >>   willing to open source a project in a personal repo.
> > > >>
> > > >> Therefore a public open GitHub organization would provide a
> *neutral*
> > > place
> > > >> helpful to facilitate the sharing and collaboration on the Flink
> > > ecosystem
> > > >> projects for developers in these situations.
> > > >>
> > > >> *** How does it work?*
> > > >>
> > > >>   1. The public ecosystem GitHub org would be created and maintained
> > by
> > > a
> > > >>   few volunteers.
> > > >>   2. The volunteers who maintain the org are only responsible for
> > > creating
> > > >>   and deleting the individual project code repositories upon the
> > > requests
> > > >>   from the project owners.
> > > >>   3. When someone wants to put a Flink ecosystem project in this
> > > >>   organization, a new GitHub repo will be created to host that
> > project.
> > > >>   4. The owners of each individual project will maintain the code
> repo
> > > of
> > > >>   that project, including merging PRs, granting commit permissions
> to
> > > >> other
> > > >>   contributors, publishing releases, etc.
> > > >>
> > > >> *Note that this open GitHub org is NOT affiliated with ASF or the
> > Apache
> > > >> Flink project, although the volunteers who maintain the org may be
> > Flink
> > > >> committers or PMC members.*
> > > >>
> > > >> *** What's next*
> > > >> If people find the public GitHub org for the ecosystem projects
> > useful,
> > > we
> > > >> will do the following:
> > > >>
> > > 

Re: [DISCUSS] FLIP-177: Extend Sink API

[Guowei Ma]

Hi Arvid

In fact, I think the way of combination is more flexible, especially in
terms of reuse. Once you make changes, you can reuse the last logic well.
For example, two AsyncIOA+AsyncIOB, you can easily reuse the previous logic
AsyncIOA+AsyncIOC,
For compatibility, I don't think there would be a big problem. For example,
for the old AsyncIOA+AsyncIOB, we can always convert to
AsyncIOA+AsyncIOB+AsyncIOC. When AsyncIOB does not have a state, simply
pass it through.

Hi Steffen
You are right, there are indeed such examples. However, I also mentioned in
my previous email that we could extend the AsyncIO capability 
XXXFuture.fail(Excelption) so that AsyncIO can re-try(put the failed
element back to the internal queue again), which will also solve the
problem you mentioned. BTW if we implement `AsyncSink` separately, there
may also be Order/UnOrder issues, which is already resolved by the `AsyncIO`

In general, I think AsyncIO and AsyncSink are very similar. There might be
some duplicated work.

Best,
Guowei


On Fri, Jul 30, 2021 at 8:16 PM Hausmann, Steffen 
wrote:

> Hey Guowei,
>
> there is one additional aspect I want to highlight that is relevant for
> the types of destinations we had in mind when designing the AsyncSink.
>
> I'll again use Kinesis as an example, but the same argument applies to
> other destinations. We are using the PutRecords API to persist up to 500
> events with a single API call to reduce the overhead compared to using
> individual calls per event. But not all of the 500 events may be persisted
> successfully, eg, a single event fails to be persisted due to server side
> throttling. With the MailboxExecutor based implementation, we can just add
> this event back to the internal queue. The event will then be retied with
> the next batch of 500 events.
>
> In my understanding, that's not possible with the AsyncIO based approach.
> During a retry, we can only retry the failed events of the original batch
> of events, which means we would need to send a single event with a separate
> PutRecords call. Depending how often that happens, this could add up.
>
> Does that make sense?
>
> Cheers, Steffen
>
>
> On 30.07.21, 05:51, "Guowei Ma"  wrote:
>
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you can confirm the sender and
> know the content is safe.
>
>
>
> Hi, Arvid & Piotr
> Sorry for the late reply.
> 1. Thank you all very much for your patience and explanation.
> Recently, I
> have also studied the related code of 'MailBox', which may not be as
> serious as I thought, after all, it is very similar to Java's
> `Executor`;
> 2. Whether to use AsyncIO or MailBox to implement Kinesis connector is
> more
> up to the contributor to decide (after all, `Mailbox` has decided to be
> exposed :-) ). It’s just that I personally prefer to combine some
> simple
> functions to complete a more advanced function.
> Best,
> Guowei
>
>
> On Sat, Jul 24, 2021 at 3:38 PM Arvid Heise  wrote:
>
> > Just to reiterate on Piotr's point: MailboxExecutor is pretty much an
> > Executor [1] with named lambdas, except for the name MailboxExecutor
> > nothing is hinting at a specific threading model.
> >
> > Currently, we expose it on StreamOperator API. Afaik the idea is to
> make
> > the StreamOperator internal and beef up ProcessFunction but for
> several use
> > cases (e.g., AsyncIO), we actually need to expose the executor
> anyways.
> >
> > We could rename MailboxExecutor to avoid exposing the name of the
> threading
> > model. For example, we could rename it to TaskThreadExecutor (but
> that's
> > pretty much the same), to CooperativeExecutor (again implies
> Mailbox), to
> > o.a.f.Executor, to DeferredExecutor... Ideas are welcome.
> >
> > We could also simply use Java's Executor interface, however, when
> working
> > with that interface, I found that the missing context of async
> executed
> > lambdas made debugging much much harder. So that's why I designed
> > MailboxExecutor to force the user to give some debug string to each
> > invokation. In the sink context, we could, however, use an adaptor
> from
> > MailboxExecutor to Java's Executor and simply bind the sink name to
> the
> > invokations.
> >
> > [1]
> >
> >
> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/Executor.html
> >
> > On Fri, Jul 23, 2021 at 5:36 PM Piotr Nowojski  >
> > wrote:
> >
> > > Hi,
> 

Re: [VOTE] FLIP-177: Extend Sink API

[Guowei Ma]

+1(binding)
Best,
Guowei


On Mon, Aug 2, 2021 at 4:04 PM Danny Cranmer 
wrote:

> +1 (binding)
>
> On Mon, Aug 2, 2021 at 12:42 AM Thomas Weise  wrote:
>
> > +1 (binding)
> >
> >
> > On Fri, Jul 30, 2021 at 5:05 AM Arvid Heise  wrote:
> >
> > > Hi all,
> > >
> > > I'd like to start a vote on FLIP-177: Extend Sink API [1] which
> provides
> > > small extensions to the Sink API introduced through FLIP-143.
> > > The vote will be open for at least 72 hours unless there is an
> objection
> > or
> > > not enough votes.
> > >
> > > Note that the FLIP was larger initially and I cut down all
> > > advanced/breaking changes.
> > >
> > > Best,
> > >
> > > Arvid
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-177%3A+Extend+Sink+API
> > >
> >
>

Re: [DISCUSS] FLIP-177: Extend Sink API

[Guowei Ma]

Hi, Arvid & Piotr
Sorry for the late reply.
1. Thank you all very much for your patience and explanation. Recently, I
have also studied the related code of 'MailBox', which may not be as
serious as I thought, after all, it is very similar to Java's `Executor`;
2. Whether to use AsyncIO or MailBox to implement Kinesis connector is more
up to the contributor to decide (after all, `Mailbox` has decided to be
exposed :-) ). It’s just that I personally prefer to combine some simple
functions to complete a more advanced function.
Best,
Guowei


On Sat, Jul 24, 2021 at 3:38 PM Arvid Heise  wrote:

> Just to reiterate on Piotr's point: MailboxExecutor is pretty much an
> Executor [1] with named lambdas, except for the name MailboxExecutor
> nothing is hinting at a specific threading model.
>
> Currently, we expose it on StreamOperator API. Afaik the idea is to make
> the StreamOperator internal and beef up ProcessFunction but for several use
> cases (e.g., AsyncIO), we actually need to expose the executor anyways.
>
> We could rename MailboxExecutor to avoid exposing the name of the threading
> model. For example, we could rename it to TaskThreadExecutor (but that's
> pretty much the same), to CooperativeExecutor (again implies Mailbox), to
> o.a.f.Executor, to DeferredExecutor... Ideas are welcome.
>
> We could also simply use Java's Executor interface, however, when working
> with that interface, I found that the missing context of async executed
> lambdas made debugging much much harder. So that's why I designed
> MailboxExecutor to force the user to give some debug string to each
> invokation. In the sink context, we could, however, use an adaptor from
> MailboxExecutor to Java's Executor and simply bind the sink name to the
> invokations.
>
> [1]
>
> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/Executor.html
>
> On Fri, Jul 23, 2021 at 5:36 PM Piotr Nowojski 
> wrote:
>
> > Hi,
> >
> > Regarding the question whether to expose the MailboxExecutor or not:
> > 1. We have plans on exposing it in the ProcessFunction (in short we want
> to
> > make StreamOperator API private/internal only, and move all of it's extra
> > functionality in one way or another to the ProcessFunction). I don't
> > remember and I'm not sure if *Dawid* had a different idea about this (do
> > not expose Mailbox but wrap it somehow?)
> > 2. If we provide a thin wrapper around MailboxExecutor, I'm not sure how
> > helpful it will be for keeping backward compatibility in the future.
> > `MailboxExecutor` is already a very generic interface that doesn't expose
> > much about the current threading model. Note that the previous threading
> > model (multi threaded with checkpoint lock), should be easy to implement
> > using the `MailboxExecutor` interface (use a direct executor that
> acquires
> > checkpoint lock).
> >
> > Having said that, I haven't spent too much time thinking about whether
> it's
> > better to enrich AsyncIO or provide the AsyncSink. If we can just as
> > efficiently provide the same functionality using the existing/enhanced
> > AsyncIO API, that may be a good idea if it indeed reduces our
> > maintenance costs.
> >
> > Piotrek
> >
> > pt., 23 lip 2021 o 12:55 Guowei Ma  napisał(a):
> >
> > > Hi, Arvid
> > >
> > > >>>The main question here is what do you think is the harm of exposing
> > > Mailbox? Is it the complexity or the maintenance overhead?
> > >
> > > I think that exposing the internal threading model might be risky. In
> > case
> > > the threading model changes, it will affect the user's api and bring
> the
> > > burden of internal modification. (Of course, you may have more say in
> how
> > > the MailBox model will develop in the future) Therefore, I think that
> if
> > an
> > > alternative solution can be found, these possible risks will be
> avoided:
> > > for example, through AsyncIO.
> > >
> > > >>>>AsyncIO has no user state, so we would be quite limited in
> > implementing
> > > at-least-once sinks especially when it comes to retries. Chaining two
> > > AsyncIO would make it even harder to reason about the built-in state.
> We
> > > would also give up any chance to implement exactly once async sinks
> (even
> > > though I'm not sure if it's possible at all).
> > >
> > > 1. Why would we need to use the state when retrying(maybe I miss
> > > something)? If a batch of asynchronous requests fails, I think it is
> > enough
> > > to retry directly in the callback. Or extend AsyncIO to give 

Re: [DISCUSS] FLIP-177: Extend Sink API

[Guowei Ma]

ately (of course, connector devs could do the same cast but then use
> an internal class and need to expect breaking changes).
>
> For your question:
>
>> 2. By the way, I have a little question. Why not directly reduce the
>> queue size to control the in-flight query, for example, according to your
>> example,
>> Is a queue size such as 150 enough? In fact, there are caches in many
>> places in the entire topology, such as buffers on the network stack.
>>
> It's a bit different. Let's take kinesis as an example. The sink collects
> 500 elements and puts them in a single request (500 is max number of
> records per batch request). Now it sends the request out. At the same time,
> the next 500 elements are collected. So the in-flight query size refers to
> the number of parallel requets (500 elements each).
> If we first batch 500*numberOfInFlightRequests, and then send out all
> numberOfInFlightRequests at the same time, we get worse latency.
>
> On Thu, Jul 22, 2021 at 12:11 PM Guowei Ma  wrote:
>
>> Hi, Steffen
>>
>> Thank you for your detailed explanation.
>>
>> >>>But whether a sink is overloaded not only depends on the queue size.
>> It also depends on the number of in-flight async requests
>>
>> 1. How about chaining two AsyncIOs? One is for controlling the size of
>> the buffer elements; The other is for controlling the in-flight async
>> requests. I think this way might do it and would not need to expose the
>> MailboxExecutor.
>> 2. By the way, I have a little question. Why not directly reduce the
>> queue size to control the in-flight query, for example, according to your
>> example,
>> Is a queue size such as 150 enough? In fact, there are caches in many
>> places in the entire topology, such as buffers on the network stack.
>>
>> >>> I’m not sure whether I’m understanding who you are referring to by
>> user.
>> Personally I mean the sink developer.
>>
>>
>> Best,
>> Guowei
>>
>>
>> On Thu, Jul 22, 2021 at 4:40 PM Hausmann, Steffen
>>  wrote:
>>
>>> Hey,
>>>
>>> We are using the `MailboxExecutor` to block calls to `write` in case the
>>> sink is somehow overloaded. Overloaded basically means that the sink cannot
>>> persist messages quickly enough into the respective destination.
>>>
>>> But whether a sink is overloaded not only depends on the queue size. It
>>> also depends on the number of in-flight async requests that must not grow
>>> too large either [1, 2]. We also need to support use cases where the
>>> destination can only ingest x messages per second or a total throughput of
>>> y per second. We are also planning to support time outs so that data is
>>> persisted into the destination at least every n seconds by means of the
>>> `ProcessingTimeService`. The batching configuration will be part of the
>>> constructor, which has only been indicated in the current prototype but is
>>> not implemented, yet [3].
>>>
>>> I’m not sure whether I’m understanding who you are referring to by user.
>>> People who are using a concrete sink, eg, to send messages into a Kinesis
>>> stream, will not be exposed to the `MailboxExecutor`. They are just using
>>> the sink and pass in the batching configuration from above [4]. The
>>> `MailboxExecutor` and `ProcessingTimeService` are only relevant for sink
>>> authors who want to create support for a new destination. I would expect
>>> that there are only few experts who are adding support for new
>>> destinations, who are capable to understand and use the advanced constructs
>>> properly.
>>>
>>> Hope that helps to clarify our thinking.
>>>
>>> Cheers, Steffen
>>>
>>>
>>> [1]
>>> https://github.com/sthm/flink/blob/51614dc9371d6e352db768a404ba3cafddad08f0/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/writer/AsyncSinkWriter.java#L118
>>> [2]
>>> https://github.com/sthm/flink/blob/51614dc9371d6e352db768a404ba3cafddad08f0/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/writer/AsyncSinkWriter.java#L155
>>> [3]
>>> https://github.com/sthm/flink/blob/51614dc9371d6e352db768a404ba3cafddad08f0/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/writer/AsyncSinkWriter.java#L43-L49
>>> [4]
>>> https://github.com/sthm/flink/blob/51614dc9371d6e352db768a404ba3cafddad08f0/flink-connectors/flink-connector-kinesis-171/src/main/java/software/amazon/fli

Re: [DISCUSS] FLIP-177: Extend Sink API

[Guowei Ma]

 implementation detail by
> the application developer. Of course, advanced users may fall in both
> categories, so the distinction does not always hold.
>
> Nevertheless, there is some overlap between both approaches and it's
> important to think if the added complexity warrants the benefit. It would
> be interesting to hear how other devs see that.
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink
> [2]
> https://github.com/sthm/flink/blob/51614dc9371d6e352db768a404ba3cafddad08f0/flink-connectors/flink-connector-kinesis-171/src/main/java/software/amazon/flink/connectors/AmazonKinesisDataStreamSink.java
>
> On Tue, Jul 20, 2021 at 11:11 AM Guowei Ma  guowei@gmail.com>> wrote:
> Hi, Avrid
> Thank you Avrid for perfecting Sink through this FLIP. I have two little
> questions
>
> 1. What do you think of us directly providing an interface as follows? In
> this way, there may be no need to expose the Mailbox to the user. We can
> implement an `AsyncSinkWriterOperator` to control the length of the queue.
> If it is too long, do not call SinkWriter::write.
> public interface AsyncSinkWriter
> extends SinkWriter>, CommT,
> WriterStateT> {  please ignore the name of Tuple2 and XXXFuture at
> first.
> int getQueueLimit();
> }
>
> 2. Another question is: If users don't care about exactly once and the
> unification of stream and batch, how about letting users use
> `AsyncFunction` directly? I don’t have an answer either. I want to hear
> your suggestions.
>
> Best,
> Guowei
>
>
> On Mon, Jul 19, 2021 at 3:38 PM Arvid Heise  ar...@apache.org>> wrote:
>
> > Dear devs,
> >
> > today I'd like to start the discussion on the Sink API. I have drafted a
> > FLIP [1] with an accompanying PR [2].
> >
> > This FLIP is a bit special as it's actually a few smaller Amend-FLIPs in
> > one. In this discussion, we should decide on the scope and cut out too
> > invasive steps if we can't reach an agreement.
> >
> > Step 1 is to add a few more pieces of information to context objects.
> > That's non-breaking and needed for the async communication pattern in
> > FLIP-171 [3]. While we need to add a new Public API (MailboxExecutor), I
> > think that this should entail the least discussions.
> >
> > Step 2 is to also offer the same context information to committers. Here
> we
> > can offer some compatibility methods to not break existing sinks. The
> main
> > use case would be some async exactly-once sink but I'm not sure if we
> would
> > use async communication patterns at all here (or simply wait for all
> async
> > requests to finish in a sync way). It may also help with async cleanup
> > tasks though.
> >
> > While drafting Step 2, I noticed the big entanglement of the current API.
> > To figure out if there is a committer during the stream graph creation,
> we
> > actually need to create a committer which can have unforeseen
> consequences.
> > Thus, I spiked if we can disentangle the interface and have separate
> > interfaces for the different use cases. The resulting step 3 would be a
> > completely breaking change and thus is probably controversial. However,
> I'd
> > also see the disentanglement as a way to prepare to make Sinks more
> > expressive (write and commit coordinator) without completely overloading
> > the main interface.
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-177%3A+Extend+Sink+API
> > [2] https://github.com/apache/flink/pull/16399
> > [3]
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink
> >
>
>
>
> Amazon Web Services EMEA SARL
> 38 avenue John F. Kennedy, L-1855 Luxembourg
> Sitz der Gesellschaft: L-1855 Luxemburg
> eingetragen im Luxemburgischen Handelsregister unter R.C.S. B186284
>
> Amazon Web Services EMEA SARL, Niederlassung Deutschland
> Marcel-Breuer-Str. 12, D-80807 Muenchen
> Sitz der Zweigniederlassung: Muenchen
> eingetragen im Handelsregister des Amtsgerichts Muenchen unter HRB 242240,
> USt-ID DE317013094
>
>
>
>

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

[Guowei Ma]

and stop-with-savepoint --drain, the job
> would
> > > not be
> > > > >>>> expected to be restarted,
> > > > >>>>
> > > > >>>>> and for stop-with-savepoint the job would be expected restart
> > > later.
> > > > >>>>> Then for finish / stop-with-savepoint --drain, currently Flink
> > > would
> > > > >>>> emit MAX_WATERMARK before the
> > > > >>>>
> > > >
> > > > >>>>> EndOfPartition. Besides, as we have discussed before [1],
> > > endOfInput /
> > > > >>>> finish() should also only be called
> > > > >>>>
> > > > >>>>> for finish / stop-with-savepoint --drain. Thus currently they
> > > always
> > > > >>>> occurs at the same time. After the change,
> > > > >>>>
> > > > >>>>> we could emit MAX_WATERMARK before endOfInput event for the
> > finish
> > > /
> > > > >>>> stop-with-savepoint --drain cases.
> > > > >>>>
> > > > >>>>>> 2) StreamOperator.finish says to flush all buffered events.
> > Would
> > > a
> > > > >>>>>> WindowOperator close all windows and emit the results upon
> > calling
> > > > >>>>>> finish, for example?
> > > > >>>>> As discussed above for stop-with-savepoint, we would always
> keep
> > > the
> > > > >>>> window as is, and restore them after restart.
> > > > >>>>
> > > > >>>>> Then for the finish / stop-with-savepoint --drain, I think
> > perhaps
> > > it
> > > > >>>> depends on the Triggers. For
> > > > >>>>
> > > >
> > > > >>>>> event-time triggers / process time triggers, it would be
> > > reasonable to
> > > > >>>> flush all the windows since logically
> > > > >>>>
> > > >
> > > > >>>>> the time would always elapse and the window would always get
> > > triggered
> > > > >>>> in a logical future. But for triggers
> > > > >>>>
> > > >
> > > > >>>>> like CountTrigger, no matter how much time pass logically, the
> > > windows
> > > > >>>> would not trigger, thus we may not
> > > > >>>>
> > > > >>>>> flush these windows. If there are requirements we may provide
> > > > >>>> additional triggers.
> > > > >>>>
> > > >
> > > > >>>>>> It's a bit messy and I'm not sure if this should be
> strengthened
> > > out?
> > > > >>>> Each one of those has a little bit different semantic/meaning,
> > > > >>>>
> > > >
> > > > >>>>>> but at the same time they are very similar. For single input
> > > operators
> > > > >>>> `endInput()` and `finish()` are actually the very same thing.
> > > > >>>>
> > > >
> > > > >>>>> Currently MAX_WATERMARK / endInput / finish indeed always
> happen
> > > at the
> > > > >>>> same time, and for single input operators `endInput()` and
> > > `finish()`
> > > > >>>>
> > > >
> > > > >>>>> are indeed the same thing. During the last discussion we ever
> > > mentioned
> > > > >>>> this issue and at then we thought that we might deprecate
> > > `endInput()`
> > > > >>>>
> > > > >>>>> in the future, then we would only have endInput(int input) and
> > > > >>>> finish().
> > > > >>>>
> > > > >>>>> Best,
> > > > >>>>> Yun
> > > > >>>>> [1] https://issues.apache.org/jira/browse/FLINK-21132
> > > > >>>>>
> > --
> > > > >>>>> From:Piotr Nowojski
> > > > >>>>> Send Time:2021 Jul. 16 (Fri.) 13:48
> > > > >>>>> To:dev
> > > > >>>>> Cc:Yun Gao
> > > &g

Re: [DISCUSS] FLIP-177: Extend Sink API

[Guowei Ma]

Hi, Arvid

1. You are right that the core difference between the two options is
whether to expose `MailboxExecutor`. My personal preference
 is that the less internal implementations are exposed to users, the better.

2. `AsyncIO` does not support Batch output, but we can implement one based
on `AsyncIO`, such as `AsyncIOBatchSinkBase`. Then we might
provide a `SinkUtil.sinkTo(inputStream, AsyncIOBatchSinkBase)`, which could
help users to build a DataStream topology.
I personally think that if a function could be done externally (for
example, encapsulating an existing UDF to complete a function),
it is better than internal implementation. As for whether it is necessary
to distinguish between a Sink and a non-Sink, I personally don't think it
is particularly important.

```for example
InputStream = blalbalba;
SinkUtil.sinkTo(inputStrean, AsyncIOBasedBatchKensisSink);
```
3. For 2, the more general question is: Is `Sink` an operator or a
topology? If we think that `Sink` is a topology,
then whether `AsyncIO` can be used as `Sink` may not be a problem. We might
think that how we could provide an interface to
help the user to build the sink topology.

Best,
Guowei


On Wed, Jul 21, 2021 at 5:08 AM Arvid Heise  wrote:

> Hi Guowei,
>
> 1. your idea is quite similar to FLIP-171 [1]. The question is if we
> implement FLIP-171 based on public interfaces (which would require exposing
> MailboxExecutor as described here in FLIP-177) or if it's better to
> implement it internally and hide it.
> The first option is an abstract base class; your second option would be an
> abstract interface that has matching implementation internally (similarly
> to AsyncIO).
> There is an example for option 1 in [2]; I think the idea was to
> additionally specify the batch size and batch timeout in the ctor.
> @Hausmann,
> Steffen  knows more.
>
> 2. I guess your question is if current AsyncIO is not sufficient already if
> exactly-once is not needed? The answer is currently no, because AsyncIO is
> not doing any batching. The user could do batching before that but that's
> quite a bit of code. However, we should really think if AsyncIO should also
> support batching.
> I would also say that the scope of AsyncIO and AsyncSink is quite
> different: the first one is for application developers and the second one
> is for connector developers and would be deemed an implementation detail by
> the application developer. Of course, advanced users may fall in both
> categories, so the distinction does not always hold.
>
> Nevertheless, there is some overlap between both approaches and it's
> important to think if the added complexity warrants the benefit. It would
> be interesting to hear how other devs see that.
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink
> [2]
>
> https://github.com/sthm/flink/blob/51614dc9371d6e352db768a404ba3cafddad08f0/flink-connectors/flink-connector-kinesis-171/src/main/java/software/amazon/flink/connectors/AmazonKinesisDataStreamSink.java
>
> On Tue, Jul 20, 2021 at 11:11 AM Guowei Ma  wrote:
>
> > Hi, Avrid
> > Thank you Avrid for perfecting Sink through this FLIP. I have two little
> > questions
> >
> > 1. What do you think of us directly providing an interface as follows? In
> > this way, there may be no need to expose the Mailbox to the user. We can
> > implement an `AsyncSinkWriterOperator` to control the length of the
> queue.
> > If it is too long, do not call SinkWriter::write.
> > public interface AsyncSinkWriter
> > extends SinkWriter>, CommT,
> > WriterStateT> {  please ignore the name of Tuple2 and XXXFuture at
> > first.
> > int getQueueLimit();
> > }
> >
> > 2. Another question is: If users don't care about exactly once and the
> > unification of stream and batch, how about letting users use
> > `AsyncFunction` directly? I don’t have an answer either. I want to hear
> > your suggestions.
> >
> > Best,
> > Guowei
> >
> >
> > On Mon, Jul 19, 2021 at 3:38 PM Arvid Heise  wrote:
> >
> > > Dear devs,
> > >
> > > today I'd like to start the discussion on the Sink API. I have drafted
> a
> > > FLIP [1] with an accompanying PR [2].
> > >
> > > This FLIP is a bit special as it's actually a few smaller Amend-FLIPs
> in
> > > one. In this discussion, we should decide on the scope and cut out too
> > > invasive steps if we can't reach an agreement.
> > >
> > > Step 1 is to add a few more pieces of information to context objects.
> > > That's non-breaking and needed for the async communication pattern in
> > > FLIP-171 [3]. While we need to add a new Public API (MailboxExecutor),

Re: [DISCUSS] FLIP-177: Extend Sink API

[Guowei Ma]

Hi, Avrid
Thank you Avrid for perfecting Sink through this FLIP. I have two little
questions

1. What do you think of us directly providing an interface as follows? In
this way, there may be no need to expose the Mailbox to the user. We can
implement an `AsyncSinkWriterOperator` to control the length of the queue.
If it is too long, do not call SinkWriter::write.
public interface AsyncSinkWriter
extends SinkWriter>, CommT,
WriterStateT> {  please ignore the name of Tuple2 and XXXFuture at
first.
int getQueueLimit();
}

2. Another question is: If users don't care about exactly once and the
unification of stream and batch, how about letting users use
`AsyncFunction` directly? I don’t have an answer either. I want to hear
your suggestions.

Best,
Guowei


On Mon, Jul 19, 2021 at 3:38 PM Arvid Heise  wrote:

> Dear devs,
>
> today I'd like to start the discussion on the Sink API. I have drafted a
> FLIP [1] with an accompanying PR [2].
>
> This FLIP is a bit special as it's actually a few smaller Amend-FLIPs in
> one. In this discussion, we should decide on the scope and cut out too
> invasive steps if we can't reach an agreement.
>
> Step 1 is to add a few more pieces of information to context objects.
> That's non-breaking and needed for the async communication pattern in
> FLIP-171 [3]. While we need to add a new Public API (MailboxExecutor), I
> think that this should entail the least discussions.
>
> Step 2 is to also offer the same context information to committers. Here we
> can offer some compatibility methods to not break existing sinks. The main
> use case would be some async exactly-once sink but I'm not sure if we would
> use async communication patterns at all here (or simply wait for all async
> requests to finish in a sync way). It may also help with async cleanup
> tasks though.
>
> While drafting Step 2, I noticed the big entanglement of the current API.
> To figure out if there is a committer during the stream graph creation, we
> actually need to create a committer which can have unforeseen consequences.
> Thus, I spiked if we can disentangle the interface and have separate
> interfaces for the different use cases. The resulting step 3 would be a
> completely breaking change and thus is probably controversial. However, I'd
> also see the disentanglement as a way to prepare to make Sinks more
> expressive (write and commit coordinator) without completely overloading
> the main interface.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-177%3A+Extend+Sink+API
> [2] https://github.com/apache/flink/pull/16399
> [3]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink
>

Re: [DISCUSS] FLIP-171: Async Sink

[Guowei Ma]

Hi,
I'm very sorry to participate in this discussion so late. But I have a
little question. I understand the goal of this FLIP is to make `Writer`
support asynchronous. But my question is: why not let `Committer` support
asynchronization? If only `Writer` supports asynchronization, ExactlyOnce
is impossible. Since I am not an expert in Kineses, I would like everyone
to point out if I missed anything.
Best,
Guowei


On Sat, Jul 17, 2021 at 12:27 AM Till Rohrmann  wrote:

> Sure, thanks for the pointers.
>
> Cheers,
> Till
>
> On Fri, Jul 16, 2021 at 6:19 PM Hausmann, Steffen
> 
> wrote:
>
> > Hi Till,
> >
> > You are right, I’ve left out some implementation details, which have
> > actually changed a couple of time as part of the ongoing discussion. You
> > can find our current prototype here [1] and a sample implementation of
> the
> > KPL free Kinesis sink here [2].
> >
> > I plan to update the FLIP. But I think would it be make sense to wait
> > until the implementation has stabilized enough before we update the FLIP
> to
> > the final state.
> >
> > Does that make sense?
> >
> > Cheers, Steffen
> >
> > [1]
> >
> https://github.com/sthm/flink/tree/flip-171-177/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink
> > [2]
> >
> https://github.com/sthm/flink/blob/flip-171-177/flink-connectors/flink-connector-kinesis-171/src/main/java/software/amazon/flink/connectors/AmazonKinesisDataStreamSink.java
> >
> > From: Till Rohrmann 
> > Date: Friday, 16. July 2021 at 18:10
> > To: Piotr Nowojski 
> > Cc: Steffen Hausmann , "dev@flink.apache.org" <
> > dev@flink.apache.org>, Arvid Heise 
> > Subject: RE: [EXTERNAL] [DISCUSS] FLIP-171: Async Sink
> >
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> >
> >
> > Hi Steffen,
> >
> > I've taken another look at the FLIP and I stumbled across a couple of
> > inconsistencies. I think it is mainly because of the lacking code. For
> > example, it is not fully clear to me based on the current FLIP how we
> > ensure that there are no in-flight requests when
> > AsyncSinkWriter.snapshotState is called. Also the concrete implementation
> > of the AsyncSinkCommitter could be helpful for understanding how the
> > AsyncSinkWriter works in the end. Do you plan to update the FLIP
> > accordingly?
> >
> > Cheers,
> > Till
> >
> > On Wed, Jun 30, 2021 at 8:36 AM Piotr Nowojski  > > wrote:
> > Thanks for addressing this issue :)
> >
> > Best, Piotrek
> >
> > wt., 29 cze 2021 o 17:58 Hausmann, Steffen  > shau...@amazon.de>> napisał(a):
> > Hey Poitr,
> >
> > I've just adapted the FLIP and changed the signature for the
> > `submitRequestEntries` method:
> >
> > protected abstract void submitRequestEntries(List
> > requestEntries, ResultFuture requestResult);
> >
> > In addition, we are likely to use an AtomicLong to track the number of
> > outstanding requests, as you have proposed in 2b). I've already indicated
> > this in the FLIP, but it's not fully fleshed out. But as you have said,
> > that seems to be an implementation detail and the important part is the
> > change of the `submitRequestEntries` signature.
> >
> > Thanks for your feedback!
> >
> > Cheers, Steffen
> >
> >
> > On 25.06.21, 17:05, "Hausmann, Steffen" 
> wrote:
> >
> > CAUTION: This email originated from outside of the organization. Do
> > not click links or open attachments unless you can confirm the sender and
> > know the content is safe.
> >
> >
> >
> > Hi Piotr,
> >
> > I’m happy to take your guidance on this. I need to think through your
> > proposals and I’ll follow-up on Monday with some more context so that we
> > can close the discussion on these details. But for now, I’ll close the
> vote.
> >
> > Thanks, Steffen
> >
> > From: Piotr Nowojski  pnowoj...@apache.org
> > >>
> > Date: Friday, 25. June 2021 at 14:48
> > To: Till Rohrmann mailto:trohrm...@apache.org
> >>
> > Cc: Steffen Hausmann mailto:shau...@amazon.de>>,
> "
> > dev@flink.apache.org"  > >, Arvid Heise  > ar...@apache.org>>
> > Subject: RE: [EXTERNAL] [DISCUSS] FLIP-171: Async Sink
> >
> >
> > CAUTION: This email originated from outside of the organization. Do
> > not click links or open attachments unless you can confirm the sender and
> > know the content is safe.
> >
> >
> > Hey,
> >
> > I've just synced with Arvid about a couple of more remarks from my
> > side and he shared mine concerns.
> >
> > 1. I would very strongly recommend ditching `CompletableFuture `
> > from the  `protected abstract CompletableFuture
> > submitRequestEntries(List requestEntries);`  in favor of
> > something like
> > `org.apache.flink.streaming.api.functions.async.ResultFuture` interface.
> > `CompletableFuture` would partially make the threading model of the

Re: [VOTE] FLIP-184: Refine ShuffleMaster lifecycle management for pluggable shuffle service framework

[Guowei Ma]

+1(binding)

Best,
Guowei


On Fri, Jul 16, 2021 at 5:36 PM Yingjie Cao  wrote:

> Hi all,
>
> I'd like to start a vote on FLIP-184 [1] which was
> discussed in [2] [3]. The vote will be open for at least 72 hours
> until 7.21 unless there is an objection.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-184%3A+Refine+ShuffleMaster+lifecycle+management+for+pluggable+shuffle+service+framework
> [2]
>
> https://lists.apache.org/thread.html/radbbabfcfb6bec305ddf7aeefb983232f96b18ba013f0ae2ee500288%40%3Cdev.flink.apache.org%3E
> [3]
>
> https://lists.apache.org/thread.html/r93e3a72506f3e7ffd3c1ab860b5d1a21f8a47b059f2f2fdd05ca1d46%40%3Cdev.flink.apache.org%3E
>

Re: [DISCUSS] Lifecycle of ShuffleMaster and its Relationship with JobMaster and PartitionTracker

[Guowei Ma]

Hi,
Thank Yingjie for initiating this discussion. What I understand that the
document[1] actually mainly discusses two issues:
1. ShuffleMaster should be at the cluster level instead of the job level
2. ShuffleMaster should notify PartitionTracker that some data has been lost

Relatively speaking, I think the second problem is more serious. Because
for external or remote batch shuffling services, after the machine storing
shuffled data goes offline, PartitionTracker needs to be notified in time
to avoid repeated failures of the job. Therefore, it is hoped that when
shuffle data goes offline due to a machine error, ShuffleMaster can notify
the PartitionTracker in time. This requires ShuffleMaster to notify the
PartitionTracker with a handle such as JobShuffleContext.

So how to pass JobShuffleContext to ShuffleMaster? There are two options:
1. After ShuffleMaster is created, pass JobShuffleContext to ShuffleMaster,
such as ShuffleMaster::register(JobShuffleContext)
2. Pass ShuffleServiceFactory when creating ShuffleMaster, such as
ShuffleServiceFatroctry.createcreateShuffleMaster(JobShuffleContext).

Which one to choose is actually related to issue 1. Because if
ShuffleMaster is a cluster level, you should choose option 1, otherwise,
choose option 2. I think ShuffleMaster should be at the cluster level, for
example, because we don't need to maintain a ShuffleMaster for each job in
a SessionCluster; in addition, this ShuffleMaster should also be used by
RM's PartitionTracker in the future. Therefore, I think Option 1 is more
appropriate.

To sum up, we may give priority to solving problem 2, while taking into
account that ShuffleMaster should be a cluster-level component. Therefore,
I think we could ignore the date ShuffleMasterContext at the beginning; at
the same time, JobShuffleContext::getConfiguration/listPartitions should
not be needed.

[1]
https://docs.google.com/document/d/1_cHoapNbx_fJ7ZNraSqw4ZK1hMRiWWJDITuSZrdMDDs/edit

Best,
Guowei


On Fri, Jun 11, 2021 at 4:15 PM Yingjie Cao  wrote:

> Hi devs,
>
> I'd like to start a discussion about "Lifecycle of ShuffleMaster and its
> Relationship with JobMaster and PartitionTracker". (These are things we
> found when moving our external shuffle to the pluggable shuffle service
> framework.)
>
> The mail client may fail to display the right format. If so, please refer
> to this document:
>
> https://docs.google.com/document/d/1_cHoapNbx_fJ7ZNraSqw4ZK1hMRiWWJDITuSZrdMDDs/edit?usp=sharing
> .
> Lifecycle of ShuffleMaster
>
> Currently, the lifecycle of ShuffleMaster seems unclear.  The
> ShuffleServiceFactory is loaded for each JobMaster instance and then
> ShuffleServiceFactory#createShuffleMaster will be called to create a
> ShuffleMaser instance. However, the default NettyShuffleServiceFactory
> always returns the same ShuffleMaser singleton instance for all jobs. Based
> on the current implementation, the lifecycle of ShuffleMaster seems open
> and depends on the shuffle plugin themselves. However, at the TM side,
> the ShuffleEnvironment
> is a part of the TaskManagerServices whose lifecycle is decoupled with jobs
> which is more like a service. It means there is also an inconsistency
> between the TM side and the JM side.
>
> From my understanding, the reason for this is that the pluggable shuffle
> framework is still not completely finished yet, for example, there is a
> follow up umbrella ticket  FLINK-19551
>  for the pluggable
> shuffle service framework and in its subtasks, there is one task (
> FLINK-12731 ) which
> aims
> to load shuffle plugin with the PluginManager. I think this can solve the
> issue mentioned above. After the corresponding factory  loaded by the
> PluginManager, all ShuffleMaster instances can be stored in a map indexed
> by the corresponding factory class name  which can be shared by all jobs.
> After that, the ShuffleMaster becomes a cluster level service which is
> consistent with the ShuffleEnvironment at the TM side.
>
> As a summary, we propose to finish FLINK-12731
>  and make the shuffle
> service a real cluster level service first. Furthermore, we add two
> lifecycle methods to the ShuffleMaster interface, including start and
> close responding
> for initialization (for example, contacting the external system) and
> graceful shutdown (for example, releasing the resources) respectively
> (these methods already exist in the ShuffleEnvironment interface at the TM
> side). What do you think?
> Relationship of ShuffleMaster & JobMaster
>
> Currently, JobMaster holds the reference to the corresponding ShuffleMaster
> and it can register partitions (allocate ShuffleDescriptor from) to
> ShuffleMaster
> by the registerPartitionWithProducer method. To support use cases like
> allocating external resources when a job starts and releasing all allocated
> resources when a job 

Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished

[Guowei Ma]

+1 (binding)
Best,
Guowei


On Fri, Jul 2, 2021 at 11:56 AM Leonard Xu  wrote:

> +1,
>
> This may help https://issues.apache.org/jira/browse/FLINK-22764 <
> https://issues.apache.org/jira/browse/FLINK-22764>
>
> Best,
> Leonard
>
> > 在 2021年7月2日，10:17，JING ZHANG  写道：
> >
> > +1 (binding)
> >
> >
> > Arvid Heise  于2021年7月1日周四 下午3:10写道：
> >
> >> Looks good: +1 (binding)
> >>
> >> On Tue, Jun 29, 2021 at 5:06 AM 刘建刚  wrote:
> >>
> >>> +1 (binding)
> >>>
> >>> Best
> >>> liujiangang
> >>>
> >>> Piotr Nowojski  于2021年6月29日周二 上午2:05写道：
> >>>
>  +1 (binding)
> 
>  Piotrek
> 
>  pon., 28 cze 2021 o 12:48 Dawid Wysakowicz 
>  napisał(a):
> 
> > +1 (binding)
> >
> > Best,
> >
> > Dawid
> >
> > On 28/06/2021 10:45, Yun Gao wrote:
> >> Hi all,
> >>
> >> For FLIP-147[1] which targets at supports checkpoints after tasks
> > finished and modify operator
> >> API and implementation to ensures the commit of last piece of data,
> > since after the last vote
> >> we have more discussions[2][3] and a few updates, including changes
> >>> to
> > PublicEvolving API,
> >> I'd like to have another VOTE on the current state of the FLIP.
> >>
> >> The vote will last at least 72 hours (Jul 1st), following the
> >>> consensus
> >> voting process.
> >>
> >> thanks,
> >> Yun
> >>
> >>
> >> [1] https://cwiki.apache.org/confluence/x/mw-ZCQ
> >> [2]
> >
> 
> >>>
> >>
> https://lists.apache.org/thread.html/r400da9898ff66fd613c25efea15de440a86f14758ceeae4950ea25cf%40%3Cdev.flink.apache.org
> >> [3]
> >
> 
> >>>
> >>
> https://lists.apache.org/thread.html/r3953df796ef5ac67d5be9f2251a95ad72efbca31f1d1555d13e71197%40%3Cdev.flink.apache.org%3E
> >
> >
> 
> >>>
> >>
>
>

Re: Got multiple issues when running the tutorial project "table-walkthrough" on IDEA

[Guowei Ma]

Hi, Lingfeng

These job errors you posted happened when the job(`SpendReport`) was
running on the IDE?
According to my understanding, this document[1] & repository[2] mean that
the example is to be run in docker, not in IDE.

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/try-flink/table_api/
[2] https://github.com/apache/flink-playgrounds

Best,
Guowei


On Tue, Jun 15, 2021 at 10:29 AM Lingfeng Pu  wrote:

> Hi,
>
> I'm currently trying to debug the tutorial project "table-walkthrough" on
> IDEA on a standalone Flink environment. I installed the required software
> (Java 8 or 11, Maven, Docker) according to the tutorial. I'll provide some
> key environment info about the running environment before describing the
> issues as follows:
>
> 1. OS: Fedora 34;
> 2. Java version: 1.8;
> 3.Maven version: 3.6.3;
> 4. Docker version: 20.10.7;
> 5. Flink version: 1.13.1;
> 6. Scala version: 2.11;
> 7. No Kafka, Yarn, etc. are installed.
>
> *The issues are:*
> When I run the code on IDEA, it returns me WARNs and exceptions. Due to
> this tutorial project did not require things to be installed like Kafka,
> Yarn, I couldn't find any proper solution(s) for my problem after searching
> on the web. *The full issue report is shown below:*
>
>
> *=*
> /usr/java/jdk1.8.0_291-amd64/bin/java
> -javaagent:/var/lib/snapd/snap/intellij-idea-community/302/lib/idea_rt.jar=39087:/var/lib/snapd/snap/intellij-idea-community/302/bin
> -Dfile.encoding=UTF-8 -classpath
> 

Re: [DISCUSS] Limit size of already processed files in File Source SplitEnumerator

[Guowei Ma]

It would really simplify a lot if the modification timestamp of each newly 
scanned file is increased.

 We only need to record the file list corresponding to the largest timestamp.  
Timestamp of each scanned file
 1. It is smaller than the maximum timestamp, which means it has been processed;
 2. The timestamps are equal, so you need to see if it is in this file list, if 
it is, you don't need to process it, if it is not, you need to process it;
 3. It is larger than the maximum timestamp and has not been processed

 If the maximum timestamp is dynamically restored from the file list every time 
it is started, the state compatibility issue can be ignored.


BTW I haven't done any test, but I am actually a little curious, if a lot of 
files have been processed, is the scan itself already very slow?  I mean maybe 
the bottleneck at the beginning might be scan?



> 在 2021年6月8日，下午5:41，Till Rohrmann  写道：
> 
> Hi Tianxin,
> 
> thanks for starting this discussion. I am pulling in Arvid who works on
> Flink's connectors.
> 
> I think the problem you are describing can happen.
> 
> From what I understand you are proposing to keep track of the watermark of
> processed file input splits and then filter out splits based on their
> modification timestamps and the watermark. What is the benefit of keeping
> for every split the modification timestamp in the map? Could it also work
> if we sort the input splits according to their modification timestamps and
> then remember the last processed split? That way we only remember a single
> value and upon recovery, we only process those splits which have a newer
> modification timestamp.
> 
> Cheers,
> Till
> 
>> On Tue, Jun 8, 2021 at 12:11 AM Tianxin Zhao  wrote:
>> 
>> Hi!
>> 
>> Currently Flink File Source relies on a Set pathsAlreadyProcessed in
>> SplitEnumerator to decide which file has been processed and avoids
>> reprocessing files if a file is already in this set. However this set could
>> be ever growing and ultimately exceed memory size if there are new files
>> continuously added to the input path.
>> 
>> I submitted https://issues.apache.org/jira/browse/FLINK-22792 and would
>> like to be assigned to the ticket.
>> 
>> Current proposed change as belows, would like to get an agreement on the
>> approach taken.
>> 
>>   1.
>> 
>>   Maintain fileWatermark updated by new files modification time in
>>   ContinuousFileSplitEnumerator
>>   2.
>> 
>>   Change Set pathsAlreadyProcessed to a HashMap
>>   pathsAlreadyProcessed where the key is same as before which is the file
>>   path of already processed files, and the value is file modification
>> time,
>>   expose getModificationTime() method to FileSourceSplit.
>> 
>> 
>>   1.
>> 
>>   Adding a fileExpireTime user configurable config, any files older
>> than fileWatermark
>>   - fileExpireTime would get ignored.
>>   2.
>> 
>>   When snapshotting splitEnumerator, remove files that are older than
>> fileWatermark
>>   - fileExpireTime from the pathsAlreadyProcessed map.
>>   3.
>> 
>>   Adding alreadyProcessedPaths map and fileWatermark to
>>   PendingSplitsCheckpoint, modify the current
>>   PendingSplitsCheckpointSerializer to add a version2 serializer that
>>   serialize the alreadyProcessedPaths map which included file modification
>>   time.
>>   4.
>> 
>>   Subclass of PendingSplitsCheckpoint like
>>   ContinuousHivePendingSplitsCheckpoint would not be impacted by
>>   initializing an empty alreadyProcessedMap and 0 as initial watermark.
>> 
>> Thanks!
>> 

[jira] [Created] (FLINK-22779) KafkaChangelogTableITCase.testKafkaDebeziumChangelogSource fail due to ConcurrentModificationException

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22779:
-

 Summary: 
KafkaChangelogTableITCase.testKafkaDebeziumChangelogSource fail due to 
ConcurrentModificationException
 Key: FLINK-22779
 URL: https://issues.apache.org/jira/browse/FLINK-22779
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.13.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18330=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5=6608





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22775) CassandraConnectorITCase.testCassandraTableSink Fail

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22775:
-

 Summary: CassandraConnectorITCase.testCassandraTableSink Fail
 Key: FLINK-22775
 URL: https://issues.apache.org/jira/browse/FLINK-22775
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Cassandra
Affects Versions: 1.14.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18328=logs=ba53eb01-1462-56a3-8e98-0dd97fbcaab5=bfbc6239-57a0-5db0-63f3-41551b4f7d51=14105


{code:java}
 2021-05-25T23:03:44.0756266Z May 25 23:03:44 [ERROR] 
testCassandraTableSink(org.apache.flink.streaming.connectors.cassandra.CassandraConnectorITCase)
  Time elapsed: 13.673 s  <<< ERROR!
2021-05-25T23:03:44.0757635Z May 25 23:03:44 
java.util.concurrent.ExecutionException: 
org.apache.flink.table.api.TableException: Failed to wait job finish
2021-05-25T23:03:44.0760262Z May 25 23:03:44at 
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
2021-05-25T23:03:44.0761504Z May 25 23:03:44at 
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
2021-05-25T23:03:44.0762906Z May 25 23:03:44at 
org.apache.flink.table.api.internal.TableResultImpl.awaitInternal(TableResultImpl.java:129)
2021-05-25T23:03:44.0763878Z May 25 23:03:44at 
org.apache.flink.table.api.internal.TableResultImpl.await(TableResultImpl.java:92)
2021-05-25T23:03:44.0764918Z May 25 23:03:44at 
org.apache.flink.streaming.connectors.cassandra.CassandraConnectorITCase.testCassandraTableSink(CassandraConnectorITCase.java:520)
2021-05-25T23:03:44.0768225Z May 25 23:03:44at 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2021-05-25T23:03:44.0769100Z May 25 23:03:44at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2021-05-25T23:03:44.0769917Z May 25 23:03:44at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2021-05-25T23:03:44.0770645Z May 25 23:03:44at 
java.lang.reflect.Method.invoke(Method.java:498)
2021-05-25T23:03:44.0771387Z May 25 23:03:44at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2021-05-25T23:03:44.0772228Z May 25 23:03:44at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2021-05-25T23:03:44.0773541Z May 25 23:03:44at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2021-05-25T23:03:44.0774367Z May 25 23:03:44at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2021-05-25T23:03:44.0775246Z May 25 23:03:44at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
2021-05-25T23:03:44.0776088Z May 25 23:03:44at 
org.apache.flink.testutils.junit.RetryRule$RetryOnExceptionStatement.evaluate(RetryRule.java:192)
2021-05-25T23:03:44.0776946Z May 25 23:03:44at 
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
2021-05-25T23:03:44.0777685Z May 25 23:03:44at 
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
2021-05-25T23:03:44.0778447Z May 25 23:03:44at 
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2021-05-25T23:03:44.0779110Z May 25 23:03:44at 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2021-05-25T23:03:44.0779893Z May 25 23:03:44at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2021-05-25T23:03:44.0780744Z May 25 23:03:44at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2021-05-25T23:03:44.0781493Z May 25 23:03:44at 
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2021-05-25T23:03:44.0782154Z May 25 23:03:44at 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2021-05-25T23:03:44.0782899Z May 25 23:03:44at 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2021-05-25T23:03:44.0783576Z May 25 23:03:44at 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2021-05-25T23:03:44.0784312Z May 25 23:03:44at 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2021-05-25T23:03:44.0785020Z May 25 23:03:44at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
2021-05-25T23:03:44.0785815Z May 25 23:03:44at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
2021-05-25T23:03:44.0786619Z May 25 23:03:44at 
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
2021-05-25T23:03:44.0787343Z May 25 23:03:44at 
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2021-05-25T23:03:44.0788202Z May 25 23:03:44at 
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2021-05-25T23:03:44.0789018Z May 25 23:03:44at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
2021-05

[jira] [Created] (FLINK-22756) DispatcherTest.testJobStatusIsShownDuringTermination fail

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22756:
-

 Summary: DispatcherTest.testJobStatusIsShownDuringTermination fail
 Key: FLINK-22756
 URL: https://issues.apache.org/jira/browse/FLINK-22756
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.12.4
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18255=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=05b74a19-4ee4-5036-c46f-ada307df6cf0=7871


{code:java}
2021-05-21T21:11:09.4635978Z java.util.concurrent.ExecutionException: 
org.apache.flink.util.FlinkException: JobMaster has been shut down.
2021-05-21T21:11:09.4637019Zat 
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
2021-05-21T21:11:09.4637464Zat 
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
2021-05-21T21:11:09.4637981Zat 
org.apache.flink.runtime.dispatcher.DispatcherTest.lambda$testJobStatusIsShownDuringTermination$4(DispatcherTest.java:883)
2021-05-21T21:11:09.4638553Zat 
org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:145)
2021-05-21T21:11:09.4639055Zat 
org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:129)
2021-05-21T21:11:09.4639610Zat 
org.apache.flink.runtime.dispatcher.DispatcherTest.testJobStatusIsShownDuringTermination(DispatcherTest.java:878)
2021-05-21T21:11:09.4640216Zat 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2021-05-21T21:11:09.4640602Zat 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2021-05-21T21:11:09.4641247Zat 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2021-05-21T21:11:09.4641741Zat 
java.lang.reflect.Method.invoke(Method.java:498)
2021-05-21T21:11:09.4642140Zat 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2021-05-21T21:11:09.4642614Zat 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2021-05-21T21:11:09.4643071Zat 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2021-05-21T21:11:09.4643542Zat 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2021-05-21T21:11:09.4644005Zat 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
2021-05-21T21:11:09.4644439Zat 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
2021-05-21T21:11:09.4644856Zat 
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
2021-05-21T21:11:09.4645262Zat 
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
2021-05-21T21:11:09.4645791Zat 
org.apache.flink.runtime.util.TestingFatalErrorHandlerResource$CloseableStatement.evaluate(TestingFatalErrorHandlerResource.java:91)
2021-05-21T21:11:09.4646462Zat 
org.apache.flink.runtime.util.TestingFatalErrorHandlerResource$CloseableStatement.access$200(TestingFatalErrorHandlerResource.java:83)
2021-05-21T21:11:09.4647082Zat 
org.apache.flink.runtime.util.TestingFatalErrorHandlerResource$1.evaluate(TestingFatalErrorHandlerResource.java:55)
2021-05-21T21:11:09.4647575Zat 
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
2021-05-21T21:11:09.4654167Zat 
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2021-05-21T21:11:09.4654899Zat 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2021-05-21T21:11:09.4655596Zat 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2021-05-21T21:11:09.4656331Zat 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2021-05-21T21:11:09.4657027Zat 
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2021-05-21T21:11:09.4657627Zat 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2021-05-21T21:11:09.4658233Zat 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2021-05-21T21:11:09.4658829Zat 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2021-05-21T21:11:09.4659467Zat 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2021-05-21T21:11:09.4660188Zat 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
2021-05-21T21:11:09.4660863Zat 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
2021-05-21T21:11:09.4661874Zat 
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2021-05-21T21:11:09.4662519Zat 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
2021-05-21T21:11:09.4663240Zat 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
2021-05-21T21:11:09.4664297Zat 
org.apache.maven.surefire.junit4

[jira] [Created] (FLINK-22755) SequenceStreamingFileSinkITCase.testWriteSequenceFile fail

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22755:
-

 Summary: SequenceStreamingFileSinkITCase.testWriteSequenceFile fail
 Key: FLINK-22755
 URL: https://issues.apache.org/jira/browse/FLINK-22755
 Project: Flink
  Issue Type: Bug
  Components: Connectors / FileSystem
Affects Versions: 1.14.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18224=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=03dca39c-73e8-5aaf-601d-328ae5c35f20=14321


{code:java}
May 21 09:00:00 [ERROR] Failures: 
May 21 09:00:00 [ERROR]   
SequenceStreamingFileSinkITCase.testWriteSequenceFile:97->validateResults:119 
expected:<1> but was:<2>
May 21 09:00:00 [INFO] 

{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22739) CassandraSinkBaseTest.testTimeoutExceptionOnInvoke fail due to TestTimedOutException

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22739:
-

 Summary: CassandraSinkBaseTest.testTimeoutExceptionOnInvoke fail 
due to TestTimedOutException
 Key: FLINK-22739
 URL: https://issues.apache.org/jira/browse/FLINK-22739
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Cassandra
Affects Versions: 1.13.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18207=logs=e9af9cde-9a65-5281-a58e-2c8511d36983=b6c4efed-9c7d-55ea-03a9-9bd7d5b08e4c=13438


{code:java}
2021-05-20T22:21:49.9790943Z May 20 22:21:49 [ERROR] 
testTimeoutExceptionOnInvoke(org.apache.flink.streaming.connectors.cassandra.CassandraSinkBaseTest)
  Time elapsed: 5.742 s  <<< ERROR!
2021-05-20T22:21:49.9791886Z May 20 22:21:49 
org.junit.runners.model.TestTimedOutException: test timed out after 5000 
milliseconds
2021-05-20T22:21:49.9792584Z May 20 22:21:49at 
java.base@11.0.10/java.io.RandomAccessFile.readBytes(Native Method)
2021-05-20T22:21:49.9793258Z May 20 22:21:49at 
java.base@11.0.10/java.io.RandomAccessFile.read(RandomAccessFile.java:406)
2021-05-20T22:21:49.9794205Z May 20 22:21:49at 
java.base@11.0.10/java.util.zip.ZipFile$Source.readAt(ZipFile.java:1331)
2021-05-20T22:21:49.9794931Z May 20 22:21:49at 
java.base@11.0.10/java.util.zip.ZipFile$ZipFileInputStream.read(ZipFile.java:1028)
2021-05-20T22:21:49.9795769Z May 20 22:21:49at 
java.base@11.0.10/java.util.zip.ZipFile$ZipFileInflaterInputStream.fill(ZipFile.java:468)
2021-05-20T22:21:49.9796538Z May 20 22:21:49at 
java.base@11.0.10/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:159)
2021-05-20T22:21:49.9797988Z May 20 22:21:49at 
java.base@11.0.10/jdk.internal.loader.Resource.getBytes(Resource.java:124)
2021-05-20T22:21:49.9798766Z May 20 22:21:49at 
java.base@11.0.10/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:797)
2021-05-20T22:21:49.9799605Z May 20 22:21:49at 
java.base@11.0.10/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
2021-05-20T22:21:49.9800472Z May 20 22:21:49at 
java.base@11.0.10/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
2021-05-20T22:21:49.9801568Z May 20 22:21:49at 
java.base@11.0.10/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
2021-05-20T22:21:49.9802449Z May 20 22:21:49at 
java.base@11.0.10/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
2021-05-20T22:21:49.9803167Z May 20 22:21:49at 
java.base@11.0.10/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
2021-05-20T22:21:49.9803984Z May 20 22:21:49at 
app//com.google.common.collect.ImmutableMap.of(ImmutableMap.java:70)
2021-05-20T22:21:49.9804648Z May 20 22:21:49at 
app//com.google.common.reflect.TypeResolver$TypeTable.(TypeResolver.java:218)
2021-05-20T22:21:49.9805331Z May 20 22:21:49at 
app//com.google.common.reflect.TypeResolver.(TypeResolver.java:60)
2021-05-20T22:21:49.9805972Z May 20 22:21:49at 
app//com.google.common.reflect.TypeToken.where(TypeToken.java:215)
2021-05-20T22:21:49.9806669Z May 20 22:21:49at 
app//com.google.common.reflect.TypeToken.where(TypeToken.java:239)
2021-05-20T22:21:49.9807514Z May 20 22:21:49at 
app//com.datastax.driver.core.TypeTokens.mapOf(TypeTokens.java:106)
2021-05-20T22:21:49.9808329Z May 20 22:21:49at 
app//com.datastax.driver.core.SanityChecks.checkGuava(SanityChecks.java:50)
2021-05-20T22:21:49.9809035Z May 20 22:21:49at 
app//com.datastax.driver.core.SanityChecks.check(SanityChecks.java:36)
2021-05-20T22:21:49.9809696Z May 20 22:21:49at 
app//com.datastax.driver.core.Cluster.(Cluster.java:67)
2021-05-20T22:21:49.9810253Z May 20 22:21:49at 
jdk.internal.reflect.GeneratedSerializationConstructorAccessor2.newInstance(Unknown
 Source)
2021-05-20T22:21:49.9810930Z May 20 22:21:49at 
java.base@11.0.10/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
2021-05-20T22:21:49.9811707Z May 20 22:21:49at 
app//org.objenesis.instantiator.sun.SunReflectionFactoryInstantiator.newInstance(SunReflectionFactoryInstantiator.java:45)
2021-05-20T22:21:49.9812475Z May 20 22:21:49at 
app//org.objenesis.ObjenesisBase.newInstance(ObjenesisBase.java:73)
2021-05-20T22:21:49.9813232Z May 20 22:21:49at 
app//org.mockito.internal.creation.instance.ObjenesisInstantiator.newInstance(ObjenesisInstantiator.java:19)
2021-05-20T22:21:49.9814154Z May 20 22:21:49at 
app//org.mockito.internal.creation.bytebuddy.SubclassByteBuddyMockMaker.createMock(SubclassByteBuddyMockMaker.java:47)
2021-05-20T22:21:49.9814990Z May 20 22:21:49at 
app//org.mockito.internal.creation.bytebuddy.ByteBuddyMockMaker.createMock(ByteBuddyMockMaker.java:25)
2021-05-20T22:21:49.9815814Z May 20 22:21:49  

[jira] [Created] (FLINK-22738) org.apache.flink.connector.jdbc.xa.JdbcExactlyOnceSinkE2eTest no output for 900 seconds due to block on downloading docker image

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22738:
-

 Summary: 
org.apache.flink.connector.jdbc.xa.JdbcExactlyOnceSinkE2eTest no output for 900 
seconds due to block on downloading  docker image 
 Key: FLINK-22738
 URL: https://issues.apache.org/jira/browse/FLINK-22738
 Project: Flink
  Issue Type: Bug
  Components: Connectors / JDBC
Affects Versions: 1.13.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18207=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=03dca39c-73e8-5aaf-601d-328ae5c35f20=13626

{code:java}
2021-05-20T23:12:33.7620808Z May 20 23:12:33 "main" #1 prio=5 os_prio=0 
tid=0x7fa6c400b800 nid=0x45f3 runnable [0x7fa6cb1cd000]
2021-05-20T23:12:33.7621514Z May 20 23:12:33java.lang.Thread.State: RUNNABLE
2021-05-20T23:12:33.7622205Z May 20 23:12:33at 
org.testcontainers.shaded.okio.Buffer.indexOf(Buffer.java:1463)
2021-05-20T23:12:33.7623070Z May 20 23:12:33at 
org.testcontainers.shaded.okio.RealBufferedSource.indexOf(RealBufferedSource.java:352)
2021-05-20T23:12:33.7624186Z May 20 23:12:33at 
org.testcontainers.shaded.okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:230)
2021-05-20T23:12:33.7625161Z May 20 23:12:33at 
org.testcontainers.shaded.okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:224)
2021-05-20T23:12:33.7626488Z May 20 23:12:33at 
org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.readChunkSize(Http1ExchangeCodec.java:489)
2021-05-20T23:12:33.7627762Z May 20 23:12:33at 
org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.read(Http1ExchangeCodec.java:471)
2021-05-20T23:12:33.7628864Z May 20 23:12:33at 
org.testcontainers.shaded.okhttp3.internal.Util.skipAll(Util.java:204)
2021-05-20T23:12:33.7629713Z May 20 23:12:33at 
org.testcontainers.shaded.okhttp3.internal.Util.discard(Util.java:186)
2021-05-20T23:12:33.7630720Z May 20 23:12:33at 
org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.close(Http1ExchangeCodec.java:511)
2021-05-20T23:12:33.7631677Z May 20 23:12:33at 
org.testcontainers.shaded.okio.ForwardingSource.close(ForwardingSource.java:43)
2021-05-20T23:12:33.7632685Z May 20 23:12:33at 
org.testcontainers.shaded.okhttp3.internal.connection.Exchange$ResponseBodySource.close(Exchange.java:313)
2021-05-20T23:12:33.7633800Z May 20 23:12:33at 
org.testcontainers.shaded.okio.RealBufferedSource.close(RealBufferedSource.java:476)
2021-05-20T23:12:33.7634690Z May 20 23:12:33at 
org.testcontainers.shaded.okhttp3.internal.Util.closeQuietly(Util.java:139)
2021-05-20T23:12:33.763Z May 20 23:12:33at 
org.testcontainers.shaded.okhttp3.ResponseBody.close(ResponseBody.java:192)
2021-05-20T23:12:33.7636403Z May 20 23:12:33at 
org.testcontainers.shaded.okhttp3.Response.close(Response.java:290)
2021-05-20T23:12:33.7637381Z May 20 23:12:33at 
org.testcontainers.shaded.com.github.dockerjava.okhttp.OkDockerHttpClient$OkResponse.close(OkDockerHttpClient.java:285)
2021-05-20T23:12:33.7638608Z May 20 23:12:33at 
org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$null$0(DefaultInvocationBuilder.java:272)
2021-05-20T23:12:33.7639473Z May 20 23:12:33at 
org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder$$Lambda$83/733374718.close(Unknown
 Source)
2021-05-20T23:12:33.7640236Z May 20 23:12:33at 
com.github.dockerjava.api.async.ResultCallbackTemplate.close(ResultCallbackTemplate.java:77)
2021-05-20T23:12:33.7641079Z May 20 23:12:33at 
org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:202)
2021-05-20T23:12:33.7641912Z May 20 23:12:33at 
org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:205)
2021-05-20T23:12:33.7643123Z May 20 23:12:33- locked <0x991ac238> 
(a [Ljava.lang.Object;)
2021-05-20T23:12:33.7644023Z May 20 23:12:33at 
org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
2021-05-20T23:12:33.7644880Z May 20 23:12:33at 
org.testcontainers.LazyDockerClient.authConfig(LazyDockerClient.java:12)
2021-05-20T23:12:33.7645712Z May 20 23:12:33at 
org.testcontainers.containers.GenericContainer.start(GenericContainer.java:310)
2021-05-20T23:12:33.7646636Z May 20 23:12:33at 
org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1029)
2021-05-20T23:12:33.7647619Z May 20 23:12:33at 
org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:29)
2021-05-20T23:12:33.7648654Z May 20 23:12:33at 
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2021-05-20T23:12:33.7649382Z May 20 23:12:33at 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2021-05-20T23:12:33.7650202Z

[jira] [Created] (FLINK-22735) HiveTableSourceITCase.testStreamPartitionReadByCreateTime failed because of times out

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22735:
-

 Summary: HiveTableSourceITCase.testStreamPartitionReadByCreateTime 
failed because of times out 
 Key: FLINK-22735
 URL: https://issues.apache.org/jira/browse/FLINK-22735
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Hive
Affects Versions: 1.14.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18205=logs=245e1f2e-ba5b-5570-d689-25ae21e5302f=e7f339b2-a7c3-57d9-00af-3712d4b15354=23726


{code:java}
May 20 22:22:26 [ERROR] Tests run: 19, Failures: 0, Errors: 1, Skipped: 0, Time 
elapsed: 225.004 s <<< FAILURE! - in 
org.apache.flink.connectors.hive.HiveTableSourceITCase
May 20 22:22:26 [ERROR] 
testStreamPartitionReadByCreateTime(org.apache.flink.connectors.hive.HiveTableSourceITCase)
  Time elapsed: 120.182 s  <<< ERROR!
May 20 22:22:26 org.junit.runners.model.TestTimedOutException: test timed out 
after 12 milliseconds
May 20 22:22:26 at java.lang.Thread.sleep(Native Method)
May 20 22:22:26 at 
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sleepBeforeRetry(CollectResultFetcher.java:237)
May 20 22:22:26 at 
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:113)
May 20 22:22:26 at 
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106)
May 20 22:22:26 at 
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
May 20 22:22:26 at 
org.apache.flink.table.api.internal.TableResultImpl$CloseableRowIteratorWrapper.hasNext(TableResultImpl.java:370)
May 20 22:22:26 at 
org.apache.flink.connectors.hive.HiveTableSourceITCase.fetchRows(HiveTableSourceITCase.java:712)
May 20 22:22:26 at 
org.apache.flink.connectors.hive.HiveTableSourceITCase.testStreamPartitionReadByCreateTime(HiveTableSourceITCase.java:652)
May 20 22:22:26 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
May 20 22:22:26 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
May 20 22:22:26 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
May 20 22:22:26 at java.lang.reflect.Method.invoke(Method.java:498)
May 20 22:22:26 at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
May 20 22:22:26 at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
May 20 22:22:26 at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
May 20 22:22:26 at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
May 20 22:22:26 at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
May 20 22:22:26 at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
May 20 22:22:26 at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
May 20 22:22:26 at java.lang.Thread.run(Thread.java:748)

{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22720) UpsertKafkaTableITCase.testAggregate fail due to ConcurrentModificationException

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22720:
-

 Summary: UpsertKafkaTableITCase.testAggregate fail due to 
ConcurrentModificationException
 Key: FLINK-22720
 URL: https://issues.apache.org/jira/browse/FLINK-22720
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka, Tests
Affects Versions: 1.14.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18151=logs=c5612577-f1f7-5977-6ff6-7432788526f7=53f6305f-55e6-561c-8f1e-3a1dde2c77df=6613


{code:java}
2021-05-19T21:28:02.8689083Z May 19 21:28:02 [ERROR] testAggregate[format = 
avro](org.apache.flink.streaming.connectors.kafka.table.UpsertKafkaTableITCase) 
 Time elapsed: 2.067 s  <<< ERROR!
2021-05-19T21:28:02.8708337Z May 19 21:28:02 
java.util.ConcurrentModificationException
2021-05-19T21:28:02.8710333Z May 19 21:28:02at 
java.util.HashMap$HashIterator.nextNode(HashMap.java:1445)
2021-05-19T21:28:02.8712083Z May 19 21:28:02at 
java.util.HashMap$ValueIterator.next(HashMap.java:1474)
2021-05-19T21:28:02.8712680Z May 19 21:28:02at 
java.util.AbstractCollection.toArray(AbstractCollection.java:141)
2021-05-19T21:28:02.8713142Z May 19 21:28:02at 
java.util.ArrayList.addAll(ArrayList.java:583)
2021-05-19T21:28:02.8716029Z May 19 21:28:02at 
org.apache.flink.table.planner.factories.TestValuesRuntimeFunctions.lambda$getResults$0(TestValuesRuntimeFunctions.java:114)
2021-05-19T21:28:02.8717007Z May 19 21:28:02at 
java.util.HashMap$Values.forEach(HashMap.java:981)
2021-05-19T21:28:02.8718041Z May 19 21:28:02at 
org.apache.flink.table.planner.factories.TestValuesRuntimeFunctions.getResults(TestValuesRuntimeFunctions.java:114)
2021-05-19T21:28:02.8719339Z May 19 21:28:02at 
org.apache.flink.table.planner.factories.TestValuesTableFactory.getResults(TestValuesTableFactory.java:184)
2021-05-19T21:28:02.8720309Z May 19 21:28:02at 
org.apache.flink.streaming.connectors.kafka.table.KafkaTableTestUtils.waitingExpectedResults(KafkaTableTestUtils.java:82)
2021-05-19T21:28:02.8721311Z May 19 21:28:02at 
org.apache.flink.streaming.connectors.kafka.table.UpsertKafkaTableITCase.wordFreqToUpsertKafka(UpsertKafkaTableITCase.java:440)
2021-05-19T21:28:02.8730402Z May 19 21:28:02at 
org.apache.flink.streaming.connectors.kafka.table.UpsertKafkaTableITCase.testAggregate(UpsertKafkaTableITCase.java:73)
2021-05-19T21:28:02.8731390Z May 19 21:28:02at 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2021-05-19T21:28:02.8732095Z May 19 21:28:02at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2021-05-19T21:28:02.8732935Z May 19 21:28:02at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2021-05-19T21:28:02.8733726Z May 19 21:28:02at 
java.lang.reflect.Method.invoke(Method.java:498)
2021-05-19T21:28:02.8734598Z May 19 21:28:02at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2021-05-19T21:28:02.8735450Z May 19 21:28:02at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2021-05-19T21:28:02.8736313Z May 19 21:28:02at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2021-05-19T21:28:02.8737329Z May 19 21:28:02at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2021-05-19T21:28:02.8738165Z May 19 21:28:02at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
2021-05-19T21:28:02.8738989Z May 19 21:28:02at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
2021-05-19T21:28:02.8739741Z May 19 21:28:02at 
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
2021-05-19T21:28:02.8740563Z May 19 21:28:02at 
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
2021-05-19T21:28:02.8741340Z May 19 21:28:02at 
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
2021-05-19T21:28:02.8742077Z May 19 21:28:02at 
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2021-05-19T21:28:02.8742802Z May 19 21:28:02at 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2021-05-19T21:28:02.8743594Z May 19 21:28:02at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2021-05-19T21:28:02.8744811Z May 19 21:28:02at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2021-05-19T21:28:02.8745580Z May 19 21:28:02at 
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2021-05-19T21:28:02.8746330Z May 19 21:28:02at 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2021-05-19T21:28:02.8747222Z May 19 21:28:02at 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2021-05-19T21:28:02.8748007Z May 19

[jira] [Created] (FLINK-22710) ParquetProtoStreamingFileSinkITCase.testParquetProtoWriters failed

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22710:
-

 Summary: 
ParquetProtoStreamingFileSinkITCase.testParquetProtoWriters failed
 Key: FLINK-22710
 URL: https://issues.apache.org/jira/browse/FLINK-22710
 Project: Flink
  Issue Type: Bug
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.14.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18114=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=03dca39c-73e8-5aaf-601d-328ae5c35f20=13181


{code:java}

May 19 03:00:31 [ERROR]   
ParquetProtoStreamingFileSinkITCase.testParquetProtoWriters:83->validateResults:92
 expected:<1> but was:<2>

{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22705) SQL Client end-to-end test (Old planner) Elasticsearch (v7.5.1) failed due to fail to download the tar

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22705:
-

 Summary: SQL Client end-to-end test (Old planner) Elasticsearch 
(v7.5.1) failed due to fail to download the tar
 Key: FLINK-22705
 URL: https://issues.apache.org/jira/browse/FLINK-22705
 Project: Flink
  Issue Type: Bug
  Components: Connectors / ElasticSearch
Affects Versions: 1.13.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18100=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529=18408

{code:java}

May 18 17:24:23 Preparing Elasticsearch (version=7)...
May 18 17:24:23 Downloading Elasticsearch from 
https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-7.5.1-no-jdk-linux-x86_64.tar.gz
 ...
--2021-05-18 17:24:23--  
https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-7.5.1-no-jdk-linux-x86_64.tar.gz
Resolving artifacts.elastic.co (artifacts.elastic.co)... 34.120.127.130, 
2600:1901:0:1d7::
Connecting to artifacts.elastic.co 
(artifacts.elastic.co)|34.120.127.130|:443... failed: Connection timed out.
Connecting to artifacts.elastic.co 
(artifacts.elastic.co)|2600:1901:0:1d7::|:443... failed: Network is unreachable.
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed

  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0curl: (7) Failed to connect to localhost port 9200: Connection refused
May 18 17:26:34 [FAIL] Test script contains errors.
May 18 17:26:34 Checking for errors...
May 18 17:26:34 No errors in log files.
May 18 17:26:34 Checking for exceptions...
May 18 17:26:34 No exceptions in log files.
May 18 17:26:34 Checking for non-empty .out files...
grep: /home/vsts/work/_temp/debug_files/flink-logs/*.out: No such file or 
directory
May 18 17:26:34 No non-empty .out files.
May 18 17:26:34 
May 18 17:26:34 [FAIL] 'SQL Client end-to-end test (Old planner) Elasticsearch 
(v7.5.1)' failed after 2 minutes and 36 seconds! Test exited with exit code 1
May 18 17:26:34
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22704) ZooKeeperHaServicesTest.testCleanupJobData failed

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22704:
-

 Summary: ZooKeeperHaServicesTest.testCleanupJobData failed
 Key: FLINK-22704
 URL: https://issues.apache.org/jira/browse/FLINK-22704
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.13.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18108=logs=77a9d8e1-d610-59b3-fc2a-4766541e0e33=7c61167f-30b3-5893-cc38-a9e3d057e392=8172


{code:java}
May 19 01:30:02 Expected: a collection containing 
"1a2850d5759a2e1f4fef9cc7e6abc675"
May 19 01:30:02  but: was "resource_manager_lock"

{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22703) SavepointITCase.testTriggerSavepointAndResumeWithFileBasedCheckpoints failed

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22703:
-

 Summary: 
SavepointITCase.testTriggerSavepointAndResumeWithFileBasedCheckpoints failed
 Key: FLINK-22703
 URL: https://issues.apache.org/jira/browse/FLINK-22703
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.14.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18106=logs=4d4a0d10-fca2-5507-8eed-c07f0bdf4887=c2734c79-73b6-521c-e85a-67c7ecae9107=9765


{code:java}

May 19 00:51:42 Caused by: 
org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint triggering 
task Map (3/4) of job 1785ead58e598bc669c92d5a2cd14618 has not being executed 
at the moment. Aborting checkpoint. Failure reason: Not all required tasks are 
currently running.
May 19 00:51:42 at 
org.apache.flink.runtime.checkpoint.DefaultCheckpointPlanCalculator.checkTasksStarted(DefaultCheckpointPlanCalculator.java:152)
May 19 00:51:42 at 
org.apache.flink.runtime.checkpoint.DefaultCheckpointPlanCalculator.lambda$calculateCheckpointPlan$1(DefaultCheckpointPlanCalculator.java:114)
May 19 00:51:42 at 
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
May 19 00:51:42 at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:440)
May 19 00:51:42 at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:208)
May 19 00:51:42 at 
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77)
May 19 00:51:42 at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158)
May 19 00:51:42 at 
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
May 19 00:51:42 at 
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
May 19 00:51:42 at 
scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
May 19 00:51:42 at 
akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
May 19 00:51:42 at 
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
May 19 00:51:42 at 
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
May 19 00:51:42 at 
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
May 19 00:51:42 at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
May 19 00:51:42 at 
akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
May 19 00:51:42 at 
akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
May 19 00:51:42 at akka.actor.ActorCell.invoke(ActorCell.scala:561)
May 19 00:51:42 at 
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
May 19 00:51:42 at akka.dispatch.Mailbox.run(Mailbox.scala:225)
May 19 00:51:42 at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
May 19 00:51:42 at 
akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
May 19 00:51:42 at 
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
May 19 00:51:42 at 
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
May 19 00:51:42 at 
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22702) KafkaSourceITCase.testRedundantParallelism failed

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22702:
-

 Summary: KafkaSourceITCase.testRedundantParallelism failed
 Key: FLINK-22702
 URL: https://issues.apache.org/jira/browse/FLINK-22702
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.12.3
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18107=logs=1fc6e7bf-633c-5081-c32a-9dea24b05730=80a658d1-f7f6-5d93-2758-53ac19fd5b19=6847

{code:java}
Caused by: java.lang.RuntimeException: One or more fetchers have encountered 
exception
at 
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:199)
at 
org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:154)
at 
org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:116)
at 
org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:275)
at 
org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:67)
at 
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:398)
at 
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:191)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:619)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:583)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:758)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:573)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: SplitFetcher thread 0 received 
unexpected exception while polling the records
at 
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:146)
at 
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:101)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: java.lang.IllegalStateException: Consumer is not subscribed to any 
topics or assigned any partitions
at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1223)
at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1211)
at 
org.apache.flink.connector.kafka.source.reader.KafkaPartitionSplitReader.fetch(KafkaPartitionSplitReader.java:97)
at 
org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:56)
at 
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:138)
... 6 more

{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22662) YARNHighAvailabilityITCase.testKillYarnSessionClusterEntrypoint fail

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22662:
-

 Summary: 
YARNHighAvailabilityITCase.testKillYarnSessionClusterEntrypoint fail
 Key: FLINK-22662
 URL: https://issues.apache.org/jira/browse/FLINK-22662
 Project: Flink
  Issue Type: Bug
  Components: Deployment / YARN
Affects Versions: 1.13.0
Reporter: Guowei Ma



{code:java}
2021-05-14T00:24:57.8487649Z May 14 00:24:57 [ERROR] 
testKillYarnSessionClusterEntrypoint(org.apache.flink.yarn.YARNHighAvailabilityITCase)
  Time elapsed: 34.667 s  <<< ERROR!
2021-05-14T00:24:57.8488567Z May 14 00:24:57 
java.util.concurrent.ExecutionException: 
2021-05-14T00:24:57.8489301Z May 14 00:24:57 
org.apache.flink.runtime.rest.util.RestClientException: 
[org.apache.flink.runtime.rest.handler.RestHandlerException: 
org.apache.flink.runtime.messages.FlinkJobNotFoundException: Could not find 
Flink job (610ed4b159ece04c8ee2ec40e7d0c143)
2021-05-14T00:24:57.8493142Z May 14 00:24:57at 
org.apache.flink.runtime.rest.handler.job.JobExecutionResultHandler.propagateException(JobExecutionResultHandler.java:94)
2021-05-14T00:24:57.8495823Z May 14 00:24:57at 
org.apache.flink.runtime.rest.handler.job.JobExecutionResultHandler.lambda$handleRequest$1(JobExecutionResultHandler.java:84)
2021-05-14T00:24:57.8496733Z May 14 00:24:57at 
java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884)
2021-05-14T00:24:57.8497640Z May 14 00:24:57at 
java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866)
2021-05-14T00:24:57.8498491Z May 14 00:24:57at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-05-14T00:24:57.8499222Z May 14 00:24:57at 
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
2021-05-14T00:24:57.853Z May 14 00:24:57at 
org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:234)
2021-05-14T00:24:57.8500872Z May 14 00:24:57at 
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
2021-05-14T00:24:57.8501702Z May 14 00:24:57at 
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
2021-05-14T00:24:57.8502662Z May 14 00:24:57at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-05-14T00:24:57.8503472Z May 14 00:24:57at 
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
2021-05-14T00:24:57.8504269Z May 14 00:24:57at 
org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:1079)
2021-05-14T00:24:57.8504892Z May 14 00:24:57at 
akka.dispatch.OnComplete.internal(Future.scala:263)
2021-05-14T00:24:57.8505565Z May 14 00:24:57at 
akka.dispatch.OnComplete.internal(Future.scala:261)
2021-05-14T00:24:57.8506062Z May 14 00:24:57at 
akka.dispatch.japi$CallbackBridge.apply(Future.scala:191)
2021-05-14T00:24:57.8506819Z May 14 00:24:57at 
akka.dispatch.japi$CallbackBridge.apply(Future.scala:188)
2021-05-14T00:24:57.8507418Z May 14 00:24:57at 
scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
2021-05-14T00:24:57.8508373Z May 14 00:24:57at 
org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:73)
2021-05-14T00:24:57.8509144Z May 14 00:24:57at 
scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
2021-05-14T00:24:57.8509972Z May 14 00:24:57at 
scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
2021-05-14T00:24:57.8510675Z May 14 00:24:57at 
akka.pattern.PromiseActorRef.$bang(AskSupport.scala:572)
2021-05-14T00:24:57.8511376Z May 14 00:24:57at 
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:23)
2021-05-14T00:24:57.851Z May 14 00:24:57at 
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:21)
2021-05-14T00:24:57.8513090Z May 14 00:24:57at 
scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:436)
2021-05-14T00:24:57.8513835Z May 14 00:24:57at 
scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:435)
2021-05-14T00:24:57.8514576Z May 14 00:24:57at 
scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
2021-05-14T00:24:57.8515344Z May 14 00:24:57at 
akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
2021-05-14T00:24:57.8516317Z May 14 00:24:57at 
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
2021-05-14T00:24:57.8517537Z May 14 00:24:57at 
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
2021-05-14T00:24:57.8518525Z May 14 00:24:57at 
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$ru

[jira] [Created] (FLINK-22613) FlinkKinesisITCase.testStopWithSavepoint fails

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22613:
-

 Summary: FlinkKinesisITCase.testStopWithSavepoint fails
 Key: FLINK-22613
 URL: https://issues.apache.org/jira/browse/FLINK-22613
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kinesis
Affects Versions: 1.13.0
Reporter: Guowei Ma



{code:java}
2021-05-10T03:09:18.4601182Z May 10 03:09:18 [ERROR] 
testStopWithSavepoint(org.apache.flink.streaming.connectors.kinesis.FlinkKinesisITCase)
  Time elapsed: 3.526 s  <<< FAILURE!
2021-05-10T03:09:18.4601884Z May 10 03:09:18 java.lang.AssertionError: 
2021-05-10T03:09:18.4605902Z May 10 03:09:18 
2021-05-10T03:09:18.4616154Z May 10 03:09:18 Expected: a collection with size a 
value less than <10>
2021-05-10T03:09:18.4616818Z May 10 03:09:18  but: collection size <10> was 
equal to <10>
2021-05-10T03:09:18.4618087Z May 10 03:09:18at 
org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
2021-05-10T03:09:18.4618702Z May 10 03:09:18at 
org.junit.Assert.assertThat(Assert.java:956)
2021-05-10T03:09:18.4619467Z May 10 03:09:18at 
org.junit.Assert.assertThat(Assert.java:923)
2021-05-10T03:09:18.4620391Z May 10 03:09:18at 
org.apache.flink.streaming.connectors.kinesis.FlinkKinesisITCase.testStopWithSavepoint(FlinkKinesisITCase.java:126)
2021-05-10T03:09:18.4621115Z May 10 03:09:18at 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2021-05-10T03:09:18.4621751Z May 10 03:09:18at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2021-05-10T03:09:18.4622475Z May 10 03:09:18at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2021-05-10T03:09:18.4623142Z May 10 03:09:18at 
java.lang.reflect.Method.invoke(Method.java:498)
2021-05-10T03:09:18.4623783Z May 10 03:09:18at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2021-05-10T03:09:18.4624514Z May 10 03:09:18at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2021-05-10T03:09:18.4625246Z May 10 03:09:18at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2021-05-10T03:09:18.4625967Z May 10 03:09:18at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2021-05-10T03:09:18.4626671Z May 10 03:09:18at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
2021-05-10T03:09:18.4627349Z May 10 03:09:18at 
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
2021-05-10T03:09:18.4627979Z May 10 03:09:18at 
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2021-05-10T03:09:18.4628582Z May 10 03:09:18at 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2021-05-10T03:09:18.4629251Z May 10 03:09:18at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2021-05-10T03:09:18.4629950Z May 10 03:09:18at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2021-05-10T03:09:18.4630616Z May 10 03:09:18at 
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2021-05-10T03:09:18.4631339Z May 10 03:09:18at 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2021-05-10T03:09:18.4631986Z May 10 03:09:18at 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2021-05-10T03:09:18.4632630Z May 10 03:09:18at 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2021-05-10T03:09:18.4633269Z May 10 03:09:18at 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2021-05-10T03:09:18.4634016Z May 10 03:09:18at 
org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:30)
2021-05-10T03:09:18.4634786Z May 10 03:09:18at 
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
2021-05-10T03:09:18.4635412Z May 10 03:09:18at 
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2021-05-10T03:09:18.4635995Z May 10 03:09:18at 
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2021-05-10T03:09:18.4636656Z May 10 03:09:18at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
2021-05-10T03:09:18.4637398Z May 10 03:09:18at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
2021-05-10T03:09:18.4638141Z May 10 03:09:18at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
2021-05-10T03:09:18.4638869Z May 10 03:09:18at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
2021-05-10T03:09:18.4639619Z May 10 03:09:18at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
2021-05-10T03:09:

[jira] [Created] (FLINK-22548) UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint fail due to IllegalReferenceCountException

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22548:
-

 Summary: 
UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint fail due to 
IllegalReferenceCountException
 Key: FLINK-22548
 URL: https://issues.apache.org/jira/browse/FLINK-22548
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing
Affects Versions: 1.14.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17499=logs=34f41360-6c0d-54d3-11a1-0292a2def1d9=2d56e022-1ace-542f-bf1a-b37dd63243f2=9615


{code:java}
2021-05-02T23:18:10.5449890Z May 02 23:18:10 [ERROR] 
shouldRescaleUnalignedCheckpoint[upscale pipeline from 1 to 
2](org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase)  Time 
elapsed: 2.645 s  <<< ERROR!
2021-05-02T23:18:10.5451234Z May 02 23:18:10 
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
2021-05-02T23:18:10.5452049Z May 02 23:18:10at 
org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
2021-05-02T23:18:10.5453118Z May 02 23:18:10at 
org.apache.flink.test.checkpointing.UnalignedCheckpointTestBase.execute(UnalignedCheckpointTestBase.java:167)
2021-05-02T23:18:10.5454315Z May 02 23:18:10at 
org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint(UnalignedCheckpointRescaleITCase.java:368)
2021-05-02T23:18:10.5455228Z May 02 23:18:10at 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2021-05-02T23:18:10.5499169Z May 02 23:18:10at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2021-05-02T23:18:10.5500258Z May 02 23:18:10at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2021-05-02T23:18:10.5501465Z May 02 23:18:10at 
java.lang.reflect.Method.invoke(Method.java:498)
2021-05-02T23:18:10.5502547Z May 02 23:18:10at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2021-05-02T23:18:10.5503965Z May 02 23:18:10at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2021-05-02T23:18:10.5504867Z May 02 23:18:10at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2021-05-02T23:18:10.5505729Z May 02 23:18:10at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2021-05-02T23:18:10.5506527Z May 02 23:18:10at 
org.junit.rules.Verifier$1.evaluate(Verifier.java:35)
2021-05-02T23:18:10.5507319Z May 02 23:18:10at 
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
2021-05-02T23:18:10.5508119Z May 02 23:18:10at 
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
2021-05-02T23:18:10.5508896Z May 02 23:18:10at 
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
2021-05-02T23:18:10.5509592Z May 02 23:18:10at 
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2021-05-02T23:18:10.5510318Z May 02 23:18:10at 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2021-05-02T23:18:10.5517041Z May 02 23:18:10at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2021-05-02T23:18:10.5518247Z May 02 23:18:10at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2021-05-02T23:18:10.5520181Z May 02 23:18:10at 
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2021-05-02T23:18:10.5520928Z May 02 23:18:10at 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2021-05-02T23:18:10.5521660Z May 02 23:18:10at 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2021-05-02T23:18:10.5522323Z May 02 23:18:10at 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2021-05-02T23:18:10.5523524Z May 02 23:18:10at 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2021-05-02T23:18:10.5524309Z May 02 23:18:10at 
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2021-05-02T23:18:10.5524934Z May 02 23:18:10at 
org.junit.runners.Suite.runChild(Suite.java:128)
2021-05-02T23:18:10.5525581Z May 02 23:18:10at 
org.junit.runners.Suite.runChild(Suite.java:27)
2021-05-02T23:18:10.5526293Z May 02 23:18:10at 
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2021-05-02T23:18:10.5526966Z May 02 23:18:10at 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2021-05-02T23:18:10.5527688Z May 02 23:18:10at 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2021-05-02T23:18:10.5535712Z May 02 23:18:10at 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2021-05-02T23:18:10.5536510Z May 02 23:18:10at 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2021-05-02T23:18:10.5537216Z May 02

[jira] [Created] (FLINK-22547) OperatorCoordinatorHolderTest. verifyCheckpointEventOrderWhenCheckpointFutureCompletesLate fail

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22547:
-

 Summary: OperatorCoordinatorHolderTest. 
verifyCheckpointEventOrderWhenCheckpointFutureCompletesLate fail
 Key: FLINK-22547
 URL: https://issues.apache.org/jira/browse/FLINK-22547
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing
Affects Versions: 1.14.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17499=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=05b74a19-4ee4-5036-c46f-ada307df6cf0=7502



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22546) testKafkaSourceSinkWithKeyAndFullValue failed

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22546:
-

 Summary: testKafkaSourceSinkWithKeyAndFullValue failed
 Key: FLINK-22546
 URL: https://issues.apache.org/jira/browse/FLINK-22546
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Ecosystem
Affects Versions: 1.13.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17502=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5=6686



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22545) JVM crashes when runing OperatorEventSendingCheckpointITCase.testOperatorEventAckLost

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22545:
-

 Summary: JVM crashes when runing 
OperatorEventSendingCheckpointITCase.testOperatorEventAckLost
 Key: FLINK-22545
 URL: https://issues.apache.org/jira/browse/FLINK-22545
 Project: Flink
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.12.3
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17501=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=a99e99c7-21cd-5a1f-7274-585e62b72f56=4287




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22520) KafkaSourceLegacyITCase.testMultipleSourcesOnePartition hangs

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22520:
-

 Summary: KafkaSourceLegacyITCase.testMultipleSourcesOnePartition 
hangs
 Key: FLINK-22520
 URL: https://issues.apache.org/jira/browse/FLINK-22520
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.14.0
Reporter: Guowei Ma


There is no any error messages.
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17363=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5=42023


{code:java}
"main" #1 prio=5 os_prio=0 tid=0x7f4d3400b000 nid=0x203f waiting on 
condition [0x7f4d3be2e000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0xa68f3b68> (a 
java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at 
java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
at 
java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at 
java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
at 
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at org.apache.flink.test.util.TestUtils.tryExecute(TestUtils.java:49)
at 
org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase.runMultipleSourcesOnePartitionExactlyOnceTest(KafkaConsumerTestBase.java:1112)
at 
org.apache.flink.connector.kafka.source.KafkaSourceLegacyITCase.testMultipleSourcesOnePartition(KafkaSourceLegacyITCase.java:87)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)

{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Release 1.13.0, release candidate #2

[Guowei Ma]

Hi, Matthias

Thank you very much for your careful inspection.
I check the flink-python_2.11-1.13.0.jar and we do not bundle
org.conscrypt:conscrypt-openjdk-uber:2.5.1 to it.
So I think we may not need to add this to the NOTICE file. (BTW The jar's
scope is runtime)

Best,
Guowei


On Thu, Apr 29, 2021 at 2:33 AM Matthias Pohl 
wrote:

> Thanks Dawid and Guowei for managing this release.
>
> - downloaded the sources and binaries and checked the checksums
> - built Flink from the downloaded sources
> - executed example jobs with standalone deployments - I didn't find
> anything suspicious in the logs
> - reviewed release announcement pull request
>
> - I did a pass over dependency updates: git diff release-1.12.2
> release-1.13.0-rc2 */*.xml
> There's one thing someone should double-check whether that's suppose to be
> like that: We added org.conscrypt:conscrypt-openjdk-uber:2.5.1 as a
> dependency but I don't see it being reflected in the NOTICE file of the
> flink-python module. Or is this automatically added later on?
>
> +1 (non-binding; please see remark on dependency above)
>
> Matthias
>
> On Wed, Apr 28, 2021 at 1:52 PM Stephan Ewen  wrote:
>
> > Glad to hear that outcome. And no worries about the false alarm.
> > Thank you for doing thorough testing, this is very helpful!
> >
> > On Wed, Apr 28, 2021 at 1:04 PM Caizhi Weng 
> wrote:
> >
> > > After the investigation we found that this issue is caused by the
> > > implementation of connector, not by the Flink framework.
> > >
> > > Sorry for the false alarm.
> > >
> > > Stephan Ewen  于2021年4月28日周三 下午3:23写道：
> > >
> > > > @Caizhi and @Becket - let me reach out to you to jointly debug this
> > > issue.
> > > >
> > > > I am wondering if there is some incorrect reporting of failed events?
> > > >
> > > > On Wed, Apr 28, 2021 at 8:53 AM Caizhi Weng 
> > > wrote:
> > > >
> > > > > -1
> > > > >
> > > > > We're testing this version on batch jobs with large (600~1000)
> > > > parallelisms
> > > > > and the following exception messages appear with high frequency:
> > > > >
> > > > > 2021-04-27 21:27:26
> > > > > org.apache.flink.util.FlinkException: An OperatorEvent from an
> > > > > OperatorCoordinator to a task was lost. Triggering task failover to
> > > > ensure
> > > > > consistency. Event: '[NoMoreSplitEvent]', targetTask:  -
> > > > > execution #0
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.operators.coordination.SubtaskGatewayImpl.lambda$sendEvent$0(SubtaskGatewayImpl.java:81)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:440)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:208)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158)
> > > > > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> > > > > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> > > > > at
> scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> > > > > at akka.japi.pf
> > .UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> > > > > at
> > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
> > > > > at
> > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> > > > > at
> > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> > > > > at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
> > > > > at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
> > > > > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
> > > > > at akka.actor.ActorCell.invoke(ActorCell.scala:561)
> > > > > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
> > > > > at akka.dispatch.Mailbox.run(Mailbox.scala:225)
> > > > > at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
> > > > > at
> akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> > > > > at
> > > akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> > > > >
> > > > > Becket Qin is investigating this 

[jira] [Created] (FLINK-22496) ClusterEntrypointTest.testCloseAsyncShouldBeExecutedInShutdownHook failed

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22496:
-

 Summary: 
ClusterEntrypointTest.testCloseAsyncShouldBeExecutedInShutdownHook failed
 Key: FLINK-22496
 URL: https://issues.apache.org/jira/browse/FLINK-22496
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.14.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17313=logs=21408240-6569-5a01-c099-3adfe83ce651=b2761bb8-3852-5a0d-bc43-6a1d327b63cb=6207


{code:java}
52 [ERROR] 
testCloseAsyncShouldBeExecutedInShutdownHook(org.apache.flink.runtime.entrypoint.ClusterEntrypointTest)
  Time elapsed: 9.83 s  <<< FAILURE!
Apr 27 21:20:52 java.lang.AssertionError: 
Apr 27 21:20:52 Process 843 does not exit within 3000 ms
Apr 27 21:20:52 Expected: is 
Apr 27 21:20:52  but: was 
Apr 27 21:20:52 at 
org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
Apr 27 21:20:52 at org.junit.Assert.assertThat(Assert.java:956)
Apr 27 21:20:52 at 
org.apache.flink.runtime.entrypoint.ClusterEntrypointTest.testCloseAsyncShouldBeExecutedInShutdownHook(ClusterEntrypointTest.java:224)
Apr 27 21:20:52 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
Apr 27 21:20:52 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Apr 27 21:20:52 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Apr 27 21:20:52 at java.lang.reflect.Method.invoke(Method.java:498)
Apr 27 21:20:52 at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
Apr 27 21:20:52 at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Apr 27 21:20:52 at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
Apr 27 21:20:52 at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
Apr 27 21:20:52 at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
Apr 27 21:20:52 at 
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
Apr 27 21:20:52 at 
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
Apr 27 21:20:52 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
Apr 27 21:20:52 at 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
Apr 27 21:20:52 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
Apr 27 21:20:52 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
Apr 27 21:20:52 at 
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
Apr 27 21:20:52 at 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
Apr 27 21:20:52 at 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
Apr 27 21:20:52 at 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
Apr 27 21:20:52 at 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
Apr 27 21:20:52 at 
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
Apr 27 21:20:52 at 
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
Apr 27 21:20:52 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
Apr 27 21:20:52 at 
org.junit.runners.ParentRunner.run(ParentRunner.java:363)

{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22466) KafkaSourceLegacyITCase.testOneToOneSources fail because the OperatorEvent lost

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22466:
-

 Summary: KafkaSourceLegacyITCase.testOneToOneSources fail because 
the OperatorEvent lost
 Key: FLINK-22466
 URL: https://issues.apache.org/jira/browse/FLINK-22466
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.13.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17110=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5=7010

{code:java}
2021-04-23T14:31:37.1620668Z Apr 23 14:31:37 [INFO] Running 
org.apache.flink.connector.kafka.source.KafkaSourceLegacyITCase
2021-04-23T14:32:27.0398155Z java.util.concurrent.ExecutionException: 
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
2021-04-23T14:32:27.0400673Zat 
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
2021-04-23T14:32:27.0401550Zat 
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
2021-04-23T14:32:27.0402365Zat 
org.apache.flink.test.util.TestUtils.tryExecute(TestUtils.java:49)
2021-04-23T14:32:27.0403227Zat 
org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase.runOneToOneExactlyOnceTest(KafkaConsumerTestBase.java:1009)
2021-04-23T14:32:27.0403937Zat 
org.apache.flink.connector.kafka.source.KafkaSourceLegacyITCase.testOneToOneSources(KafkaSourceLegacyITCase.java:77)
2021-04-23T14:32:27.0404881Zat 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2021-04-23T14:32:27.0405293Zat 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2021-04-23T14:32:27.0406792Zat 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2021-04-23T14:32:27.0407333Zat 
java.lang.reflect.Method.invoke(Method.java:498)
2021-04-23T14:32:27.0407743Zat 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2021-04-23T14:32:27.0408318Zat 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2021-04-23T14:32:27.0408784Zat 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2021-04-23T14:32:27.0409246Zat 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2021-04-23T14:32:27.0409742Zat 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
2021-04-23T14:32:27.0410251Zat 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
2021-04-23T14:32:27.0410727Zat 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
2021-04-23T14:32:27.0411065Zat java.lang.Thread.run(Thread.java:748)
2021-04-23T14:32:27.0411430Z Caused by: 
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
2021-04-23T14:32:27.0411931Zat 
org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
2021-04-23T14:32:27.0412631Zat 
org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
2021-04-23T14:32:27.0413144Zat 
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
2021-04-23T14:32:27.0413605Zat 
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
2021-04-23T14:32:27.0414063Zat 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-04-23T14:32:27.0414497Zat 
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
2021-04-23T14:32:27.0415002Zat 
org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:237)
2021-04-23T14:32:27.0415526Zat 
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
2021-04-23T14:32:27.0416026Zat 
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
2021-04-23T14:32:27.0416498Zat 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-04-23T14:32:27.0417403Zat 
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
2021-04-23T14:32:27.0417945Zat 
org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:1081)
2021-04-23T14:32:27.0418481Zat 
akka.dispatch.OnComplete.internal(Future.scala:264)
2021-04-23T14:32:27.0418820Zat 
akka.dispatch.OnComplete.internal(Future.scala:261)
2021-04-23T14:32:27.0419188Zat 
akka.dispatch.japi$CallbackBridge.apply(Future.scala:191)
2021-04-23T14:32:27.0419574Zat 
akka.dispatch.japi$CallbackBridge.apply(Future.scala:188)
2021-04-23T14:32:27.0419956Zat 
scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
2021-04-23T14:32:27.0420414Zat 
org.apache.flink.runtime.concurrent.Executors

[jira] [Created] (FLINK-22465) KafkaSourceITCase.testValueOnlyDeserializer hangs

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22465:
-

 Summary: KafkaSourceITCase.testValueOnlyDeserializer hangs
 Key: FLINK-22465
 URL: https://issues.apache.org/jira/browse/FLINK-22465
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.14.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17104=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5=28977


{code:java}

at 
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106)
at 
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
at java.util.Iterator.forEachRemaining(Iterator.java:115)
at 
org.apache.flink.connector.kafka.source.KafkaSourceITCase.testValueOnlyDeserializer(KafkaSourceITCase.java:155)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.jav
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22462) JdbcExactlyOnceSinkE2eTest.testInsert failed because of too many clients.

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22462:
-

 Summary: JdbcExactlyOnceSinkE2eTest.testInsert failed because of 
too many clients.
 Key: FLINK-22462
 URL: https://issues.apache.org/jira/browse/FLINK-22462
 Project: Flink
  Issue Type: Bug
  Components: Connectors / JDBC
Affects Versions: 1.14.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17178=logs=ba53eb01-1462-56a3-8e98-0dd97fbcaab5=bfbc6239-57a0-5db0-63f3-41551b4f7d51=13514


{code:java}
Apr 25 23:05:31 [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time 
elapsed: 138.743 s <<< FAILURE! - in 
org.apache.flink.connector.jdbc.xa.JdbcExactlyOnceSinkE2eTest
Apr 25 23:05:31 [ERROR] 
testInsert(org.apache.flink.connector.jdbc.xa.JdbcExactlyOnceSinkE2eTest)  Time 
elapsed: 137.267 s  <<< ERROR!
Apr 25 23:05:31 org.postgresql.util.PSQLException: FATAL: sorry, too many 
clients already
Apr 25 23:05:31 at 
org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:524)
Apr 25 23:05:31 at 
org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:145)
Apr 25 23:05:31 at 
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:196)
Apr 25 23:05:31 at 
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
Apr 25 23:05:31 at 
org.postgresql.jdbc.PgConnection.(PgConnection.java:211)
Apr 25 23:05:31 at org.postgresql.Driver.makeConnection(Driver.java:459)
Apr 25 23:05:31 at org.postgresql.Driver.connect(Driver.java:261)
Apr 25 23:05:31 at 
java.sql.DriverManager.getConnection(DriverManager.java:664)
Apr 25 23:05:31 at 
java.sql.DriverManager.getConnection(DriverManager.java:247)
Apr 25 23:05:31 at 
org.apache.flink.connector.jdbc.xa.JdbcXaFacadeTestHelper.getInsertedIds(JdbcXaFacadeTestHelper.java:81)
Apr 25 23:05:31 at 
org.apache.flink.connector.jdbc.xa.JdbcExactlyOnceSinkE2eTest.testInsert(JdbcExactlyOnceSinkE2eTest.java:119)
Apr 25 23:05:31 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)

{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22457) KafkaSourceLegacyITCase.testMultipleSourcesOnePartition fails because of timeout

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22457:
-

 Summary: KafkaSourceLegacyITCase.testMultipleSourcesOnePartition 
fails because of timeout
 Key: FLINK-22457
 URL: https://issues.apache.org/jira/browse/FLINK-22457
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.14.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17140=logs=1fc6e7bf-633c-5081-c32a-9dea24b05730=80a658d1-f7f6-5d93-2758-53ac19fd5b19=7045


{code:java}
Apr 24 23:47:33 [ERROR] Tests run: 21, Failures: 0, Errors: 1, Skipped: 0, Time 
elapsed: 174.335 s <<< FAILURE! - in 
org.apache.flink.connector.kafka.source.KafkaSourceLegacyITCase
Apr 24 23:47:33 [ERROR] 
testMultipleSourcesOnePartition(org.apache.flink.connector.kafka.source.KafkaSourceLegacyITCase)
  Time elapsed: 60.019 s  <<< ERROR!
Apr 24 23:47:33 org.junit.runners.model.TestTimedOutException: test timed out 
after 6 milliseconds
Apr 24 23:47:33 at sun.misc.Unsafe.park(Native Method)
Apr 24 23:47:33 at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
Apr 24 23:47:33 at 
java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
Apr 24 23:47:33 at 
java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
Apr 24 23:47:33 at 
java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
Apr 24 23:47:33 at 
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
Apr 24 23:47:33 at 
org.apache.flink.test.util.TestUtils.tryExecute(TestUtils.java:49)
Apr 24 23:47:33 at 
org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase.runMultipleSourcesOnePartitionExactlyOnceTest(KafkaConsumerTestBase.java:1112)
Apr 24 23:47:33 at 
org.apache.flink.connector.kafka.source.KafkaSourceLegacyITCase.testMultipleSourcesOnePartition(KafkaSourceLegacyITCase.java:87)
Apr 24 23:47:33 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
Apr 24 23:47:33 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Apr 24 23:47:33 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Apr 24 23:47:33 at java.lang.reflect.Method.invoke(Method.java:498)
Apr 24 23:47:33 at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
Apr 24 23:47:33 at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Apr 24 23:47:33 at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
Apr 24 23:47:33 at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
Apr 24 23:47:33 at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
Apr 24 23:47:33 at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
Apr 24 23:47:33 at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
Apr 24 23:47:33 at java.lang.Thread.run(Thread.java:748)

{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22440) UpsertKafkaTableITCase fail due to create topic timeout

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22440:
-

 Summary: UpsertKafkaTableITCase fail due to create topic timeout
 Key: FLINK-22440
 URL: https://issues.apache.org/jira/browse/FLINK-22440
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.13.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17132=logs=b0097207-033c-5d9a-b48c-6d4796fbe60d=e8fcc430-213e-5cce-59d4-6942acf09121=6594



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22439) UnalignedCheckpointITCase hangs

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22439:
-

 Summary: UnalignedCheckpointITCase hangs
 Key: FLINK-22439
 URL: https://issues.apache.org/jira/browse/FLINK-22439
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing
Affects Versions: 1.14.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17120=logs=5c8e7682-d68f-54d1-16a2-a09310218a49=f508e270-48d6-5f1e-3138-42a17e0714f0=11771



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22420) UnalignedCheckpointITCase failed

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22420:
-

 Summary: UnalignedCheckpointITCase failed
 Key: FLINK-22420
 URL: https://issues.apache.org/jira/browse/FLINK-22420
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing
Affects Versions: 1.14.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17052=logs=34f41360-6c0d-54d3-11a1-0292a2def1d9=2d56e022-1ace-542f-bf1a-b37dd63243f2=9442

As described in the comment 
https://issues.apache.org/jira/browse/FLINK-21996?focusedCommentId=17326449=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17326449
 we might need to adjust the tests  to allow failover.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Re: Re: [ANNOUNCE] New Apache Flink Committer - Rui Li

[Guowei Ma]

Congratulations, Rui!
Best,
Guowei


On Fri, Apr 23, 2021 at 10:38 AM Yun Tang  wrote:

> Congratulations, Rui!
>
> Best,
> Yun Tang
> 
> From: Xuannan Su 
> Sent: Friday, April 23, 2021 10:01
> To: dev@flink.apache.org ; matth...@ververica.com <
> matth...@ververica.com>
> Subject: Re: Re: Re: [ANNOUNCE] New Apache Flink Committer - Rui Li
>
> Congratulations Rui!
>
> Best,
> Xuannan
> On Apr 22, 2021, 6:23 PM +0800, M Shengkai Fang ,
> wrote:
> >
> > Congratulations Rui!
>

[jira] [Created] (FLINK-22363) Python test pipeline failed because of failure of downloading deps.

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22363:
-

 Summary: Python test pipeline failed because of failure of 
downloading deps. 
 Key: FLINK-22363
 URL: https://issues.apache.org/jira/browse/FLINK-22363
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Affects Versions: 1.13.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16780=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=8d78fe4f-d658-5c70-12f8-4921589024c3=21193

{code:java}

Apr 19 11:39:38 ERROR: could not install deps [pytest, apache-beam==2.27.0, 
cython==0.29.16, grpcio>=1.17.0,<=1.26.0, grpcio-tools>=1.3.5,<=1.14.2, 
apache-flink-libraries]; v = 
InvocationError("/__w/2/s/flink-python/dev/install_command.sh pytest 
apache-beam==2.27.0 cython==0.29.16 'grpcio>=1.17.0,<=1.26.0' 
'grpcio-tools>=1.3.5,<=1.14.2' apache-flink-libraries", 1)
Apr 19 11:39:38 ___ summary 

Apr 19 11:39:38 ERROR:   py38: could not install deps [pytest, 
apache-beam==2.27.0, cython==0.29.16, grpcio>=1.17.0,<=1.26.0, 
grpcio-tools>=1.3.5,<=1.14.2, apache-flink-libraries]; v = 
InvocationError("/__w/2/s/flink-python/dev/install_command.sh pytest 
apache-beam==2.27.0 cython==0.29.16 'grpcio>=1.17.0,<=1.26.0' 
'grpcio-tools>=1.3.5,<=1.14.2' apache-flink-libraries", 1)
Apr 19 11:39:38 tox checks... [FAILED]
Apr 19 11:39:38 Process exited with EXIT CODE: 1.
Apr 19 11:39:38 Trying to KILL watchdog (3373).

{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[VOTE] Release 1.13.0, release candidate #1

[Guowei Ma]

Hi everyone,

Currently there are still some on-going efforts for 1.13.0. However we
think the current state is stable enough for the community to test. The
earlier the test, the more time we can fix unexpected issues if there is
any.

We also cut out the release-1.13 branch, so any fix needs to be submitted
to both the master and release-1.13.

Please review and vote on the release candidate #1 for the version 1.13.0,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:
* the official Apache source release and binary convenience releases to be
deployed to dist.apache.org [1], which are signed with the key with
fingerprint D7C86B9C[2],
* all artifacts to be deployed to the Maven Central Repository [3],
* source code tag "release-1.13.0-rc1" [4],

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Your help testing the release will be greatly appreciated!


[1] https://dist.apache.org/repos/dist/dev/flink/flink-1.13.0-rc1/
[2] https://dist.apache.org/repos/dist/release/flink/KEYS
[3] https://repository.apache.org/content/repositories/orgapacheflink-1418/
[4] https://github.com/apache/flink/tree/release-1.13.0-rc1

Best,
Guowei

[jira] [Created] (FLINK-22333) Elasticsearch7DynamicSinkITCase.testWritingDocuments fail due to deploy task timeout.

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22333:
-

 Summary: Elasticsearch7DynamicSinkITCase.testWritingDocuments fail 
due to deploy task timeout.
 Key: FLINK-22333
 URL: https://issues.apache.org/jira/browse/FLINK-22333
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.13.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16694=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=03dca39c-73e8-5aaf-601d-328ae5c35f20=12329


{code:java}
2021-04-16T23:37:23.5719280Z Apr 16 23:37:23 
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
2021-04-16T23:37:23.5739250Z Apr 16 23:37:23at 
org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
2021-04-16T23:37:23.5759329Z Apr 16 23:37:23at 
org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
2021-04-16T23:37:23.5779145Z Apr 16 23:37:23at 
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
2021-04-16T23:37:23.5799204Z Apr 16 23:37:23at 
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
2021-04-16T23:37:23.5819302Z Apr 16 23:37:23at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-04-16T23:37:23.5839106Z Apr 16 23:37:23at 
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
2021-04-16T23:37:23.5859276Z Apr 16 23:37:23at 
org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:237)
2021-04-16T23:37:23.5868964Z Apr 16 23:37:23at 
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
2021-04-16T23:37:23.5869925Z Apr 16 23:37:23at 
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
2021-04-16T23:37:23.5919839Z Apr 16 23:37:23at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-04-16T23:37:23.5959562Z Apr 16 23:37:23at 
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
2021-04-16T23:37:23.5989732Z Apr 16 23:37:23at 
org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:1081)
2021-04-16T23:37:23.6019422Z Apr 16 23:37:23at 
akka.dispatch.OnComplete.internal(Future.scala:264)
2021-04-16T23:37:23.6039067Z Apr 16 23:37:23at 
akka.dispatch.OnComplete.internal(Future.scala:261)
2021-04-16T23:37:23.6060126Z Apr 16 23:37:23at 
akka.dispatch.japi$CallbackBridge.apply(Future.scala:191)
2021-04-16T23:37:23.6089258Z Apr 16 23:37:23at 
akka.dispatch.japi$CallbackBridge.apply(Future.scala:188)
2021-04-16T23:37:23.6119150Z Apr 16 23:37:23at 
scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
2021-04-16T23:37:23.6139149Z Apr 16 23:37:23at 
org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:73)
2021-04-16T23:37:23.6159077Z Apr 16 23:37:23at 
scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
2021-04-16T23:37:23.6189432Z Apr 16 23:37:23at 
scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
2021-04-16T23:37:23.6215243Z Apr 16 23:37:23at 
akka.pattern.PromiseActorRef.$bang(AskSupport.scala:572)
2021-04-16T23:37:23.6219148Z Apr 16 23:37:23at 
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:22)
2021-04-16T23:37:23.6220221Z Apr 16 23:37:23at 
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:21)
2021-04-16T23:37:23.6249411Z Apr 16 23:37:23at 
scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:436)
2021-04-16T23:37:23.6259145Z Apr 16 23:37:23at 
scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:435)
2021-04-16T23:37:23.6289272Z Apr 16 23:37:23at 
scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
2021-04-16T23:37:23.6309243Z Apr 16 23:37:23at 
akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
2021-04-16T23:37:23.6359306Z Apr 16 23:37:23at 
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
2021-04-16T23:37:23.6369399Z Apr 16 23:37:23at 
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
2021-04-16T23:37:23.6389444Z Apr 16 23:37:23at 
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
2021-04-16T23:37:23.6429180Z Apr 16 23:37:23at 
scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
2021-04-16T23:37:23.6449179Z Apr 16 23:37:23at 
akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
2021-04-16T23:37:23.6479350Z Apr 16 23:37:23

[jira] [Created] (FLINK-22332) ConnectedComponentsWithObjectMapITCase.testJobWithoutObjectReuse due to NPE when calling "notifyDataAvailable"

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22332:
-

 Summary: 
ConnectedComponentsWithObjectMapITCase.testJobWithoutObjectReuse due to NPE 
when  calling "notifyDataAvailable"
 Key: FLINK-22332
 URL: https://issues.apache.org/jira/browse/FLINK-22332
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Network
Affects Versions: 1.11.3
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16711=logs=219e462f-e75e-506c-3671-5017d866ccf6=94b2a454-a9e3-5226-421d-758b172639ef=4476



--
This message was sent by Atlassian Jira
(v8.3.4#803005)