[VOTE] FLIP-147: Support Checkpoint After Tasks Finished

Hi all, I would like to start the vote for FLIP-147[1], which propose to support checkpoints after tasks finished and is discussed in [2]. The vote will last at least 72 hours (Jan 20th due to weekend), following the consensus voting process. thanks, Yun [1] https://cwiki.apache.org/conflu

[VOTE] FLIP-147: Support Checkpoint After Tasks Finished

Hi all, For FLIP-147[1] which targets at supports checkpoints after tasks finished and modify operator API and implementation to ensures the commit of last piece of data, since after the last vote we have more discussions[2][3] and a few updates, including changes to PublicEvolving API, I'd l

Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished

+1 non-binding Best, Guowei On Fri, Jan 15, 2021 at 10:56 PM Yun Gao wrote: > > Hi all, > > I would like to start the vote for FLIP-147[1], which propose to support > checkpoints after > tasks finished and is discussed in [2]. > > The vote will last at least 72 hours (Jan 20th due to weekend),

Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished

+1 (non-binding) Thanks for driving the FLIP! Regards, Roman On Tue, Jan 19, 2021 at 2:58 AM Guowei Ma wrote: > +1 non-binding > Best, > Guowei > > > On Fri, Jan 15, 2021 at 10:56 PM Yun Gao > wrote: > > > > > Hi all, > > > > I would like to start the vote for FLIP-147[1], which propose to s

Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished

+1 (binding) Best, Aljoscha On 2021/01/15 22:55, Yun Gao wrote: Hi all, I would like to start the vote for FLIP-147[1], which propose to support checkpoints after tasks finished and is discussed in [2]. The vote will last at least 72 hours (Jan 20th due to weekend), following the consensus

Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished

+1 (binding) Best, Arvid On Wed, Jan 20, 2021 at 12:13 PM Aljoscha Krettek wrote: > +1 (binding) > > Best, > Aljoscha > > On 2021/01/15 22:55, Yun Gao wrote: > > > >Hi all, > > > >I would like to start the vote for FLIP-147[1], which propose to support > checkpoints after > >tasks finished and

Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished

+1 (binding) Best, Dawid On 28/06/2021 10:45, Yun Gao wrote: > Hi all, > > For FLIP-147[1] which targets at supports checkpoints after tasks finished > and modify operator > API and implementation to ensures the commit of last piece of data, since > after the last vote > we have more discussi

Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished

+1 (binding) Piotrek pon., 28 cze 2021 o 12:48 Dawid Wysakowicz napisał(a): > +1 (binding) > > Best, > > Dawid > > On 28/06/2021 10:45, Yun Gao wrote: > > Hi all, > > > > For FLIP-147[1] which targets at supports checkpoints after tasks > finished and modify operator > > API and implementation

Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished

+1 (binding) Best liujiangang Piotr Nowojski 于2021年6月29日周二 上午2:05写道： > +1 (binding) > > Piotrek > > pon., 28 cze 2021 o 12:48 Dawid Wysakowicz > napisał(a): > > > +1 (binding) > > > > Best, > > > > Dawid > > > > On 28/06/2021 10:45, Yun Gao wrote: > > > Hi all, > > > > > > For FLIP-147[1] whic

Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished

Looks good: +1 (binding) On Tue, Jun 29, 2021 at 5:06 AM 刘建刚 wrote: > +1 (binding) > > Best > liujiangang > > Piotr Nowojski 于2021年6月29日周二 上午2:05写道： > > > +1 (binding) > > > > Piotrek > > > > pon., 28 cze 2021 o 12:48 Dawid Wysakowicz > > napisał(a): > > > > > +1 (binding) > > > > > > Best, >

Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished

+1 (binding) Arvid Heise 于2021年7月1日周四 下午3:10写道： > Looks good: +1 (binding) > > On Tue, Jun 29, 2021 at 5:06 AM 刘建刚 wrote: > > > +1 (binding) > > > > Best > > liujiangang > > > > Piotr Nowojski 于2021年6月29日周二 上午2:05写道： > > > > > +1 (binding) > > > > > > Piotrek > > > > > > pon., 28 cze 2021 o 1

Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished

+1, This may help https://issues.apache.org/jira/browse/FLINK-22764 Best, Leonard > 在 2021年7月2日，10:17，JING ZHANG 写道： > > +1 (binding) > > > Arvid Heise 于2021年7月1日周四 下午3:10写道： > >> Looks good: +1 (binding) >> >> On Tue, Jun 29, 2021 at 5

Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished

+1 (binding) Best, Guowei On Fri, Jul 2, 2021 at 11:56 AM Leonard Xu wrote: > +1, > > This may help https://issues.apache.org/jira/browse/FLINK-22764 < > https://issues.apache.org/jira/browse/FLINK-22764> > > Best, > Leonard > > > 在 2021年7月2日，10:17，JING ZHANG 写道： > > > > +1 (binding) > > > > >

[RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

Hi there, Since the voting time of FLIP-147[1] has passed, I'm closing the vote now. There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes: - Dawid Wysakowicz (binding) - Piotr Nowojski(binding) - Jiangang Liu (binding) - Arvid Heise (binding) - Jing Zhang (binding) - Leonard Xu (non-b

Re: Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished

nding) * Aljoscha (binding) * Arvid (binding) Very thanks everyone for feedbacks and participating! Best, Yun -- Sender:Arvid Heise Date:2021/01/20 19:15:26 Recipient:dev Theme:Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finishe

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

Hi everyone, I am a bit late to the voting party but let me ask three questions: 1) Why do we execute the trigger plan computation in the main thread if we cannot guarantee that all tasks are still running when triggering the checkpoint? Couldn't we do the computation in a different thread in ord

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

t. Best, Yun -- From:Till Rohrmann Send Time:2021 Jul. 14 (Wed.) 22:05 To:dev Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished Hi everyone, I am a bit late to the voting party but let me ask three questions: 1) Why do we execut

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

de enough details for this part and I'll > complement the FLIP with the details on > the process of the final checkpoint / savepoint. > > Best, > Yun > > > > -------------- > From:Till Rohrmann > Send Time:2021 Ju

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

>> Very sorry for not include enough details for this part and I'll >> complement the FLIP with the details on >> the process of the final checkpoint / savepoint. >> >> Best, >> Yun >> >> >> >>

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

consideration of the tradeoff > >> between the performance and > >> the complexity ? > >> > >> 3) We are going to emit the EndOfInput event exactly after the finish() > >> method and before the last > >> snapshotState() method so that we could shut down the whole topology > with > >> a

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

mputation based on the last triggered tasks, we may not > > >> easily encode this information > > >> into the shortcuts on the job vertex level. And since the time seems > to > > >> be short, perhaps it is enough > > >> to do re-computation from the scratch in consideration of the tradeoff > > >

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

finish(). Best, Yun [1] https://issues.apache.org/jira/browse/FLINK-21132 -- From:Piotr Nowojski Send Time:2021 Jul. 16 (Fri.) 13:48 To:dev Cc:Yun Gao Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finish

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

Jul. 16 (Fri.) 16:00 To:dev Cc:Yun Gao Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished I think we should try to sort this out because it might affect how and when finish() will be called (or in general how the operator lifecycle looks like). To give an exampl

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

> there are no other concerns I'll update the FLIP with the above conclusions > to make it precise in this part. > > > Best, > Yun > > -------------- > From:Till Rohrmann > Send Time:2021 Jul. 16 (Fri.) 16:00 &

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

ould work. >> >> Very sorry for not making the whole thing clear in the FLIP again, if >> there are no other concerns I'll update the FLIP with the above conclusions >> to make it precise in this part. >> >> >> Best, >> Yun >> >> -

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

events would only be sent once for each task ? Best, Yun -- From:Till Rohrmann Send Time:2021 Jul. 16 (Fri.) 17:26 To:Yun Gao Cc:dev Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished Ok, so the plan is that finish() will flush all pending events and then se

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

n do the > > >> computation, both the two step do not need to be in the main thread. > > >> > > >> 2) For the computation logic, in fact currently we benefit a lot from > > >> some shortcuts on all-to-all > > >> edges and job vertex with all ta

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

ckpoint or a savepoint. Best, Yun -- From:Till Rohrmann Send Time:2021 Jul. 16 (Fri.) 23:05 To:Yun Gao Cc:dev Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished Thanks for your responses Dawid and Yun,

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

to the shortcuts on the job vertex level. And since the time seems to > >> be short, perhaps it is enough > >> to do re-computation from the scratch in consideration of the tradeoff > >> between the performance and > >> the complexity ? > >> > >

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

> |||}| > |}| > | > | > > I think we should still keep the change ?  > > Best, > Yun > > --Original Mail -- > *Sender:*Dawid Wysakowicz > *Send Date:*Sun Jul 18 18:44:50 2021 > *Recipients:*Flink Dev , Yun Gao >

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

nceof CheckpointListener && > ((CheckpointListener) userFunction).requiresFinalCheckpoint(); > } > } > > > I think we should still keep the change ? > > Best, > Yun > > --------------Original Mail ------ > *Sender:*Dawid Wysakow

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

>>>> As discussed above for stop-with-savepoint, we would always keep the >>> window as is, and restore them after restart. >>> >>>> Then for the finish / stop-with-savepoint --drain, I think perhaps it >>> depends on the Triggers. For >>> >>>> event-time triggers / p

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

gt; return true; >>> } >>> } >>> >>> interface CheckpointListener { >>> >>> default boolean requiresFinalCheckpoint() { >>> return true; >>> } >>> } >>> >>> class Abstract

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

Wysakowicz Send Time:2021 Jul. 19 (Mon.) 17:51 To:dev ; Till Rohrmann Cc:Yun Gao ; Yun Gao Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished Small correction. I meant we need to adjust the EndOfInputEvent of course. Best, Dawid On 19/07/2021 11:48, Dawid Wysakowicz wrote

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

version.  > > Best > Yun > > -- > From:Dawid Wysakowicz > Send Time:2021 Jul. 19 (Mon.) 17:51 > To:dev ; Till Rohrmann > Cc:Yun Gao ; Yun Gao > > Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

-pc, we might skip waiting for > >>> the > > >>> final checkpoint (but we could not skip the savepoint). To allow users to > >>> express > >>> the logic, we have proposed to add one more method to StreamOperator & > >>> CheckpointLis

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

gt;>> Dawid > > >>> On 19/07/2021 10:31, Yun Gao wrote: > > >>> > > >>> Hi, > > >>> > > >>> Very thanks Dawid for the thoughts! > > >>> > > >>> Currently I also do not have different opi

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

kpoints do not leave lingering > > > external resources. > > > > > > *Checkpointing from a single subtask / UnionListState case* > > > There are operators that checkpoint from a single subtask only. Usually > > > from the subtask index=0. If we let

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

gt; > should > > > be > > > > >>>> > > > > >>>> in control of the StreamTask, by firing timers or by processing > a > > > next > > > > >>>> > > > > >>>> record from upstream. > &

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

-- From:Piotr Nowojski Send Time:2021 Jul. 22 (Thu.) 02:46 To:dev Cc:Yun Gao ; Till Rohrmann ; Yun Gao ; Piotr Nowojski Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished Hi Steven, > I probably missed sth h

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

e > https://issues.apache.org/jira/browse/FLINK-21080. > > Best, > Yun > > > -- > From:Piotr Nowojski > Send Time:2021 Jul. 22 (Thu.) 02:46 > To:dev > Cc:Yun Gao ; Till Rohrmann ; > Yun Gao ; Pio

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

Nowojski Send Time:2021 Jul. 22 (Thu.) 15:33 To:dev ; Yun Gao Cc:Till Rohrmann ; Yun Gao ; Piotr Nowojski Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished Hi Guowei, > Thank Dawid and Piotr for sharing the problem. +1 to EndInput/Finish can > becalled repeated

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

s. > > Very thanks for the careful checks on the whole process. > > Best, > Yun > > > ------------------ > From:Piotr Nowojski > Send Time:2021 Jul. 22 (Thu.) 15:33 > To:dev ; Yun Gao > Cc:Till Rohrmann ; Yun Gao

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

Send Time:2021 Jul. 22 (Thu.) 15:33 To:dev ; Yun Gao Cc:Till Rohrmann ; Yun Gao ; Piotr Nowojski Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished Hi Guowei, > Thank Dawid and Piotr for sharing the problem. +1 to EndInput/Finish can > becalled repeated

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

Time:2021 Jul. 22 (Thu.) 17:57 > To:Yun Gao > Cc:Piotr Nowojski ; dev ; Yun Gao > ; Piotr Nowojski > Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished > > Thanks everyone for this discussion. I think this is very helpful. > > I do agree with Piotr

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

-- > > From:Till Rohrmann > > Send Time:2021 Jul. 22 (Thu.) 17:57 > > To:Yun Gao > > Cc:Piotr Nowojski ; dev ; > Yun Gao ; Piotr Nowojski < > pi...@ververica.com> > > Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks > Finished > >

Re: Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

userFunction).requiresFinalCheckpoint(); } } I think we should still keep the change ? Best, Yun --Original Mail -- Sender:Dawid Wysakowicz Send Date:Sun Jul 18 18:44:50 2021 Recipients:Flink Dev , Yun Gao Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Task