t; From:Till Rohrmann
> > Send Time:2021 Jul. 22 (Thu.) 17:57
> > To:Yun Gao
> > Cc:Piotr Nowojski ; dev ;
> Yun Gao ; Piotr Nowojski <
> pi...@ververica.com>
> > Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks
> Finished
> >
> >
:2021 Jul. 22 (Thu.) 17:57
> To:Yun Gao
> Cc:Piotr Nowojski ; dev ; Yun Gao
> ; Piotr Nowojski
> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
>
> Thanks everyone for this discussion. I think this is very helpful.
>
> I do agree with Piotr's pr
1 Jul. 22 (Thu.) 15:33
To:dev ; Yun Gao
Cc:Till Rohrmann ; Yun Gao
; Piotr Nowojski
Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Hi Guowei,
> Thank Dawid and Piotr for sharing the problem. +1 to EndInput/Finish can
> becalled repeatedly.
Just to cl
ry thanks for the careful checks on the whole process.
>
> Best,
> Yun
>
>
> ----------------------
> From:Piotr Nowojski
> Send Time:2021 Jul. 22 (Thu.) 15:33
> To:dev ; Yun Gao
> Cc:Till Rohrmann ; Yun Gao
> ; Piotr No
jski
Send Time:2021 Jul. 22 (Thu.) 15:33
To:dev ; Yun Gao
Cc:Till Rohrmann ; Yun Gao
; Piotr Nowojski
Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Hi Guowei,
> Thank Dawid and Piotr for sharing the problem. +1 to EndInput/Finish can
> becalled repeat
s.apache.org/jira/browse/FLINK-21080.
>
> Best,
> Yun
>
>
> --
> From:Piotr Nowojski
> Send Time:2021 Jul. 22 (Thu.) 02:46
> To:dev
> Cc:Yun Gao ; Till Rohrmann ;
> Yun Gao ; Piotr Nowojski <
>
--
From:Piotr Nowojski
Send Time:2021 Jul. 22 (Thu.) 02:46
To:dev
Cc:Yun Gao ; Till Rohrmann ; Yun
Gao ; Piotr Nowojski
Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Hi Steven,
> I probably missed sth h
mers or by processing
> a
> > > next
> > > > >>>>
> > > > >>>> record from upstream.
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > >
> > > > >&
> *Checkpointing from a single subtask / UnionListState case*
> > > There are operators that checkpoint from a single subtask only. Usually
> > > from the subtask index=0. If we let those subtasks finish, subsequent
> > > checkpoints will miss this information.
&
;>>
> > >>> Hi,
> > >>>
> > >>> Very thanks Dawid for the thoughts!
> > >>>
> > >>> Currently I also do not have different opinions regarding this part.
> > >>> But I have one more issue to confirm: du
ckpoint (but we could not skip the savepoint). To allow users to
> >>> express
> >>> the logic, we have proposed to add one more method to StreamOperator &
> >>> CheckpointListener:
> >>>
> >>> interface StreamOperator {
> >>
version.
>
> Best
> Yun
>
> --
> From:Dawid Wysakowicz
> Send Time:2021 Jul. 19 (Mon.) 17:51
> To:dev ; Till Rohrmann
> Cc:Yun Gao ; Yun Gao
>
> Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After
Wysakowicz
Send Time:2021 Jul. 19 (Mon.) 17:51
To:dev ; Till Rohrmann
Cc:Yun Gao ; Yun Gao
Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Small correction. I meant we need to adjust the EndOfInputEvent of course.
Best,
Dawid
On 19/07/2021 11:48, Dawid Wysakowicz wrote
t;> }
>>> }
>>>
>>> interface CheckpointListener {
>>>
>>> default boolean requiresFinalCheckpoint() {
>>> return true;
>>> }
>>> }
>>>
>>> class AbstractUdfStreamOperator {
>>>
&g
always keep the
>>> window as is, and restore them after restart.
>>>
>>>> Then for the finish / stop-with-savepoint --drain, I think perhaps it
>>> depends on the Triggers. For
>>>
>>>> event-time triggers / process time triggers, it would be reasonable to
>>> flush all
ner &&
> ((CheckpointListener) userFunction).requiresFinalCheckpoint();
> }
> }
>
>
> I think we should still keep the change ?
>
> Best,
> Yun
>
> ------Original Mail ------
> *Sender:*Dawid Wysakowicz
>
> *Send Dat
> |}|
> |
> |
>
> I think we should still keep the change ?
>
> Best,
> Yun
>
> --Original Mail ------
> *Sender:*Dawid Wysakowicz
> *Send Date:*Sun Jul 18 18:44:50 2021
> *Recipients:*Flink Dev , Yun Gao
>
> *Subj
userFunction).requiresFinalCheckpoint();
}
}
I think we should still keep the change ?
Best,
Yun
--Original Mail --
Sender:Dawid Wysakowicz
Send Date:Sun Jul 18 18:44:50 2021
Recipients:Flink Dev , Yun Gao
Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Task
s to
> >> be short, perhaps it is enough
> >> to do re-computation from the scratch in consideration of the tradeoff
> >> between the performance and
> >> the complexity ?
> >>
> >> 3) We are going to emit the EndOfInput event exactly after t
ckpoint or a savepoint.
Best,
Yun
--
From:Till Rohrmann
Send Time:2021 Jul. 16 (Fri.) 23:05
To:Yun Gao
Cc:dev
Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Thanks for your responses Dawid and Yun,
; > >>
> > >> 2) For the computation logic, in fact currently we benefit a lot from
> > >> some shortcuts on all-to-all
> > >> edges and job vertex with all tasks running, these shortcuts could do
> > >> checks on the job vertex level
> > &g
s would
only be sent once for each task ?
Best,
Yun
--
From:Till Rohrmann
Send Time:2021 Jul. 16 (Fri.) 17:26
To:Yun Gao
Cc:dev
Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Ok, so the plan is that finish() will flush all pending events and then send
t
ot making the whole thing clear in the FLIP again, if
>> there are no other concerns I'll update the FLIP with the above conclusions
>> to make it precise in this part.
>>
>>
>> Best,
>> Yun
>>
>> ---
no other concerns I'll update the FLIP with the above conclusions
> to make it precise in this part.
>
>
> Best,
> Yun
>
> ----------
> From:Till Rohrmann
> Send Time:2021 Jul. 16 (Fri.) 16:00
> To:dev
> Cc:
16:00
To:dev
Cc:Yun Gao
Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
I think we should try to sort this out because it might affect how and when
finish() will be called (or in general how the operator lifecycle looks like).
To give an example let's ta
Best,
Yun
[1] https://issues.apache.org/jira/browse/FLINK-21132
--
From:Piotr Nowojski
Send Time:2021 Jul. 16 (Fri.) 13:48
To:dev
Cc:Yun Gao
Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Hi Til
> > >> easily encode this information
> > >> into the shortcuts on the job vertex level. And since the time seems
> to
> > >> be short, perhaps it is enough
> > >> to do re-computation from the scratch in consideration of the tradeoff
> > >> between the performance and
> > >> the
e tradeoff
> >> between the performance and
> >> the complexity ?
> >>
> >> 3) We are going to emit the EndOfInput event exactly after the finish()
> >> method and before the last
> >> snapshotState() method so that we could shut down the whole topology
> with
> >> a single final checkp
> Very sorry for not include enough details for this part and I'll
>> complement the FLIP with the details on
>> the process of the final checkpoint / savepoint.
>>
>> Best,
>> Yun
>>
>>
>>
>> --
ough details for this part and I'll
> complement the FLIP with the details on
> the process of the final checkpoint / savepoint.
>
> Best,
> Yun
>
>
>
> ------------------
> From:Till Rohrmann
> Send Time:2021 Jul. 14 (Wed
Yun
--
From:Till Rohrmann
Send Time:2021 Jul. 14 (Wed.) 22:05
To:dev
Subject:Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished
Hi everyone,
I am a bit late to the voting party but let me ask three questions:
1) Why do we execute the trig
Hi everyone,
I am a bit late to the voting party but let me ask three questions:
1) Why do we execute the trigger plan computation in the main thread if we
cannot guarantee that all tasks are still running when triggering the
checkpoint? Couldn't we do the computation in a different thread in
Hi there,
Since the voting time of FLIP-147[1] has passed, I'm closing the vote now.
There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes:
- Dawid Wysakowicz (binding)
- Piotr Nowojski(binding)
- Jiangang Liu (binding)
- Arvid Heise (binding)
- Jing Zhang (binding)
- Leonard Xu
33 matches
Mail list logo
