[jira] [Commented] (FLINK-23408) Wait for a checkpoint completed after finishing a task

2023-01-04 Thread Yun Tang (Jira) 


[ 
https://issues.apache.org/jira/browse/FLINK-23408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17654463#comment-17654463
 ] 

Yun Tang commented on FLINK-23408:
--

Some of the commits are included in https://github.com/apache/flink/pull/16761

> Wait for a checkpoint completed after finishing a task
> --
>
> Key: FLINK-23408
> URL: https://issues.apache.org/jira/browse/FLINK-23408
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Checkpointing, Runtime / Task
>Reporter: Dawid Wysakowicz
>Assignee: Dawid Wysakowicz
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.0
>
> Attachments: image-2022-12-09-16-53-08-286.png, 
> image-2022-12-09-16-54-05-453.png
>
>
> Before finishing a task we should wait for a checkpoint issued after 
> {{finish()}} to commit all pending transactions created from the {{finish()}} 
> method.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (FLINK-23408) Wait for a checkpoint completed after finishing a task

2022-12-09 Thread qiunan (Jira) 


[ 
https://issues.apache.org/jira/browse/FLINK-23408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17645202#comment-17645202
 ] 

qiunan commented on FLINK-23408:


[~dwysakowicz]  Ok, thanks, I'm looking at the code for release-1.14.

> Wait for a checkpoint completed after finishing a task
> --
>
> Key: FLINK-23408
> URL: https://issues.apache.org/jira/browse/FLINK-23408
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Checkpointing, Runtime / Task
>Reporter: Dawid Wysakowicz
>Assignee: Dawid Wysakowicz
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.0
>
> Attachments: image-2022-12-09-16-53-08-286.png, 
> image-2022-12-09-16-54-05-453.png
>
>
> Before finishing a task we should wait for a checkpoint issued after 
> {{finish()}} to commit all pending transactions created from the {{finish()}} 
> method.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (FLINK-23408) Wait for a checkpoint completed after finishing a task

2022-12-09 Thread Dawid Wysakowicz (Jira) 


[ 
https://issues.apache.org/jira/browse/FLINK-23408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17645192#comment-17645192
 ] 

Dawid Wysakowicz commented on FLINK-23408:
--

I don't know where is this code coming from. As I said, in master both drain 
and w/o drain stops the source first: 
https://github.com/apache/flink/blob/d86ae5d642fa578fb85118e81dd4140d504f818a/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/SourceStreamTask.java#L260
 or 
https://github.com/apache/flink/blob/d86ae5d642fa578fb85118e81dd4140d504f818a/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/SourceOperatorStreamTask.java#L179

> Wait for a checkpoint completed after finishing a task
> --
>
> Key: FLINK-23408
> URL: https://issues.apache.org/jira/browse/FLINK-23408
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Checkpointing, Runtime / Task
>Reporter: Dawid Wysakowicz
>Assignee: Dawid Wysakowicz
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.0
>
> Attachments: image-2022-12-09-16-53-08-286.png, 
> image-2022-12-09-16-54-05-453.png
>
>
> Before finishing a task we should wait for a checkpoint issued after 
> {{finish()}} to commit all pending transactions created from the {{finish()}} 
> method.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (FLINK-23408) Wait for a checkpoint completed after finishing a task

2022-12-09 Thread qiunan (Jira) 


[ 
https://issues.apache.org/jira/browse/FLINK-23408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17645187#comment-17645187
 ] 

qiunan commented on FLINK-23408:


!image-2022-12-09-16-54-05-453.png!

[~dwysakowicz] I see that only drain will stop the source operator. 

> Wait for a checkpoint completed after finishing a task
> --
>
> Key: FLINK-23408
> URL: https://issues.apache.org/jira/browse/FLINK-23408
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Checkpointing, Runtime / Task
>Reporter: Dawid Wysakowicz
>Assignee: Dawid Wysakowicz
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.0
>
> Attachments: image-2022-12-09-16-53-08-286.png, 
> image-2022-12-09-16-54-05-453.png
>
>
> Before finishing a task we should wait for a checkpoint issued after 
> {{finish()}} to commit all pending transactions created from the {{finish()}} 
> method.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (FLINK-23408) Wait for a checkpoint completed after finishing a task

2022-12-09 Thread Dawid Wysakowicz (Jira) 


[ 
https://issues.apache.org/jira/browse/FLINK-23408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17645177#comment-17645177
 ] 

Dawid Wysakowicz commented on FLINK-23408:
--

In current master both in case of drain and w/o drain we stop the source 
operator first. In 1.14 we did not make it in time to change the behaviour for 
w/o drain as well. W/o drain in 1.14 works as before.

> Wait for a checkpoint completed after finishing a task
> --
>
> Key: FLINK-23408
> URL: https://issues.apache.org/jira/browse/FLINK-23408
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Checkpointing, Runtime / Task
>Reporter: Dawid Wysakowicz
>Assignee: Dawid Wysakowicz
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.0
>
>
> Before finishing a task we should wait for a checkpoint issued after 
> {{finish()}} to commit all pending transactions created from the {{finish()}} 
> method.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (FLINK-23408) Wait for a checkpoint completed after finishing a task

2022-12-09 Thread qiunan (Jira) 


[ 
https://issues.apache.org/jira/browse/FLINK-23408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17645175#comment-17645175
 ] 

qiunan commented on FLINK-23408:


[~dwysakowicz] hello，Currently, the stop-with-savepoint [-drain] parameter will 
block the source task if it is true, but why not block the source task if it is 
false?

If I want to stop the job and then recover from savepoint, the data will be 
consumed repeatedly if the source task is not blocked.

> Wait for a checkpoint completed after finishing a task
> --
>
> Key: FLINK-23408
> URL: https://issues.apache.org/jira/browse/FLINK-23408
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Checkpointing, Runtime / Task
>Reporter: Dawid Wysakowicz
>Assignee: Dawid Wysakowicz
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.0
>
>
> Before finishing a task we should wait for a checkpoint issued after 
> {{finish()}} to commit all pending transactions created from the {{finish()}} 
> method.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (FLINK-23408) Wait for a checkpoint completed after finishing a task

2022-12-09 Thread Dawid Wysakowicz (Jira) 


[ 
https://issues.apache.org/jira/browse/FLINK-23408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17645165#comment-17645165
 ] 

Dawid Wysakowicz commented on FLINK-23408:
--

[~qiunan] Can you elaborate what do you mean?

> Wait for a checkpoint completed after finishing a task
> --
>
> Key: FLINK-23408
> URL: https://issues.apache.org/jira/browse/FLINK-23408
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Checkpointing, Runtime / Task
>Reporter: Dawid Wysakowicz
>Assignee: Dawid Wysakowicz
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.0
>
>
> Before finishing a task we should wait for a checkpoint issued after 
> {{finish()}} to commit all pending transactions created from the {{finish()}} 
> method.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (FLINK-23408) Wait for a checkpoint completed after finishing a task

2022-12-08 Thread qiunan (Jira) 


[ 
https://issues.apache.org/jira/browse/FLINK-23408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17645097#comment-17645097
 ] 

qiunan commented on FLINK-23408:


Why not suspend source at  suspend  savepoint ？

> Wait for a checkpoint completed after finishing a task
> --
>
> Key: FLINK-23408
> URL: https://issues.apache.org/jira/browse/FLINK-23408
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Checkpointing, Runtime / Task
>Reporter: Dawid Wysakowicz
>Assignee: Dawid Wysakowicz
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.0
>
>
> Before finishing a task we should wait for a checkpoint issued after 
> {{finish()}} to commit all pending transactions created from the {{finish()}} 
> method.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)