[
https://issues.apache.org/jira/browse/FLINK-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135667#comment-17135667
]
Zhijiang edited comment on FLINK-18238 at 6/15/20, 9:29 AM:
------------------------------------------------------------
> Ok I get it. It would be I think OK to cancel the alignment directly from the
> {{SubtaskCheckpointCoordinator}}, but as > I wrote above, it might open up
> possibilities for some race conditions with task not started.
> This could be handled by checking if the checkpoint has already been aborted
> or not.
I guess we can not cancel the alignment easily now when receiving the abort rpc
call, since the SubtaskCheckpointCoordinator can not access the component of
CheckpointBarrierHandler directly. Also we could not avoid duplicated broadcast
since these two components work separately.
> With the abort RPC call, cancellations can overtake the pending checkpoint
> barriers, so in the before mentioned scenario, we would cancel all
> checkpoints, from N to N+5. I'm not sure if this can happen on master as it
> is without broadcasting {{CancelCheckpointMarker}}, but I think it could
> happen with broadcasting.
I think we need to finalize the semantic of aborting checkpoint from global
coordinator firstly. If the checkpoint N+5 is aborted from coordinator, does
that mean all the pending checkpoints smaller than N+5 should also be aborted
as well?
The previous semantic is for precise abort for the indicated checkpoint id
from rpc call. If we changed this behavior to abort all the preceding pending
checkpoint, we might need to think more and redefine the semantic/behavior.
Anyway I think it might be a separate topic to discuss. Now we can firstly
confirm how to cancel the dedicated checkpoint to keep the previous behavior.
was (Author: zjwang):
> Ok I get it. It would be I think OK to cancel the alignment directly from the
>{{SubtaskCheckpointCoordinator}}, but as > I wrote above, it might open up
>possibilities for some race conditions with task not started.
> This could be handled by checking if the checkpoint has already been aborted
> or not.
I guess we can not cancel the alignment easily now when receiving the abort rpc
call, since the SubtaskCheckpointCoordinator can not access the component of
CheckpointBarrierHandler directly. Also we could not avoid duplicated broadcast
since these two components work separately.
>With the abort RPC call, cancellations can overtake the pending checkpoint
>barriers, so in the before mentioned scenario, we would cancel all
>checkpoints, from N to N+5. I'm not sure if this can happen on master as it is
>without broadcasting {{CancelCheckpointMarker}}, but I think it could happen
>with broadcasting.
I think we need to finalize the semantic of aborting checkpoint from global
coordinator firstly. If the checkpoint N+5 is aborted from coordinator, does
that mean all the pending checkpoints smaller than N+5 should also be aborted
as well?
The previous semantic is for precise abort for the indicated checkpoint id from
rpc call. If we changed this behavior to abort all the preceding pending
checkpoint, we might need to think more and redefine the semantic/behavior.
Anyway I think it might be a separate topic to discuss. Now we can firstly
confirm how to cancel the dedicated checkpoint to keep the previous behavior.
> RemoteChannelThroughputBenchmark deadlocks
> ------------------------------------------
>
> Key: FLINK-18238
> URL: https://issues.apache.org/jira/browse/FLINK-18238
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
> Affects Versions: 1.11.0
> Reporter: Piotr Nowojski
> Assignee: Yingjie Cao
> Priority: Blocker
> Fix For: 1.11.0
>
> Attachments: consoleText_remote_benchmark_deadlock.txt
>
>
> In the last couple of days
> {{RemoteChannelThroughputBenchmark.remoteRebalance}} deadlocked for the
> second time:
> http://codespeed.dak8s.net:8080/job/flink-master-benchmarks/6019/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
