[
https://issues.apache.org/jira/browse/FLINK-40518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Fan updated FLINK-40518:
----------------------------
Description:
isCheckpointingDuringRecoveryEnabled checks only
UNALIGNED_RECOVER_OUTPUT_ON_DOWNSTREAM and
CHECKPOINTING_DURING_RECOVERY_ENABLED, not unaligned.enabled [1]. So a job
restoring from an unaligned checkpoint but running with UC disabled still takes
the CDR recovery branch [2] and goes RUNNING before the drain finishes.
But InputProcessorUtil threads recoveryCheckpointTrigger only into
alternating() (UC on); the aligned() handler used when UC is off gets NO_OP
[3]. The aligned path never considered a barrier delivered mid-recovery: the
barrier is stashed, so it can neither snapshot-during-recovery nor decline
TASK_NOT_READY. A checkpoint triggered in the recovery window hangs and expires
(tolerable-failed-checkpoints default 0), leading to a failover loop.
Fix: the aligned path can't do CDR anyway, so don't enable CDR there — add "&&
isUnalignedCheckpointEnabled(config)" inside
isCheckpointingDuringRecoveryEnabled. It is the single central gate for all CDR
call sites, so UC-off falls every path back to the tested non-CDR recovery. No
functional loss.
[1]
[https://github.com/apache/flink/blob/76e774e04ff84ea305714153c5f4790fff98437c/flink-core/src/main/java/org/apache/flink/configuration/CheckpointingOptions.java#L844-L849]
[2]
[https://github.com/apache/flink/blob/76e774e04ff84ea305714153c5f4790fff98437c/flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L943-L945]
[3]
[https://github.com/apache/flink/blob/76e774e04ff84ea305714153c5f4790fff98437c/flink-runtime/src/main/java/org/apache/flink/streaming/runtime/io/checkpointing/InputProcessorUtil.java#L186-L199]
was:
`isCheckpointingDuringRecoveryEnabled` checks only
`UNALIGNED_RECOVER_OUTPUT_ON_DOWNSTREAM` and
`CHECKPOINTING_DURING_RECOVERY_ENABLED`, **not** `unaligned.enabled`
([CheckpointingOptions.java#L844-L849](https://github.com/apache/flink/blob/76e774e04ff84ea305714153c5f4790fff98437c/flink-core/src/main/java/org/apache/flink/configuration/CheckpointingOptions.java#L844-L849)).
So a job restoring from an unaligned checkpoint but running with UC disabled
still takes the CDR recovery branch
([StreamTask.java#L943-L945](https://github.com/apache/flink/blob/76e774e04ff84ea305714153c5f4790fff98437c/flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L943-L945))
and goes RUNNING before the drain finishes.
But `InputProcessorUtil` threads `recoveryCheckpointTrigger` only into
`alternating()` (UC on); the `aligned()` handler used when UC is off gets
`NO_OP`
([InputProcessorUtil.java#L186-L199](https://github.com/apache/flink/blob/76e774e04ff84ea305714153c5f4790fff98437c/flink-runtime/src/main/java/org/apache/flink/streaming/runtime/io/checkpointing/InputProcessorUtil.java#L186-L199)).
The aligned path never considered a barrier delivered mid-recovery: the
barrier is stashed, so it can neither snapshot-during-recovery nor decline
`TASK_NOT_READY`. A checkpoint triggered in the recovery window hangs and
expires (`tolerable-failed-checkpoints` default 0) → failover loop.
**Fix**: the aligned path can't do CDR anyway, so don't enable CDR there — add
`&& isUnalignedCheckpointEnabled(config)` inside
`isCheckpointingDuringRecoveryEnabled`. It is the single central gate for all
CDR call sites, so UC-off falls every path back to the tested non-CDR recovery.
No functional loss.
> CDR is wrongly kept enabled on the aligned path when unaligned checkpoint is
> disabled
> -------------------------------------------------------------------------------------
>
> Key: FLINK-40518
> URL: https://issues.apache.org/jira/browse/FLINK-40518
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Checkpointing
> Reporter: Rui Fan
> Assignee: Rui Fan
> Priority: Major
>
> isCheckpointingDuringRecoveryEnabled checks only
> UNALIGNED_RECOVER_OUTPUT_ON_DOWNSTREAM and
> CHECKPOINTING_DURING_RECOVERY_ENABLED, not unaligned.enabled [1]. So a job
> restoring from an unaligned checkpoint but running with UC disabled still
> takes the CDR recovery branch [2] and goes RUNNING before the drain finishes.
> But InputProcessorUtil threads recoveryCheckpointTrigger only into
> alternating() (UC on); the aligned() handler used when UC is off gets NO_OP
> [3]. The aligned path never considered a barrier delivered mid-recovery: the
> barrier is stashed, so it can neither snapshot-during-recovery nor decline
> TASK_NOT_READY. A checkpoint triggered in the recovery window hangs and
> expires (tolerable-failed-checkpoints default 0), leading to a failover loop.
> Fix: the aligned path can't do CDR anyway, so don't enable CDR there — add
> "&& isUnalignedCheckpointEnabled(config)" inside
> isCheckpointingDuringRecoveryEnabled. It is the single central gate for all
> CDR call sites, so UC-off falls every path back to the tested non-CDR
> recovery. No functional loss.
> [1]
> [https://github.com/apache/flink/blob/76e774e04ff84ea305714153c5f4790fff98437c/flink-core/src/main/java/org/apache/flink/configuration/CheckpointingOptions.java#L844-L849]
>
> [2]
> [https://github.com/apache/flink/blob/76e774e04ff84ea305714153c5f4790fff98437c/flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L943-L945]
>
> [3]
> [https://github.com/apache/flink/blob/76e774e04ff84ea305714153c5f4790fff98437c/flink-runtime/src/main/java/org/apache/flink/streaming/runtime/io/checkpointing/InputProcessorUtil.java#L186-L199]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
