[
https://issues.apache.org/jira/browse/FLINK-39704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18081960#comment-18081960
]
Prashant Bhardwaj commented on FLINK-39704:
-------------------------------------------
I also looked at FLIP-194.
My understanding is that FLIP-194 introduced JobResultStore specifically so
that after JM/Dispatcher failover, the new leader can detect that a job already
reached a globally terminal state and avoid restarting it. FLIP-194 also notes
that `delete-on-commit=false` provides the stronger failover guarantee, while
`true` mainly preserves old cleanup behavior.
So this does not appear to be a `delete-on-commit=true` misconfiguration. The
behavior we are seeing seems to violate the guarantee FLIP-194 is intended to
provide: the application job reaches globally terminal `FAILED`, but after
leadership revoke/reacquire the job is recovered again instead of remaining
terminal.
Specifically, to quote FLIP-194:
{noformat}
Ensure that we don’t restart jobs that have already been completed.
{noformat}
My current suspicion is that the terminal result is lost during the leadership
revoke/close race and gets replaced by synthetic `SUSPENDED`, after which HA
recovery treats the job as resumable. In other words, the issue seems to be in
how the terminal result is propagated/persisted during failover, rather than in
the JobResultStore configuration itself.
I am looking into a fix in this area.
> Kubernetes HA can recover a globally terminal FAILED application job after
> leadership revoke/reacquire
> ------------------------------------------------------------------------------------------------------
>
> Key: FLINK-39704
> URL: https://issues.apache.org/jira/browse/FLINK-39704
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 2.2.0, 2.4.0
> Reporter: Prashant Bhardwaj
> Priority: Major
> Attachments: flink-config-ha-reanimation-repro-cm-2026-05-19.yaml,
> jm-ha-reanimation-repro-current-2026-05-18.log,
> jm-ha-reanimation-repro-events-2026-05-18.txt,
> jm-ha-reanimation-repro-pod-describe-2026-05-18.txt,
> jm-ha-reanimation-repro-previous-2026-05-18.log
>
>
> In a Kubernetes HA application cluster, a job that has already reached the
> globally terminal FAILED state can be recovered and restarted with the same
> JobID if Kubernetes leadership is revoked/reacquired immediately after the
> terminal transition.
> Observed with apache/flink:2.2.0 and Kubernetes HA.
> *Timeline from repro:*
> {noformat}
> 20:52:51.075 Task failure after TaskManager deletion
> 20:52:51.119 Job e7ce38da0a5b4651ce64453d6ffaa25b switched RUNNING -> FAILING
> 20:52:51.122 Job e7ce38da0a5b4651ce64453d6ffaa25b switched FAILING -> FAILED
> 20:52:52.615 KubernetesLeaderElector observed empty leader holder
> 20:52:52.616 Leadership revoked
> 20:52:52.618 Dispatcher reported same job as terminal SUSPENDED
> 20:52:52.921 DefaultExecutionPlanStore released execution plan
> e7ce38da0a5b4651ce64453d6ffaa25b
> 20:52:52.926 Same job id was retrieved from KubernetesStateHandleStore
> 20:52:53.035 Same StreamGraph(jobId: e7ce38da0a5b4651ce64453d6ffaa25b) was
> recovered
> 20:53:11.340 Same job switched CREATED -> RUNNING
> {noformat}
> *Expected:*
> Once a job reaches globally terminal FAILED, later leadership
> revocation/close should not overwrite or mask the globally terminal result as
> SUSPENDED. HA metadata should be cleaned up as a globally terminal job, and
> the same job should not be recovered.
> *Actual:*
> Leadership revocation closes the running JobMaster/Dispatcher path with
> synthetic SUSPENDED after the real FAILED result. The execution plan is
> released rather than permanently removed, so the same job id remains
> recoverable from Kubernetes HA storage and is started again.
> A secondary issue is also visible in the same churn window:
> DefaultLeaderElectionService receives a grant while issuedLeaderSessionID is
> already set and throws:
> java.lang.IllegalStateException:
> The leadership should have been granted while not having the leadership
> acquired.
> This crashes the JobManager entrypoint, but the reanimation has already
> happened before the fatal error: the failed job was released/recovered from
> HA metadata.
> *Reproduction outline:*
> 1. Run a Kubernetes HA application cluster with restart-strategy.type: none.
> 2. Use a persistent HA storage dir.
> 3. Delete the TaskManager so the job reaches FAILED.
> 4. Immediately after observing RUNNING -> FAILING, patch the cluster leader
> ConfigMap annotation holderIdentity to empty, forcing leadership
> loss/reacquire.
> 5. Observe FAILED followed by SUSPENDED/release/recovery of the same JobID.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
