Han You created FLINK-40810:
-------------------------------
Summary: FlinkSessionJob deletion leaves the job running when a
transient Job Not Found error was never cleared
Key: FLINK-40810
URL: https://issues.apache.org/jira/browse/FLINK-40810
Project: Flink
Issue Type: Bug
Components: Kubernetes Operator
Reporter: Han You
When a running session job is briefly missing from the JobManager's job
overview (for example, one possible cause is
https://issues.apache.org/jira/browse/FLINK-40602), {{JobStatusObserver}} sets
{{status.error}} to {{"Job Not Found"}}. Once the job shows up again the job
state goes back to {{RUNNING}}, but the error is never cleared. Later, when the
{{FlinkSessionJob}} is deleted, {{SessionJobReconciler.cleanupInternal}} reads
that stale error as proof that the job is already gone. It removes the
finalizer without taking a savepoint or cancelling the job. The job keeps
running on the session cluster, and
- the operator no longer manages it.
- When a controller recreates the {{FlinkSessionJob}} for the same pipeline,
two jobs can end up running against the same sources and sinks. The same risk
as FLINK-39891, reached through a different path.
- The expected savepoint on deletion is never taken.
- There is no visible sign. The CR reports {{RUNNING}} / {{STABLE}}, the only
event on deletion is {{Cleanup}}, and nothing points at the stale error.
We hit this in production. A single JobManager HA recovery left every session
job on that cluster carrying the stale error.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
