[
https://issues.apache.org/jira/browse/FLINK-40388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18104789#comment-18104789
]
Han You commented on FLINK-40388:
---------------------------------
This has also been a pain point for us, but I think it's tricky to know which
case is true:
# the job experienced a transient error, and restarting will allow it to
recover and properly take a savepoint
# the job genuinely is in a bad state and restarting it is futile, we should
just remove the finalizer
If I recall correctly, once the job has exhausted its restart limit and entered
a fail state, the operator will skip taking a savepoint (assuming
kubernetes.operator.job.restart.failed is false which is the default)
> FlinkDeployment deletion can remain stuck when savepoint-on-deletion is
> enabled and the job is not running
> ----------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40388
> URL: https://issues.apache.org/jira/browse/FLINK-40388
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Affects Versions: kubernetes-operator-1.15.0
> Environment: h4. Environment
> * Flink Kubernetes Operator: 1.15.0
> * Apache Flink: 1.20.3
> * Kubernetes: EKS
> h4. Configuration
> {code:yaml}
> spec:
> flinkConfiguration:
> kubernetes.operator.job.savepoint-on-deletion: "true"
> kubernetes.operator.job.drain-on-savepoint-deletion: "true"
> {code}
> Reporter: Avi Sanwal
> Priority: Major
>
> h4. Problem
> Deleting a {{FlinkDeployment}} can remain in {{Terminating}} when
> savepoint-on-deletion is enabled but the Flink job is in a non-running state,
> such as {{RESTARTING}} during recovery or an upgrade.
> h4. Steps to reproduce
> # Deploy a stateful {{FlinkDeployment}} with high availability enabled.
> # Enable {{savepoint-on-deletion}} and {{drain-on-savepoint-deletion}}.
> # Put the job into a non-running state during recovery or an upgrade, for
> example {{RESTARTING}}.
> # Delete the {{FlinkDeployment}} before the job reaches {{RUNNING}}.
> h4. Actual behavior
> The operator attempts deletion-time savepoint handling even though the job
> cannot produce a savepoint. Cleanup is retried while the finalizer remains on
> the {{FlinkDeployment}}, leaving the resource stuck in {{Terminating}} and
> preventing a clean redeployment.
> h4. Expected behavior
> If deletion-time savepoint handling is selected but the observed job is not
> running, the Operator must not retry the same impossible savepoint operation
> indefinitely.
> It should either:
> * complete an explicit, configurable fallback cleanup while preserving
> recoverable HA metadata where available; or
> * retain the resource intentionally, but set an actionable status condition
> and emit a Kubernetes warning event explaining that a savepoint cannot be
> created and what intervention is required.
> h4. Redacted Evidence
> The following sanitized Operator logs show repeated deletion cleanup attempts
> for a non-running job:
> {code}
> [time] INFO CLEANUP | Cleaning up FlinkDeployment
> [time] INFO Observing JobManager deployment. Previous status: MISSING
> [time] INFO JobManager is being deployed
> [time] WARN Uncaught error during event processing; another reconciliation
> will be attempted
> cause=java.lang.RuntimeException:
> Unexpected job status: state=RESTARTING
> [time] INFO CLEANUP | Cleaning up FlinkDeployment
> [time] WARN Uncaught error during event processing
> cause=java.lang.RuntimeException:
> Unexpected job status: state=RESTARTING
> {code}
> The {{FlinkDeployment}} retained
> {{flinkdeployments.flink.apache.org/finalizer}} while {{status.error}}
> remained empty. No Kubernetes warning event explained that deletion was
> blocked by an unavailable savepoint.
> h5. Control case
> With the same deletion settings, deletion completed when the JobManager was
> reachable and the job was running:
> {code}
> [time] INFO CLEANUP | Cleaning up FlinkDeployment
> [time] INFO Job status changed from RESTARTING to RUNNING
> [time] INFO Suspending job with savepoint
> [time] INFO Job successfully suspended with savepoint <savepoint-path>
> [time] INFO DELETED | The resource is deleted
> {code}
> This distinguishes normal savepoint-on-deletion behavior from the failure
> case: the hang occurs when savepoint cleanup is selected for a job that
> cannot produce a savepoint.
> h4. Suggested resolution
> If deletion requests a savepoint but the job is not running, use a safe
> fallback such as *last-state cleanup*:
> * Delete the Flink cluster resources.
> * Retain HA metadata so state recovery remains possible.
> * Record that the deletion-time savepoint was skipped because the job was not
> running.
> * Remove the finalizer.
> The existing drain/savepoint behavior for a running job should remain
> unchanged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
