seanghaeli opened a new pull request, #69499:
URL: https://github.com/apache/airflow/pull/69499
## Why
`RedshiftDeleteClusterOperator` retries on `InvalidClusterStateFault` when
the cluster is mid-transition (e.g. pausing/resuming/resizing), but the retry
budget was **hardcoded** to `10 attempts * 15s = 2.5 minutes`. A cluster
transition routinely takes much longer than that (a pause is ~2–15 min, a
classic resize can be many minutes), so the loop exhausts and re-raises long
before the cluster reaches a deletable state. The task fails and the cluster is
**leaked** — it keeps running and billing until someone deletes it manually.
The real AWS error the operator hits mid-transition:
```
InvalidClusterState: There is an operation running on the Cluster.
Please try to delete it at a later time.
```
(and, once resizing: `Unable to delete the cluster ... You can only delete
clusters with PAUSED, ..., ACTIVE, ... lifecycles.`)
## What
Promote the two hardcoded constants to `__init__` parameters and raise the
default so the retry outlasts a real transition:
- `busy_retry_attempts: int = 60`
- `busy_retry_interval: int = 15`
Default is now `60 * 15s = 15 minutes`, and both are overridable.
`poll_interval` / `max_attempts` (the completion waiter) are unchanged. The
existing fail-loud behavior is preserved: once the budget is exhausted the
operator still re-raises rather than swallowing the error.
## Behavior change
- Default busy-retry window: **2.5 min → 15 min**. A delete against a
transitioning cluster now rides through the transition instead of failing.
- The private attributes `self._attempts` / `self._attempt_interval` are
replaced by the new public params. They were underscore-private, so external
dependence is unlikely.
## Scope / follow-up
This retry loop is **synchronous** and runs before the `deferrable` branch,
so in deferrable mode it still blocks the worker for the retry window. This PR
fixes the synchronous path (the reported leak). The correct async fix for
deferrable mode — defer immediately on `InvalidClusterStateFault`, poll until
the cluster settles via a trigger, then re-issue the delete (mirroring the
re-defer pattern in `EksCreateClusterOperator.deferrable_create_cluster_next`)
— is a larger change that adds a new serialized trigger, so it will be a
**separate follow-up PR** to keep this one small and focused.
## Testing
Unit tests (added/updated): default values, custom values, retry-then-raise
after a custom budget, and succeed-on-second-attempt. `16 passed` locally.
### Real-AWS end-to-end proof (before/after)
Verified against a real Redshift cluster in a dev account, using an
identical DAG and an identical ~6.5-minute transient (a classic resize) as a
controlled before/after. Same cluster, same transient window.
**BEFORE (stock `main`, 2.5-min budget) — task FAILS, cluster leaks:**
```
20:38:51 Cluster in resizing state, unable to delete. 9 attempts remaining.
20:39:06 ... 8 attempts remaining.
... (15s sleeps) ...
20:40:54 ... 1 attempts remaining.
botocore.errorfactory.InvalidClusterStateFault: An error occurred
(InvalidClusterState)
when calling the DeleteCluster operation: Unable to delete the cluster ...
DagRun ... state=failed TASK_EXIT=1
```
**AFTER (this PR, 15-min budget) — task SUCCEEDS, cluster deleted:**
```
20:42:27 Cluster in resizing state, unable to delete. 59 attempts remaining.
20:42:43 ... 58 attempts remaining.
20:43:29 ... 55 attempts remaining. <-- already past where stock main
died
...
20:44:46 ... 50 attempts remaining.
20:45:01 Task instance state updated ... new_state=success <-- delete
accepted once resize settled
DagRun ... state=success TASK_EXIT=0
```
AWS-side status timeline confirms the sequence: `resizing` throughout both
runs → BEFORE fails mid-resize at 20:41 → AFTER's delete is accepted at 20:45
(`deleting`) → `ClusterNotFound` at 20:48 (cluster fully torn down, no leak).
> Note: UI screenshots were not captured (headless environment); the task
logs above are the primary artifact. A classic resize was used as the transient
because pause/resume settled too quickly (~100–170s) to reliably exceed stock
`main`'s 150s budget — the transition *type* is immaterial, any
`InvalidClusterStateFault`-producing operation exercises the same retry loop.
---
Generated-by: Claude Code (Opus)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
