Leondon9 opened a new pull request, #73701: URL: https://github.com/apache/airflow/pull/73701
This adds a `durable` option to `EmrServerlessStartJobOperator` that persists the submitted job run id to task state (Airflow 3.3+) and, on a retry or a manual clear/rerun, reconnects to the still-running (or already-succeeded) job run instead of submitting a duplicate. ## Motivation On a clear/retry today, a deferrable EMR Serverless task re-runs `execute()` and starts a brand-new job run, so the original hourly-billed Spark job is either duplicated or (with `cancel_on_kill`) killed and re-billed. `GlueJobOperator` already avoids this via its `durable` option; EMR Serverless had no equivalent, so the two Spark-submitting operators behaved differently on the same user action. This brings them into line. `durable` is orthogonal to the existing `cancel_on_kill`: - `durable=True` + `cancel_on_kill=False` → reattach to the running job on clear (no duplicate, no re-bill). - `cancel_on_kill=True` (default) → cancel the old run and submit a fresh one. It defaults to `False`, preserving current behaviour, and has no effect on Airflow < 3.3 (task state unavailable), where it degrades to a fresh submit. ## Draft — open questions for reviewers 1. **Default.** Kept `durable=False` to avoid surprising existing users; `GlueJobOperator` defaults `durable=True` on 3.3+. Should EMR Serverless match Glue's default instead? 2. **Mechanism.** This reads/writes `task_state_store` directly on the deferrable path — the clean approach envisaged in #71485 — rather than adopting `ResumableJobMixin`, whose `execute_resumable` only serves the synchronous path. Should this instead adopt `ResumableJobMixin` for structural consistency with `GlueJobOperator` / `RedshiftDataOperator`? (related: #71485) The docs how-to update and an EMR-on-EKS (`EmrContainerOperator`) port are intentionally deferred pending validation. A companion PR will add trigger-level `cancel_on_kill` to `GlueJobOperator` so the two Spark operators expose the same toggles in both directions. ## Testing - New unit tests cover reconnect-to-active, reconnect-to-succeeded, resubmit-on-terminal, fresh-submit persistence, no-store fallback, `durable=False` no-op, and deferrable reconnect. - Full `test_emr_serverless.py` suite green (70 passed) — the `durable=False` default keeps existing behaviour unchanged. - Not yet validated against live AWS. Provider `mypy` and other Docker-dependent prek hooks were not run locally (Docker unavailable); CI will validate them. related: #71485 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 4.8) Generated-by: Claude Code (Opus 4.8) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) 🤖 AI-assisted changes - reviewed by human developer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
