jingi723 opened a new pull request, #74129: URL: https://github.com/apache/airflow/pull/74129
For workloads whose completion is managed independently of Airflow, stopping the waiting Airflow task should not necessarily stop the ECS task. Currently, Dag authors must override `EcsRunTaskOperator.on_kill()` to preserve that work, even with `stop_task_on_failure=False`: that existing option controls post-start waiter failures, not the kill callback. Add `stop_task_on_kill`, defaulting to `True`. Setting it to `False` stops local log fetching and skips the ECS `StopTask` request without requiring an ECS client. The default retains the existing kill behavior, including its client/ARN checks. `stop_task_on_failure` remains independent. The option applies when the worker calls `on_kill()`, including an execution timeout while waiting synchronously. It does not change cancellation after deferral or guarantee callback execution after an abrupt worker failure. The guide explains responsibility for monitoring and cleaning up preserved tasks, and that `reattach=True` does not recover a completed task's result or guarantee deduplication. Validation on upstream `7a8ce83f3851ec65dd2186c75f90e85c666a7cc0`, Python 3.10.21, Linux amd64 Breeze: - ECS operator, hook and trigger tests: 115 passed on current sources; the same selection also passed on Airflow 2.11.0, the provider's declared minimum. The latter used source providers in an isolated container with incompatible provider metadata removed and development dependencies installed without constraints after a Ruff pin conflict, rather than the CI wheel-install matrix. - Regressions are in the existing `test_ecs.py`, using botocore Stubber for expected and forbidden requests. Keeping the new constructor option but restoring upstream `on_kill()` makes three cases fail; the final implementation passes them. - A separate local runtime probe passed four timeout/SIGTERM cases through the real Task SDK, operator, boto3 HTTP requests and log-fetcher thread, with ECS/CloudWatch responses served by a loopback substitute. It confirmed the stop/keep distinction and thread shutdown. The SIGTERM probe checks callback behavior, not the supervisor's final process termination or the complete cancellation outcome. This exploratory fixture is not added to the Task SDK suite. - Regular and manual prek checks, provider mypy, Amazon provider documentation build, selective-check and `git diff --check` passed. Manual checks omitted the three server/environment-provisioning hooks listed in the contributor instructions. Before submission, rebased onto `9c14681a4a02deefc256dcb686f693ecb591ea42`. The three reviewed files are unchanged; the related ECS suite passed again (115 tests), and selective-check passed on the final commit. No live AWS resources were used. The full dependent-provider, supported-version and platform CI matrices were not run locally. This is separate from the exit-code retry policy in #73885. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Codex (GPT-6) Generated-by: Codex (GPT-6) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
