hussein-awala opened a new pull request, #76210: URL: https://github.com/apache/airflow/pull/76210
Clearing a deferred task, or marking it failed or succeeded, cancels its trigger and runs `on_kill()` for up to `[triggerer] on_kill_timeout` (30 s by default). The supervisor re-sends the id in every sync until the trigger finishes ([L656-L661](https://github.com/apache/airflow/blob/b95498cd56/airflow-core/src/airflow/jobs/triggerer_job_runner.py#L656-L661)), and the runner cancels the task again on each, about once per second. The second `CancelledError` lands inside `on_kill()`, where a branch written for shutdown swallows it ([L1845-L1855](https://github.com/apache/airflow/blob/b95498cd56/airflow-core/src/airflow/jobs/triggerer_job_runner.py#L1845-L1855)), so cleanup such as deleting a pod stops after about a second, possibly leaving the resource running. Record when each trigger was cancelled, and cancel it again only after `on_kill_timeout` plus a 5 s margin. related: #76209 No new configuration, database or protocol change. Only delivered cancels are recorded, so one arriving before the trigger task exists applies on the next re-send. The id is forgotten when the trigger finishes, so a reused id can be cancelled. The supervisor never moves an id between `cancelling_triggers` and `releasing_triggers`, so one record serves both. A cancelled trigger now counts in the `triggers.running` and `triggerer.capacity_left` gauges while `on_kill()` runs; claiming new triggers counts database rows and is unaffected. A task that swallows its first cancellation is cancelled again after 35 s by default instead of about a second. The 5 s margin covers logging and `cleanup()` around `on_kill()`. Shutdown is unaffected. #73702 (Glue) and #74358 (BigQuery) rely on `on_kill()` finishing. `v3-4-test` has the same `cancel_triggers()`; `v3-3-test` has only `to_cancel` and needs a small manual change. A later PR planned in #76209 wakes the runner on trigger activity (up to about 20 syncs/s); without this fix, event traffic there cuts `on_kill()` off after about 50 ms, so this should merge first. Another PR in #76209 touches the same `cancel_triggers()`/`cleanup_finished_triggers()` code, replacing the cleanup scan with a done-callback queue; whichever merges later is rebased and keeps the forget-on-finish step. In a runner-level benchmark driving the real `TriggerRunner` methods, with a trigger whose `on_kill()` needs 3.0 s and a cancel re-sent every 0.25 s: | Metric | Before | After | |---|---:|---:| | `on_kill()` completed | no | yes | | `on_kill()` start to task end | 0.50 s | 3.00 s | | Syncs that carried the cancel | 2 | 12 | | Create 500 triggers (control) | 156.6 ms | 157.4 ms | Before, the second cancel aborted `on_kill()` about 0.25 s in and the next poll saw the task end. After, the cancel is re-sent longer only because `on_kill()` now completes; only the first is delivered. Medians of five rounds, alternating which side runs first; these paths issue no SQL. Cancel rows matched every round; the control moved +0.5%, within noise. Local measurements, not end-to-end latency. `test_repeated_cancel_syncs_do_not_abort_on_kill` re-sends the cancel while a real `run_trigger()` task is held in `on_kill()` by an event the test releases. Others cover redelivery after the grace period for both queues, a cancel arriving before its trigger task exists, and a reused id. Every new test fails against the previous logic. Validation: - Focused triggerer and trigger-model tests: 210 passed, 1 skipped on SQLite. - PostgreSQL and MySQL were not run locally; this change has no database code. - Regular and manual static checks passed on the full diff. - Selected core suites and the documentation build were not run locally; CI validation is pending. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5.5) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
