BALOGUN-DAVID opened a new pull request, #65970: URL: https://github.com/apache/airflow/pull/65970
Fixes a scheduler race where queued-timeout recovery could act on a stale task instance after it had already transitioned out of `QUEUED` and into `DEFERRED`. This change makes the scheduler re-check that the task instance is still the same queued row before revoking, rescheduling, or failing it. It also adds regression coverage for the stale queued-to-deferred race so deferred tasks are not incorrectly rescheduled or failed by queued-timeout recovery. related: #65248 --- ## Description **Context:** A task instance selected as "stuck in queued" could legitimately move to `RUNNING` and then `DEFERRED` before the scheduler's recovery logic acted on it. The queued-timeout path could then overwrite that newer state and either reschedule or fail the deferred task based on stale queued state. **Approach:** - Guard queued-timeout recovery with a fresh check that the task instance is still `QUEUED` - Revoke tasks only after confirming they are still genuinely stuck in queued - Reuse the validated current task instance for both reschedule and fail paths - Add regression tests covering stale queued selections that become deferred before recovery runs **Impact:** - **Functionality:** prevents queued-timeout recovery from clobbering deferred task state - **Developer Experience:** makes the scheduler recovery path safer and easier to reason about - **Coverage/Robustness:** adds regression coverage for the queued-to-deferred race reported in the issue --- ## Tests - [x] New tests added - [x] Existing tests updated **Unit Tests Summary:** - `test_handle_stuck_queued_tasks_skips_ti_that_already_deferred` - `test_handle_stuck_queued_tasks_skips_failing_ti_that_already_deferred` **Verification notes:** - Ruff formatting and checks were run for the modified files - Targeted pytest/Breeze verification could not be completed in the current local environment because WSL was failing with system-level `git`/`ssh` I/O errors --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Codex (GPT-5) Generated-by: Codex (GPT-5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
