rjgoyln opened a new pull request, #72814: URL: https://github.com/apache/airflow/pull/72814
## Summary When a Dag's serialized row cannot be read during the task-concurrency check, the scheduler fails every `SCHEDULED` task instance of that Dag in one unbounded `UPDATE`, across all of its runs. The miss is usually transient — the Dag processor mid-write, a version briefly unreadable — and the failure bypasses `handle_failure`, so the task instances lose their retries, callbacks and logs while the Dag run stays `RUNNING`, leaving hand-clearing as the only recovery. That update was also load-bearing: flipping the rows to `FAILED` is what kept them out of the next iteration of the critical-section query. Removing it alone would let the same task instances refill every batch, so no other Dag is ever examined. ## Change - Leave the task instances `SCHEDULED` and skip them for the round. - Starve the affected Dag run so the query reaches the Dags behind it. - Log the miss once per Dag per round rather than once per task instance. The starvation key is `(dag_id, run_id)` rather than `dag_id` because resolution is per run: a run pinned to a deleted version misses while another run of the same Dag still resolves. ## Behavior change A Dag whose serialized rows are gone for good now leaves its task instances `SCHEDULED` rather than `FAILED`; bounding that case is tracked in #70056. ## Tests - Four broken runs with `max_tis=2` still let a healthy Dag queue, with the serialized rows deleted rather than mocked. - A miss on one run does not hold back the other runs of the same Dag. ## Relation to #72652 #72652 fixes the same issue with a per-Dag starvation key. This change differs in the key, in deduplicating the error per Dag, and in covering the starvation behaviour with tests. closes: #62050 ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5) Generated-by: Claude Code (Opus 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
