vatsrahul1001 opened a new pull request, #75841:
URL: https://github.com/apache/airflow/pull/75841
Follow-up to the deserialization guard backported in #75745 (manual backport
of #75263).
## Problem
#75745 backported the crash guard from #75263 but **dropped the
`starved_dag_runs` dimension**, because the full #75263 was entangled with
#70307 ("Fill scheduler task batches after concurrency rejection"), which is
not on `v3-4-test`.
Without `starved_dag_runs`, the guard skips a task instance whose Dag fails
to deserialize but never removes that dag run from the candidate set. When such
a run sorts to the top of a batch limited by `max_tis`:
1. `_build_schedulable_tis_query` fetches only the highest-priority rows
(the bad run's TI).
2. `_task_concurrency_allows_execution` skips it (deserialization fails →
`return False`), but nothing is starved.
3. `found_new_filters` is `False` → the selection loop **breaks**.
4. Lower-priority runnable TIs behind the bad run are never queued that tick
— and the same thing repeats every tick.
So the backport guarded against the *crash* but reintroduced the
*starvation* that #75263 exists to prevent. This surfaced as a branch-wide
failure of
`test_find_executable_task_instances_skips_dag_that_cannot_be_deserialized`
(shipped by #75745), which fails the `CI passed` gate on every open PR into
`v3-4-test`.
## Fix
Port only the self-contained `starved_dag_runs` slice from main's #75263
onto the selection machinery already present on `v3-4-test`:
- mark a dag run as starved when its Dag fails to deserialize (and
short-circuit an already-starved run);
- exclude starved runs from the re-query in `_build_schedulable_tis_query`;
- count a newly starved run in `found_new_filters` so the loop re-queries
and makes forward progress.
This makes the backport behave identically to `main` **without** pulling in
#70307. The `found_new_filters` refill loop and the query-exclusion builder it
relies on are already on `v3-4-test`.
## Testing
The previously-failing test now passes, along with the sibling
DB-error-propagation test and the full `test_find_executable_task_instances*` /
query-builder / critical-section suite (40 passed) locally on SQLite:
```
test_find_executable_task_instances_skips_dag_that_cannot_be_deserialized
PASSED
test_find_executable_task_instances_reraises_db_errors_from_dag_load PASSED
```
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 4.8)
Generated-by: Claude Code (Opus 4.8) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
