rjgoyln commented on issue #62050:
URL: https://github.com/apache/airflow/issues/62050#issuecomment-5464116672
Hi @kaxil, @jscheffl, and @potiuk — regarding the recent closure of #72243
and the concerns raised in #62878, I did a deep dive and reproduced this
locally on Postgres 14 against `main` without mocking. The results actually
prove your instincts right, but change the shape of the fix. I'd like to align
on the direction before opening a PR.
### 1. The current `UPDATE` is destructive, not a safety net
When a serialized DAG is missing, the current bulk `UPDATE` causes:
- **Cross-run blast radius:** It wrongly fails SCHEDULED tasks across **all
runs** of the DAG, not just the affected one.
- **Silent UI failures:** Tasks go straight to FAILED. It bypasses
`handle_failure` entirely—no retries, no callbacks, no logs.
- **$O(N)$ DB spam:** Every TI of the missing DAG triggers its own
full-table `UPDATE` in the batch.
- **Stuck runs:** The tasks are destroyed, but the DagRun remains stuck in
`RUNNING` anyway.
### 2. The Starvation Trap: Why "pure skip" (#62878 / #72243) is worse
The current `UPDATE` accidentally acts as a starvation filter. If we just
remove it ("pure skip"), the broken tasks loop forever, locking out healthy
DAGs.
Here is a 2-pass scheduler test with a broken DAG (`max_tis=2`) and a
healthy DAG:
| Scenario | PASS 1 queued | PASS 2 queued | Broken DAG's tasks | Healthy
DAG's task |
|---|---|---|---|---|
| **Current `main`** | `[]` | `[healthy_task]` | FAILED (Wrongly killed) |
Queued after 1 loop delay |
| **Pure Skip** | `[]` | `[]` | SCHEDULED | **Never queued (Cluster
Starvation!)** |
| **Skip + `starved_dags`** | `[healthy_task]` | `[]` | SCHEDULED | **Queued
immediately** |
### 3. The Proposed Fix
When `get_dag_for_run()` returns `None` in
`_task_concurrency_allows_execution`:
1. **Leave TIs as `SCHEDULED`** (stop the destructive UPDATE).
2. **Add `dag_id` to `starved_dags`** to prevent the cluster starvation
shown above.
3. **Track the missing DAG locally** to log the error only once per DAG per
batch, not once per TI.
### 4. Scope & Next Steps
The remaining concern (from #62878) is that a permanently deleted DAG leaves
TIs SCHEDULED forever. Since this is a DagRun-level gap already being addressed
by **#70056**, I plan to scope my PR strictly to the concurrency and starvation
issues above.
Does this plan look good to you? I have the local reproduction ready as a
test and can open the PR if you agree with this direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
