joaopamaral opened a new pull request, #71769:
URL: https://github.com/apache/airflow/pull/71769
## Problem
`clear_task_instances` calls `scheduler_dagbag.get_latest_version_of_dag()`
and `DagVersion.get_latest_version()` **once per task instance** when clearing
on the latest version. `get_latest_version_of_dag` deserializes the entire DAG
on every call, so clearing N task instances costs N full-DAG deserializations.
On a production DAG with ~8,400 tasks this measures at **~1.6s per cleared
task instance**:
| `run_on_latest_version` | affected TIs | duration |
|---|---|---|
| false | 50 | 5.3s |
| **true** | **50** | **84.7s** |
| true | 1 | 4.0s |
| false | 1 | 4.3s |
(same DAG, same run, same task, cleared via `POST
/api/v2/dags/{dag_id}/clearTaskInstances`; the 1-TI cases show the cost is
per-TI, not per-request)
The UI pre-selects "run on latest version" whenever the task instance's dag
version differs from the latest, so routine clears hit this path. Users read
the hung request as "clear did nothing" and retry, producing overlapping clear
requests that then fail with conflicts (the loser hits the
`task_instance_history` unique constraint → opaque 409).
## Fix
Cache both lookups per `dag_id` for the duration of the call (function-local
dicts, nothing outlives the request).
Measured on the same deployment after the fix: **84.7s → 4.4s** for the
50-TI clear; `run_on_latest_version=True` is now indistinguishable from `False`.
Side benefit: the clear is now version-consistent — previously a new version
serialized mid-request could pin task instances of the same clear to different
versions.
Same shape as #14048, which batched this function's per-row TaskReschedule
deletes.
## Test
Added `test_clear_task_instances_caches_latest_dag_lookups`: clears 4 TIs
with `run_on_latest_version=True` and asserts each lookup happens twice total
(once for the TI loop, once for the dag-run update) instead of once per TI, and
that all TIs land on the latest version.
---
^ Add meaningful description above
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
