The GitHub Actions job "Tests (AMD)" on airflow.git/fix-per-ti-latest-dag-lookup-in-clear has failed. Run started by GitHub user joaopamaral (triggered by joaopamaral).
Head commit for run: 2634fd07521afdd90c21a42431f93075b550cc97 / Joao Amaral <[email protected]> Cache latest-DAG lookups in clear_task_instances clear_task_instances calls scheduler_dagbag.get_latest_version_of_dag() and DagVersion.get_latest_version() once per task instance when clearing on the latest version. get_latest_version_of_dag deserializes the entire DAG on every call, so clearing N task instances costs N full-DAG deserializations. On a production DAG with ~8,400 tasks this measures at ~1.6s per cleared task instance: clearing one task together with its ~50 downstream tasks takes ~85s, while the identical clear with run_on_latest_version=False takes ~5s. Users read the hung request as "clear did nothing" and retry, producing overlapping clear requests that then fail with conflicts. The UI pre-selects "run on latest version" whenever the task instance's dag version differs from the latest, so routine clears hit this path. Cache both lookups per dag_id for the duration of the call. Besides removing the O(n) cost (measured ~85s -> ~4.4s for the 50-task clear), this makes the clear consistent if a new version is serialized while the request is running: previously task instances in the same clear could be pinned to different versions. Same shape as #14048, which batched this function's per-row TaskReschedule deletes. Report URL: https://github.com/apache/airflow/actions/runs/32144145078 With regards, GitHub Actions via GitBox --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
