The GitHub Actions job "Tests (AMD)" on airflow.git/backport-67672-v3-3-test has succeeded. Run started by GitHub user vatsrahul1001 (triggered by vatsrahul1001).
Head commit for run: 3403ec60709cf837823aeb1f42a3eb80ffd1b794 / Kaxil Naik <[email protected]> Avoid duplicate trigger-rule upstream-count queries per scheduling pass (#67672) TriggerRuleDep runs a `SELECT task_id, count(*) ... GROUP BY task_id` per downstream task to size its upstream set, but only when an upstream is mapped. When many downstreams share the same mapped upstream, each issues an identical query within the same scheduling pass. Memoize the result on DepContext (one scheduling pass, same lifetime as finished_tis), keyed by (dag_id, run_id, frozenset of direct-upstream task_ids). Only the simple case is cached, where the predicate is exactly `task_id IN (upstream_ids)`; downstreams inside a mapped task group keep their own per-instance map-index query. The cache is cleared in _get_ready_tis when a mapped task expands and changes its instance count. * Add newsfragment for trigger-rule upstream-count memoization * Clear upstream-count memo when a mapped task grows via revise _get_ready_tis already drops the trigger-rule upstream-count memo when a mapped task expands, but not when _revise_map_indexes_if_mapped grows an already-expanded task and adds new instances. A downstream evaluated later in the same pass could then read a stale, pre-grow count. Clear the memo in that case too, and add a regression test that drives _get_ready_tis with a fixed order so the count is recomputed after the grow (it asserts the query runs twice; fails without the clear). * Keep upstream-count memo across the UP_FOR_RESCHEDULE context rebuild are_dependencies_met rebuilds the DepContext with attrs.evolve for every UP_FOR_RESCHEDULE task instance, and UP_FOR_RESCHEDULE is in SCHEDULEABLE_STATES, so those instances reach TriggerRuleDep through the evolved context. attrs.evolve only carries over fields that __init__ accepts, so the memo's init=False meant each of them got a fresh empty dict: they neither read the memo nor warmed it for anything else. Reschedule-mode sensors fanned out behind a mapped upstream are the exact shape the memo collapses, so they were the one case opting out of it. Drop init=False (keeping the dict out of the repr instead), and route both invalidation sites through a DepContext method rather than having dagrun reach in and clear a dict only TriggerRuleDep writes. Also drop a redundant list() around _revise_map_indexes_if_mapped, which returns a list since #69565, and correct the staleness comment to name both invalidation points. (cherry picked from commit ecd4d951da1907499ea1637421473860aef18932) Report URL: https://github.com/apache/airflow/actions/runs/30630188089 With regards, GitHub Actions via GitBox --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
