ephraimbuddy opened a new pull request, #76420: URL: https://github.com/apache/airflow/pull/76420
Clearing a Dag run with "run on latest version" marked every instance of a mapped task `removed` when the producer of its input was cleared in the same request. They stayed `removed` after the producer ran again, so the mapped task never re-ran and its downstream tasks ran as if it had finished. Clearing on the latest version calls `verify_integrity`, which re-checks the mapped task's length. Since #74222 that length is read from the producer's current attempt, and a cleared producer's new attempt has no XCom yet. The check then took the branch that removes every expanded instance when the mapping can't be resolved. Nothing expands a removed index again: `_revise_map_indexes_if_mapped` only revises schedulable instances and doesn't recreate removed indexes, and no `map_index=-1` placeholder is left to expand. #71425 later made the same check run for running and queued runs as well. The expanded instances are now kept while the mapping can't be resolved. The scheduler already trims or extends them once the upstreams finish, as it did before #74222, when the length could still be read from the producer's previous run. **Changed existing test:** `test_mapped_literal_to_xcom_arg_verify_integrity` asserted that switching a task from literal to XCom mapping, before the producer has run, removes all of its instances. Those instances were never expanded again either, so the task never ran under its new mapping. The test now checks that they are kept, then revised to the producer's length once it runs. Tests run locally (SQLite): - The new `test_clear_latest_with_producer_keeps_mapped_task_for_revision` (finished and running runs, producer re-run with a shorter and a longer list) and the updated test fail without the fix and pass with it. - `test_dagrun.py`, `test_cleartasks.py`, `test_taskinstance.py`, `test_mappedoperator.py`, `test_dag.py`, `test_dynamic_region.py`, `test_task_coordinates.py`, `test_xcom_arg.py`, `test_backfill.py`, the public Dag run and task instance API tests, and the Execution API Dag run and task instance tests: 2678 passed. - `test_scheduler_job.py -k "mapped or expand or integrity or clear or version or removed"`: 101 passed. closes: #76377 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5.5) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
