abhijeets25012-tech opened a new pull request, #76669: URL: https://github.com/apache/airflow/pull/76669
The Dag integrity check restores every removed index of a mapped task while the producer's new length is still unknown. Those indexes can be out of range again once the producer finishes short, so recording the restore as a Dag re-add and allocating an attempt leaves history rows for work that never ran. The restore now keeps such an index at its current try and logs at debug level while the map length is unresolved. Once the producer finishes, `_revise_map_indexes_if_mapped` removes the indexes that are still out of range, so only indexes that are back in range survive. This follows the first option in the issue; the alternative was to move the restore into `_revise_map_indexes_if_mapped`, which only runs for schedulable instances and would not bring back a task whose indexes are all removed. closes: #76446 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: Qoder following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- ##### Testing * `airflow-core/tests/unit/models/test_dagrun.py -k "restoring_shrink_removed_map_index or restoring_removed_task_allocates_attempt_once or verifying_removed_map_index_does_not_allocate_attempt"`: 6 passed. The new test covers a producer re-run that shrinks the list and then a clear of only the producer: the restored index keeps its id and try number, `task_restored_to_dag` is not incremented, and when the producer returns the short list again the index ends REMOVED with a single attempt in its history. The test fails with the source change reverted. * The rest of `test_dagrun.py` has pre-existing failures in this environment (`ImportError: cannot import name 'iter_route_contexts' from 'fastapi.routing'` when a test starts the in-process API server); they reproduce on a clean checkout without this change. * `ruff check` and `ruff format --check` pass on both changed files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
