bnspopi opened a new pull request, #77351: URL: https://github.com/apache/airflow/pull/77351
When a mapped task has already expanded and its producer is then re-run and returns an empty list, every existing index is marked `REMOVED` and nothing is left `SKIPPED`. A first run that maps to no values marks the placeholder `SKIPPED` instead, so downstream tasks behave differently after a clear, as #76445 lays out: an `all_success` downstream runs, and a `none_failed_min_one_success` downstream becomes `upstream_failed` and fails the run. This takes the option suggested in the issue: - `DagRun._revise_map_indexes_if_mapped` and `DagRun._check_for_removed_or_restored_tasks` now mark the lowest existing index `SKIPPED` and the others `REMOVED` when the map resolves to no values. - `none_failed_min_one_success` leaves removed upstreams out when it checks whether every upstream was skipped (`skipped and skipped == upstream - removed`). Without this the downstream still became `upstream_failed`, because the removed indexes were counted as upstreams that were not skipped. The `skipped and` guard keeps the all-removed case as it is today. This changes behaviour, as the issue notes: after such a re-run, `all_success` downstreams that run today are skipped. `test_calls_to_verify_integrity_with_mapped_task_zero_length_at_runtime` pinned the all-`REMOVED` result and is updated. Two things reviewers may want to weigh: - The trigger rule change is not limited to mapped upstreams. Any `none_failed_min_one_success` task whose upstreams are a mix of skipped and removed, with no success, is now skipped rather than `upstream_failed`. - The lowest index is marked `SKIPPED` whatever state it was in, the same way the other indexes are marked `REMOVED` whatever state they were in. Tests, run on Linux with SQLite: - New `test_mapped_task_mapped_to_zero_is_skipped_on_rerun_too` runs `size >> mapped.expand(arg=size.output) >> after` through the scheduling decisions for a first run and for a re-run, with `all_success`, `none_failed_min_one_success` and `all_done` on `after`, and checks that both end the same way and that `verify_integrity` afterwards does not turn the skipped instance into a removed one. On `main` the three re-run cases fail. - `test_dagrun.py`, `test_trigger_rule_dep.py`, `test_mappedoperator.py`, `test_taskinstance.py`, `test_scheduler_job.py`: 1653 passed, 21 skipped, 1 xfailed, against 1647 passed on `main` with no failures on either. Not tested: PostgreSQL and MySQL, mapped task groups, and a real scheduler run. @sundeep8967 and @SYED-IRTEZA-HUSSAIN-RIZVI asked about this issue. I did not find a PR for it; I am happy to close this one if one of you already has it in progress. related: #76445 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (Claude Code) Generated-by: Claude Code following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
