ephraimbuddy opened a new issue, #76446:
URL: https://github.com/apache/airflow/issues/76446
### What happens
After #76420, a clear with "run on latest version" may re-run a mapped
task's producer. When it does, `DagRun._check_for_removed_or_restored_tasks`
restores every REMOVED index of that task, because the new length isn't known
yet. Once the producer finishes, `_revise_map_indexes_if_mapped` removes any
index that is out of range again.
Some indexes were REMOVED because an earlier producer run returned a shorter
list, not because the task left the Dag. For those:
- The restore logs "Restoring task … which was previously removed from DAG"
and increments `task_restored_to_dag`. Both are wrong for this case.
- The restore calls `prepare_db_for_next_try`, which adds a try. If the new
list is still short, the scheduler removes the index again, and its history
keeps a try that never ran.
### How to reproduce
1. A mapped task expands to 3, then a producer re-run shrinks it to 1, so
indexes 1 and 2 are REMOVED at try 1.
2. Clear only the producer with "run on latest version". Indexes 1 and 2 get
a try 2 in state `None`.
3. The producer returns `[1]` again. Try 2 of both indexes ends REMOVED with
no start date, alongside the archived try 1.
### What should happen
Restoring an index whose count isn't known yet shouldn't be recorded as a
Dag re-add, and shouldn't add a try that may never run.
### Notes
- The restore has to happen in `_check_for_removed_or_restored_tasks`,
because `_revise_map_indexes_if_mapped` only runs for schedulable instances.
When every index of a task is REMOVED, nothing else brings it back.
- Options:
- Restore without a new try, and with its own log line and no metric,
while the count is unresolved.
- Or let `_revise_map_indexes_if_mapped` restore REMOVED indexes that are
back in range, and keep the integrity-check restore only for tasks with no live
index.
- Raised in review of #76420 (`dagrun.py` restore block).
---
Drafted-by: Claude Code (Opus 5.5); reviewed by @ephraimbuddy before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
