dabla commented on PR #62922: URL: https://github.com/apache/airflow/pull/62922#issuecomment-5820637079
**Scope change: this PR is now Task Iteration only.** Following the feedback from today's Airflow dev call, I have split AIP-104 in two so this PR stays focused on `.iterate()` / `.iterate_kwargs()`: - **#62922 (this PR), "AIP-104: Task Iteration"**: `IterableOperator`, `AsyncAwareExecutor`, `XComIterable`, the async resolution of expand inputs and XComArgs, the runner's sub-task execution with checkpoints, outlet-event replay and `on_kill` propagation. Head is a3d7f65d86. - **#73683, "AIP-104: Task Batching"** (draft): `.batch(size=N).iterate(...)`, `MappedIterableOperator`, `BatchedExpandInput`, the runtime batch size resolved by the scheduler from `mapped_length`, and the related docs and tests. It is stacked on this PR, so its diff includes this one until this merges; the batching work is its last commit. The `size` naming question and the devlist vote move there too. The split commit is a3d7f65d86. Besides removing batching it changes one thing in what stays here: `.iterate()` no longer funnels through a `size=0` `BatchedOperator`. `OperatorPartial` and `_TaskDecorator` build the `IterableOperator` directly, and `_expand` owns the mapped-operator construction on both classes again, returning the operator so the public methods wrap it. The batching PR builds on those helpers instead of undoing them. Everything discussed in the resolved threads on this PR is unchanged apart from that. The description and title are updated accordingly. Suggested reading order for the remaining diff: `iterableoperator.py`, then `task_runner.py`, then `executor.py`, then `xcom.py`, then the async plumbing in `expandinput.py`, `xcom_arg.py` and `lazy_sequence.py`. --- Drafted-by: Claude Fable 5.1; reviewed by @dabla before posting -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
