dabla commented on PR #62922: URL: https://github.com/apache/airflow/pull/62922#issuecomment-5858273885
[#62922](https://github.com/apache/airflow/pull/62922) — AIP-104 Task Iteration (base of the stack) `.iterate()` runs a task over its input inside one task instance, with an async executor. Reworked so it resolves its input by index the way `.expand()` does, with the cross-product rule in one shared helper. No longer adds `iter_values`/`aiter_values` to the SDK. Level with main. Everything below is one commit on top of it and rebases once it merges. [#73688](https://github.com/apache/airflow/pull/73688) — AIP-104 Task Spreading `.batch(size=N).iterate(...)`: the same iteration spread over N task instances, items dealt round-robin. The scheduler fixes N before the task runs; each instance takes its share with a range over the resolved input. Naming still open, .spread() leads the devlist vote. [#73790](https://github.com/apache/airflow/pull/73790) — Chunked reads of a mapped upstream's `XCom` sequence LazyXComSequence fetched one item per request. Now one `GetXComSequenceSlice` per 32 items on both the sync and async paths, at most one chunk held, negative indices through the cached count. For a 17,000-item mapped upstream consumed by `.iterate()`, about 530 requests instead of 17,000. Chunk size is a constant for now. [#73807](https://github.com/apache/airflow/pull/73807) — `XComIterable.flatten()` An iterated task whose iterations return pages of items can hand them downstream as one sequence. The producer tallies the flattened length as it pushes, so the view knows its length without a read, and it holds one page at a time. Split out of [#62922](https://github.com/apache/airflow/pull/62922) because iteration does not need it, and documented here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
