That’s exactly the point though. This AIP/PR as currently proposed does not operate the same as itertools.batchhed, so naming it the same is confusing.
From the original post: > Ours is the opposite [to itertools.batched]: the number of task instances is > given, and the per-instance share is derived > On 18 Sep 2026, at 20:48, Jarek Potiuk <[email protected]> wrote: > > How about we follow Python 3.12 convention: > > itertools.batched(it, n) > > The ecosystem didn't fully converge on one spelling. more_itertools.batched > is now just an alias for the stdlib function, but chunked (any iterable → > lists) and sliced (sequences → real slices, type-preserving) are still > separate functions with separate semantics, because tuples aren't always > what you want. > > Worth noting for your original question: the stdlib settled on positional n, > not a keyword. So batched(it, 100), not batched(it, size=100) — and the > parameter-order question was considered explicitly rather than falling out > by accident. If you're designing a .batch(), that's the precedent to match > or consciously break. > > > J. > > > On Fri, Sep 18, 2026 at 7:50 PM Constance Martineau via dev < > [email protected]> wrote: > >> My preference is option 3. Of the names on the table, I'd rank them roughly >> `spread`, then `batch`, then `shard` with `partition` last. `spread` works >> best in my opinion because the preposition makes the intended behaviour >> really obvious. "spread across 17" can't be misread as "17 each", which is >> the risk with `batch(count=N)`. Agree that `shard` and `partition` has >> specific connotations, and Partitions are now a feature in Airflow so if we >> chose that we'd be introducing a name class. >> >> If the consensus ends up being option 2, I'd pick `tasks=` over `count=`, >> since `count` doesn't say count of what and can be misinterpreted as >> per-instance item count. Either way though, I'd make this keyword-only >> (`spread(17)` is just as ambigious as `batch(17)`) and agreed that we >> should raise a clear error if someone passes `size=`. >> >> Whichever option wins, I think you still need to make the round-robin >> distribution very clear in the docs, along with the reasoning, since it >> might not be immediately obvious to Dag authors why we chose round-robin. >> >> Constance >> >> On Fri, Sep 18, 2026 at 11:22 AM Ash Berlin-Taylor <[email protected]> wrote: >> >>> I’d like more time to think about this please. >>> >>> -ash >>> >>>> On 17 Sep 2026, at 10:24, Blain David <[email protected]> wrote: >>>> >>>> Hi all, >>>> >>>> While reviewing the AIP-104 PR ( >>> https://github.com/apache/airflow/pull/62922), Amogh raised a naming >>>> concern about the Dynamic Task Batching API that I would like to settle >>> here before the PR merges, >>>> since renaming afterwards would be a breaking change. >>>> >>>> ## What `.batch(size=N)` does today >>>> >>>> `task.batch(size=17).iterate(url=urls)` creates 17 mapped task >> instances >>> through Dynamic Task >>>> Mapping, and each of them iterates over its share of `urls` in a single >>> task instance using >>>> Iterable Tasks. The items are distributed round robin: item `i` goes to >>> task instance `i % 17`. >>>> So `size` is the number of task instances, not a chunk length, and the >>> task instances do not >>>> receive contiguous slices of the input. >>>> >>>> The round robin scheme is deliberate. The number of task instances has >>> to be fixed before the >>>> underlying iterable is consumed, because the scheduler needs the >> mapping >>> cardinality up front. >>>> With round robin that number is simply `size`. Contiguous chunking, the >>> way `itertools.batched` >>>> works, would need `ceil(len(items) / size)` task instances, which is >>> unknowable until a possibly >>>> unbounded or paginated iterable has been fully drained. That would >>> defeat the purpose of iterating >>>> lazily. >>>> >>>> ## The concern >>>> >>>> The name reads like `itertools.batched(iterable, size)`, where `size` >> is >>> the chunk length and the >>>> number of chunks is derived from it. Ours is the opposite: the number >> of >>> task instances is given, >>>> and the per-instance share is derived. The class docstring and the Task >>> SDK docs currently spend a >>>> full paragraph undoing that first impression. A name that carries the >>> meaning directly would not >>>> need it. >>>> >>>> ## Options >>>> >>>> 1. Keep `.batch(size=N)` and rely on documentation. Lowest churn, but >>> the mismatch with >>>> `itertools.batched` stays and every new reader has to be corrected. >>>> >>>> 2. Keep the method, rename the parameter: `.batch(count=N)` or >>> `.batch(tasks=N)`. Keeps the >>>> "Dynamic Task Batching" vocabulary from the AIP, and `count` or >>> `tasks` says what the number >>>> is. `.batch(tasks=17).iterate(url=urls)` reads as "spread over 17 >>> tasks, then iterate". >>>> >>>> 3. Rename the method as well, to something that describes partitioning >>> rather than chunking, for >>>> example `.partition(count=N)`, `.shard(count=N)` or >>> `.spread(across=N)`. Most descriptive, but >>>> drifts from the AIP's own terminology, and "shard" and "partition" >>> carry data-engineering >>>> connotations that may suggest contiguous ranges just as much as >>> "batch" does. >>>> >>>> My preference is option 2 with `count=`. It keeps the API shape and the >>> AIP terminology, fixes >>>> the misleading part, and costs nothing now because the feature is >>> unreleased. `size` would be >>>> kept as a rejected alias raising a clear error rather than silently >>> accepted, so nobody copies >>>> the old spelling from an early draft. >>>> >>>> Unless there are objections or a better name comes up, I will treat >> this >>> as lazy consensus in >>>> 72 hours and update the PR, the docs and the AIP page accordingly. >>>> >>>> Thanks, >>>> David >>>> >>>> General (Internal Property) >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>> >> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
