jto opened a new pull request, #40612: URL: https://github.com/apache/beam/pull/40612
Addresses #40607. Flink's `rebalance()` starts each sender's round-robin at a random channel. When senders emit only a few elements each, as when a `Reshuffle` redistributes file names or splittable DoFn restrictions, the resulting assignment is close to random: some receivers get no elements while others get several times the average. ## What changes In batch, `Reshuffle` is translated with a new `DeterministicRebalance`: sender `i` of `M` sends its `j`-th element to channel `(floor(i * N / M) + j) mod N` for `N` receivers, so already balanced inputs stay balanced. Elements are tagged with their sender's index and parallelism, since a Flink `Partitioner` is not told which sender it runs in. Streaming pipelines keep using `rebalance()`. ## Testing New `DeterministicRebalanceTest`: balance with few elements per sender, a single sender, fewer and more senders than receivers, sender key round trips, and an end-to-end batch `Reshuffle` keeping all elements. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
