benbellick opened a new issue, #27268: URL: https://github.com/apache/datafusion/issues/27268
Skipping aggregate repartitioning can preserve a heavily skewed input layout, leaving most of the grouping work on one worker even when more execution partitions are available. In [Jay's benchmark on #27135](https://github.com/apache/datafusion/pull/27135#pullrequestreview-5812854498), one of four files held 97% of 16M rows. With `preserve_file_partitions` enabled and mostly unique groups, the `SinglePartitioned` plan took 4.11 times as long as the shuffle plan at `target_partitions=8`. The two-argument `date_bin` showed the same behavior. Consider per-partition row counts when deciding whether to skip aggregate repartitioning, so a shuffle can still be used when its load-balancing benefit outweighs its cost. Related to #19269, which describes the same trade-off for hash-subset partitioning. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
