stuhood opened a new issue, #25302: URL: https://github.com/apache/datafusion/issues/25302
### Is your feature request related to a problem or challenge? In PR #24600 (*"Adapt existing partitioning for unsatisfied inputs in co-partitioned joins"*), `enforce_distribution_relationships` was introduced to adapt unsatisfied children in co-partitioned joins to an existing satisfied reference partitioning. When selecting `best_satisfied_child`: https://github.com/apache/datafusion/blob/f40d978a4c72bcbbcb6a45fb4c81e95f4e3cfc73/datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs#L1181-L1229 Notice the current selection logic: 1. When `satisfied_children.len() > 1`: DataFusion inspects `PlanSize` across the candidate satisfied children and only selects a reference candidate if there is a unique, strictly larger winner (`size_a > size_b`), avoiding arbitrary tie-breaking: https://github.com/apache/datafusion/blob/f40d978a4c72bcbbcb6a45fb4c81e95f4e3cfc73/datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs#L1213-L1226 2. But when `satisfied_children.len() == 1`: DataFusion unconditionally selects that single satisfied child as the reference partitioning without checking its size relative to the unsatisfied children that will be adapted to it: https://github.com/apache/datafusion/blob/f40d978a4c72bcbbcb6a45fb4c81e95f4e3cfc73/datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs#L1185-L1188 **The Problem in Asymmetric Joins**: In an asymmetric join where only one table has a declared partition key matching the join condition (for example, a small dimension table with 100 rows is range-partitioned, but a large fact table with 100M rows is unpartitioned on that join key): - `satisfied_children` contains only the small dimension table (`len == 1`). - DataFusion chooses the 100-row dimension table as the reference child. - The 100M-row fact table is forced to repartition (via `ref_partitioning.adapt(...)`) to match the small table's partition boundaries. - In distributed environments, this causes the 100M-row table to be shuffled across the network to match the 100-row table, rather than keeping the large table local and shuffling the small table (or falling back to symmetric hash partitioning / broadcast). ### Describe the solution you'd like When `satisfied_children.len() == 1` (or before selecting any reference child), compare the `PlanSize` of the candidate satisfied reference child against the `PlanSize` of the unsatisfied children that would be adapted to it. If an unsatisfied child is strictly larger than the satisfied reference child (`unsatisfied_size > reference_size`): - Do not select the smaller child as the reference. - Return `None` for `best_satisfied_child` so that the join falls back to standard symmetric distribution (e.g. symmetric hash repartitioning or broadcast), preventing an asymmetric 1-sided shuffle of the larger relation. ### Describe alternatives you've considered Downstream query planners currently have to defend against this by artificially withholding / refusing to stamp partition metadata on smaller tables in asymmetric joins to prevent DataFusion from shuffling the larger partner to match the smaller table. Having DataFusion check `PlanSize` before adapting larger unsatisfied children to a smaller satisfied child would make `enforce_distribution_relationships` robust against asymmetric joins out of the box. ### Additional context - Introduced in PR #24600: https://github.com/apache/datafusion/commit/d109f1b133f7d90f4806bc89b8945df0b6e029ae - Relevant code in `enforce_distribution.rs`: https://github.com/apache/datafusion/blob/f40d978a4c72bcbbcb6a45fb4c81e95f4e3cfc73/datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs#L1181-L1229 - Related PR: PR #24766 (Range partitioning scaling and preservation). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
