pan3793 opened a new pull request, #58569: URL: https://github.com/apache/spark/pull/58569
### What changes were proposed in this pull request? Add `LeftSingle` and `ExistenceJoin` to the two join-type lists that decide whether a join's right side may be duplicated: - `EnsureRequirements.canReplicateRightSide`, which gates partially clustered distribution for storage-partitioned joins (`spark.sql.sources.v2.bucketing.partiallyClusteredDistribution.enabled`). - `OptimizeSkewedJoin.canSplitLeftSide`, which gates splitting a skewed left partition under AQE. Both lists already hold `Inner`, `Cross`, `LeftSemi`, `LeftAnti` and `LeftOuter`. The four helpers now state the invariant they encode: duplicating a side is safe for any join that keeps no unmatched row from that side, since each row of the other side still lands in one task that sees every row for its key. ### Why are the changes needed? `LeftSingle` and `ExistenceJoin` emit each left row exactly once against every right row for its key, the same shape as `LeftOuter`. Both optimizations were skipped outright for them, so a storage-partitioned existence join or scalar-subquery join with a partially clustered side fell back to one task per key group, and a skewed left partition of such a join was never split. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? - `KeyGroupedPartitioningSuite`: new test with three splits for one key on the larger left side, asserting for an existence join and a left single join that partially clustered distribution keeps the left splits apart, replicates the right side, runs over five partitions instead of three key groups, adds no shuffle and returns the right rows. - `AdaptiveQueryExecSuite`: the SPARK-29544 skew join test gains an existence join case for both shuffle hints and a left single join case for the hash hint, each asserting the planned join type and that the skewed left side is split with the right side untouched. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Fable 5.1) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
