LuciferYang opened a new pull request, #57721: URL: https://github.com/apache/spark/pull/57721
### What changes were proposed in this pull request? `PushDownJoinThroughUnion` rewrites `Join(Union(c1, ..., cN), right)` into `Union(Join(c1, right), ..., Join(cN, right))`, which duplicates the right side once per Union branch. The rule gated on `canPlanAsBroadcastHashJoin`, which holds when *either* side of the join is broadcastable. An inner join can broadcast its left side, so a small `Union` on the left passed the guard even when the right side was large, and the planner then built from the left in every branch, leaving each copy of the right side as a plain probe input. Nothing reuses a bare probe-side scan (`ReuseExchangeAndSubquery` only matches `Exchange`), so the right side was read once per branch instead of once in total. This PR adds `broadcastsRightForEveryBranch`, which requires every join the rewrite would produce to broadcast its right side. It follows the planner's precedence: a hinted broadcast first, then sizes, and it skips the size check when a `SHUFFLE_MERGE` or `SHUFFLE_REPLICATE_NL` hint means the planner would not pick a broadcast hash join at all. A shuffle hash hint needs no such check because it already makes `canPlanAsBroadcastHashJoin` reject the join. The check is per branch rather than against the whole `Union`: a `Union`'s estimated size is the sum of its children, so the build side after the rewrite is decided against a smaller left, and `branch < right < union` is exactly the case where the two disagree. Five existing catalyst tests moved from an empty `LocalRelation` union to a new `largeStatsUnion` helper. This is a test-fixture artifact, not a behavior change for real plans: an empty `LocalRelation` is estimated at 0 bytes while every non-leaf right side is estimated at 1 byte at least, so `getSmallerSide` picked the left one and the rule no longer fired. Two things worth a reviewer's attention. The guard and `canPlanAsBroadcastHashJoin` are now two partial models of the same planner decision; converging them behind a shared `getPlannedBroadcastBuildSide` in `JoinSelectionHelper` would change an API used by `PushDownLeftSemiAntiJoin` and two suites, so it is left to a follow-up. And the guard reads estimates from a `Once` batch, so later rules and AQE can still re-decide the build side; it predicts rather than guarantees. ### Why are the changes needed? The rewrite is only a win when the right side is broadcast. When it is probed instead, the rule turns one scan of the right side into N. ### Does this PR introduce _any_ user-facing change? No. `spark.sql.optimizer.pushDownJoinThroughUnion.enabled` defaults to false. When it is enabled, queries whose right side would not be broadcast in every branch are no longer rewritten, which can mean giving up a pushdown that previously happened. The config documentation is updated accordingly. ### How was this patch tested? New tests in `PushDownJoinThroughUnionSuite` (catalyst and core) cover the build-side direction, the per-branch comparison, the left outer case, and each hint ordering; the core suite counts `FileSourceScanExec` nodes over the right-hand table end to end. Every new test was checked against the unfixed rule: the catalyst cases fail on the plan shape and the core cases report two scans of the right side. Existing core tests now assert how many `Join` nodes the optimized plan holds, so a fixture that stops being large enough to keep the right side on the build side fails instead of silently covering nothing. The TPC-DS benefit this rule targets has not been re-measured with the guard in place. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
