cloud-fan opened a new pull request, #58516:
URL: https://github.com/apache/spark/pull/58516
### What changes were proposed in this pull request?
This backports #58404 to `branch-4.2`. It preserves the optimized
single-column null-aware
anti join (NAAJ) hash path without making the fallback worse than it was
before #55678.
The patch recognizes the structural single-column NAAJ form independently of
whether the hash
optimization is enabled. `SparkStrategies.JoinSelection` owns the complete
physical decision for
that form. A shared `NullAwareAntiJoinPlanning` decision selects either the
specialized null-aware
`BroadcastHashJoinExec`, when the right side is within the normal broadcast
threshold and the
generic fallback would also build the right side, or
`BroadcastNestedLoopJoinExec` with the exact
build side selected by the generic fallback.
The cherry-pick had one branch-specific conflict in `joins.scala`. The
resolution retains
`branch-4.2`'s existing Boolean implementation of
`canPlanAsBroadcastHashJoin` for equi-joins and
delegates its NAAJ branch to the new shared decision. The other changes
apply directly.
### Why are the changes needed?
The null-aware hash operator always builds the right side, while the generic
broadcast nested-loop
join can build either side. If the fallback chooses `BuildLeft`, forcing the
specialized hash path
can require broadcasting a much larger right side and regress a query that
previously had a valid
plan.
An over-threshold `BuildRight` input also used the nested-loop fallback
before #55678. Keeping the
hash optimization within the existing broadcast-size boundary preserves that
fallback and its
floating-point equality semantics. Ordinary semi and anti Aggregate pushdown
also retains the
SPARK-34081 broadcast-eligibility guard.
### Does this PR introduce _any_ user-facing change?
Yes. Relative to #55678, a recognized single-column NAAJ uses the null-aware
broadcast hash join
only when the generic fallback would build the right side and that side is
within the normal
broadcast-size threshold. Otherwise Spark retains the broadcast nested-loop
join.
### How was this patch tested?
The following focused tests passed on `branch-4.2` (111 tests total):
```
build/sbt \
"catalyst/testOnly
org.apache.spark.sql.catalyst.optimizer.JoinSelectionHelperSuite" \
"catalyst/testOnly
org.apache.spark.sql.catalyst.optimizer.LeftSemiAntiJoinPushDownSuite" \
"sql/testOnly org.apache.spark.sql.JoinSuite -- -z SPARK-36082"
```
`dev/lint-scala` also passed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
