cloud-fan opened a new pull request, #58870: URL: https://github.com/apache/spark/pull/58870
### What changes were proposed in this pull request? This PR follows #58631. For a nonnegative `spark.sql.optimizeNullAwareAntiJoin.broadcastThreshold`, this PR uses the larger of that value and `spark.sql.autoBroadcastJoinThreshold` as the effective threshold. A negative dedicated threshold remains unlimited. The implementation also avoids reading plan statistics when the dedicated threshold is unlimited, or when both thresholds disable broadcasting. The configuration documentation and focused `JoinSelectionHelperSuite` coverage are updated for these semantics. ### Why are the changes needed? If the dedicated NAAJ threshold is lower than the automatic broadcast threshold, it can reject the specialized null-aware broadcast hash join even though regular join planning will still broadcast the right side. That fallback uses a nested-loop representation and changes hash lookup from `O(M + N)` to `O(M * N)` without avoiding the broadcast. Flooring the dedicated threshold by the automatic threshold prevents this performance regression. The dedicated threshold remains useful for allowing the specialized NAAJ hash join above the automatic threshold. ### Does this PR introduce _any_ user-facing change? Yes, on unreleased `master` and for an internal configuration. A nonnegative dedicated NAAJ threshold can no longer disable the specialized hash optimization for inputs that are within `spark.sql.autoBroadcastJoinThreshold`. To disable both, set both thresholds to a nonpositive value. ### How was this patch tested? Added focused `JoinSelectionHelperSuite` tests for the automatic-threshold floor, unlimited and disabled short-circuit paths, unknown statistics, and the NAAJ optimization flag. The following local command was attempted: ``` build/sbt "catalyst/testOnly org.apache.spark.sql.catalyst.optimizer.JoinSelectionHelperSuite" ``` The test process did not start because the local environment could not resolve Maven and SBT repositories while loading missing SBT plugins (`UnknownHostException`). ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
