cloud-fan opened a new pull request, #58631: URL: https://github.com/apache/spark/pull/58631
### What changes were proposed in this pull request? This PR follows [#55678](https://github.com/apache/spark/pull/55678) and [#58404](https://github.com/apache/spark/pull/58404). It restores the original planning behavior for the optimized single-column null-aware anti join (NAAJ): by default, Spark selects the specialized null-aware `BroadcastHashJoinExec` and builds the right side independently of `spark.sql.autoBroadcastJoinThreshold`. It adds the internal `spark.sql.nullAwareAntiJoinBroadcastThreshold` configuration as an explicit tuning option. The configuration defaults to `Long.MaxValue`, preserving the original behavior. When the estimated right-side size exceeds the configured value, Spark skips the specialized hash path and falls through to regular join planning. Setting it to `-1` always skips the specialized hash path. The AQE broadcast-mode validation introduced in #55678 is retained. The PR removes the additional NAAJ fallback decision and optimizer plumbing introduced in #58404. ### Why are the changes needed? #55678 made the NAAJ hash optimization conditional on the general automatic broadcast threshold, and #58404 added planning logic to preserve the exact nested-loop fallback and its build side. However, the automatic broadcast threshold and plan statistics are cost-planning heuristics; they do not prove that either input can be materialized and broadcast safely. Spark does not have a shuffle-capable NAAJ implementation. When regular join planning chooses the right side for the fallback, `BroadcastNestedLoopJoinExec` still broadcasts that same input while changing hash lookup from `O(M + N)` to nested-loop evaluation at `O(M * N)`. This may replace a broadcast failure with a much longer-running query without eliminating the memory risk. Choosing a small left side for the nested-loop fallback can still be useful for a known small-left/large-right query. The new NAAJ-specific threshold retains that option as an explicit tuning choice without changing the default algorithm based on an estimate that cannot guarantee memory safety. A generally scalable solution for large NAAJs still requires a non-broadcast implementation. ### Does this PR introduce _any_ user-facing change? Yes. Compared with the current behavior, the optimized single-column NAAJ uses the null-aware broadcast hash join by default even when the estimated right side exceeds `spark.sql.autoBroadcastJoinThreshold`. Users who need the size-based fallback can set `spark.sql.nullAwareAntiJoinBroadcastThreshold`. Its default value is `Long.MaxValue`, so the default behavior is the same as before #55678. ### How was this patch tested? Added coverage to `JoinSelectionHelperSuite` showing that the default NAAJ threshold is independent of `spark.sql.autoBroadcastJoinThreshold` and that an explicitly configured threshold rejects an oversized right side. Added coverage to `JoinSuite` showing that an explicit NAAJ threshold enables the build-left nested-loop fallback for a small-left/large-right query. The tests were not run locally. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
