MrHappyEnding opened a new pull request, #57791: URL: https://github.com/apache/spark/pull/57791
### What changes were proposed in this pull request? Restrict `OptimizeJoinCondition` to rewriting null-safe equality patterns only at the root of a join condition or beneath `AND`/`OR`. The rule no longer performs this rewrite beneath `NOT`, where the difference between `NULL` and `FALSE` is observable. ### Why are the changes needed? When exactly one operand is `NULL`, the original pattern returns `NULL`, while `<=>` returns `FALSE`. This difference does not matter at the root of a join condition, but under `NOT` it can cause Spark to incorrectly keep extra rows. ### Does this PR introduce *any* user-facing change? Yes. Join conditions containing this pattern beneath `NOT` now return the correct rows. For the SPARK-58384 reproduction, the result changes from four rows to only: ```text [0,10,1,22] ``` ### How was this patch tested? Added tests covering: - No rewrite beneath `NOT`. - Continued rewriting beneath `AND` and `OR`. - The end-to-end join result with null values. Ran: ```text build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.optimizer.OptimizeJoinConditionSuite' build/sbt 'sql/testOnly org.apache.spark.sql.DataFrameJoinSuite -- -z "SPARK-58384"' build/sbt 'catalyst/scalastyle' 'catalyst/Test/scalastyle' 'sql/Test/scalastyle' ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
