[
https://issues.apache.org/jira/browse/SPARK-60819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhan updated SPARK-60819:
------------------------------
Description:
# Problem
A transpiled a == b becomes a CASE WHEN expression. Spark cannot extract a join
key from it, so a join whose only condition is a transpiled == runs as a
CartesianProduct. With spark.sql.crossJoin.enabled=false, SPARK-58780 makes
that join fail with the cartesian product error, as it does without
transpilation. This ticket would make it fast instead.
# Proposal
- Lower a == b to a null-safe equal (a <=> b) in a join condition, so Spark can
use it as a join key.
- Python's None == None is True, which matches <=>, so ints and strings need no
non-null condition.
- Floats are out of scope for now: NaN <=> NaN is true in Spark and False in
Python.
- Operators other than == (e.g. <) have no join key, so they keep the error.
# Notes
- The cross-join guard added in SPARK-58780 must treat the lowered key as a
real key, so it does not send that call back to Python.
- The == lowering code is also changed by #59730 and #58901.
> Use == as a join key in transpiled join conditions
> --------------------------------------------------
>
> Key: SPARK-60819
> URL: https://issues.apache.org/jira/browse/SPARK-60819
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark, SQL
> Affects Versions: 4.4.0
> Reporter: Jing Zhan
> Priority: Major
>
> # Problem
> A transpiled a == b becomes a CASE WHEN expression. Spark cannot extract a
> join key from it, so a join whose only condition is a transpiled == runs as a
> CartesianProduct. With spark.sql.crossJoin.enabled=false, SPARK-58780 makes
> that join fail with the cartesian product error, as it does without
> transpilation. This ticket would make it fast instead.
> # Proposal
> - Lower a == b to a null-safe equal (a <=> b) in a join condition, so Spark
> can use it as a join key.
> - Python's None == None is True, which matches <=>, so ints and strings need
> no non-null condition.
> - Floats are out of scope for now: NaN <=> NaN is true in Spark and False in
> Python.
> - Operators other than == (e.g. <) have no join key, so they keep the error.
> # Notes
> - The cross-join guard added in SPARK-58780 must treat the lowered key as a
> real key, so it does not send that call back to Python.
> - The == lowering code is also changed by #59730 and #58901.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
