Hyukjin Kwon created SPARK-61124:
------------------------------------
Summary: Check the arguments of nearestByJoin, and take them, in
the Python client as PySpark does
Key: SPARK-61124
URL: https://issues.apache.org/jira/browse/SPARK-61124
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark 4.3.0's {{DataFrame.nearestByJoin(other, rankingExpression, numResults,
mode, direction, *, joinType="inner")}} checks its arguments before it builds
the plan, as the Scala {{Dataset.validateNearestByJoinArgs}} does. It raises
{{AnalysisException}} of these error classes:
* {{NEAREST_BY_JOIN.NUM_RESULTS_OUT_OF_RANGE}}, for a {{numResults}} not
between 1 and 100000
* {{NEAREST_BY_JOIN.UNSUPPORTED_JOIN_TYPE}}, for a {{joinType}} that is not
{{inner}}, {{left}} or {{leftouter}}
* {{NEAREST_BY_JOIN.UNSUPPORTED_MODE}}, for a {{mode}} that is not {{approx}}
or {{exact}}
* {{NEAREST_BY_JOIN.UNSUPPORTED_DIRECTION}}, for a {{direction}} that is not
{{distance}} or {{similarity}}
{{numResults}}, {{mode}} and {{direction}} have no defaults, and {{joinType}}
is keyword-only.
The Python client checks none of them, so an invalid argument fails only when
the query runs, or not at all. Its {{numResults}}, {{mode}} and {{direction}}
default to {{1}}, {{"brute"}} and {{"asc"}}, values that PySpark rejects.
The fix is to check the arguments, and to take them, as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
