[
https://issues.apache.org/jira/browse/SPARK-61124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61124:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Check the arguments of nearestByJoin, and take them, in the Python client as
> PySpark does
> -----------------------------------------------------------------------------------------
>
> Key: SPARK-61124
> URL: https://issues.apache.org/jira/browse/SPARK-61124
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> PySpark 4.3.0's {{DataFrame.nearestByJoin(other, rankingExpression,
> numResults, mode, direction, *, joinType="inner")}} checks its arguments
> before it builds the plan, as the Scala {{Dataset.validateNearestByJoinArgs}}
> does. It raises {{AnalysisException}} of these error classes:
> * {{NEAREST_BY_JOIN.NUM_RESULTS_OUT_OF_RANGE}}, for a {{numResults}} not
> between 1 and 100000
> * {{NEAREST_BY_JOIN.UNSUPPORTED_JOIN_TYPE}}, for a {{joinType}} that is not
> {{inner}}, {{left}} or {{leftouter}}
> * {{NEAREST_BY_JOIN.UNSUPPORTED_MODE}}, for a {{mode}} that is not {{approx}}
> or {{exact}}
> * {{NEAREST_BY_JOIN.UNSUPPORTED_DIRECTION}}, for a {{direction}} that is not
> {{distance}} or {{similarity}}
> {{numResults}}, {{mode}} and {{direction}} have no defaults, and {{joinType}}
> is keyword-only.
> The Python client checks none of them, so an invalid argument fails only when
> the query runs, or not at all. Its {{numResults}}, {{mode}} and {{direction}}
> default to {{1}}, {{"brute"}} and {{"asc"}}, values that PySpark rejects.
> The fix is to check the arguments, and to take them, as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
