[
https://issues.apache.org/jira/browse/SPARK-60968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-60968:
------------------------------------
Assignee: Hyukjin Kwon
> Treat explicit None arguments of SQL functions in Spark Connect Rust as
> PySpark does
> ------------------------------------------------------------------------------------
>
> Key: SPARK-60968
> URL: https://issues.apache.org/jira/browse/SPARK-60968
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> The Python wrappers of the SQL functions that the Rust-backed Python client
> generates take an optional argument as given or not given, while PySpark
> treats an explicit {{None}} as each function does: as not given for most, or
> as a null literal, or as the default. So for 79 of the 204 optional
> parameters whose PySpark default is {{None}}, passing {{None}} explicitly
> differs from PySpark 4.3.0, for example:
> * {{F.bround(c, None)}}, {{F.ceil(c, None)}}, {{F.floor(c, None)}} and
> {{F.approx_count_distinct(c, None)}} leave the argument out in PySpark; the
> client sends a null literal, so {{bround(c, NULL)}} returns null.
> * {{F.first(c, None)}}, {{F.approx_percentile(c, p, None)}} and
> {{F.from_unixtime(c, None)}} send a null literal in PySpark; the client sends
> the default ({{false}}, {{10000}}, {{yyyy-MM-dd HH:mm:ss}}).
> * 37 others raise in one client and not in the other.
> Treat an explicit {{None}} as PySpark does for each function, for example by
> generating the wrappers with PySpark's handling of {{None}}, checked against
> its columns.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
