Hyukjin Kwon created SPARK-60968:
------------------------------------
Summary: Treat explicit None arguments of SQL functions in Spark
Connect Rust as PySpark does
Key: SPARK-60968
URL: https://issues.apache.org/jira/browse/SPARK-60968
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
The Python wrappers of the SQL functions that the Rust-backed Python client
generates take an optional argument as given or not given, while PySpark treats
an explicit {{None}} as each function does: as not given for most, or as a null
literal, or as the default. So for 79 of the 204 optional parameters whose
PySpark default is {{None}}, passing {{None}} explicitly differs from PySpark
4.3.0, for example:
* {{F.bround(c, None)}}, {{F.ceil(c, None)}}, {{F.floor(c, None)}} and
{{F.approx_count_distinct(c, None)}} leave the argument out in PySpark; the
client sends a null literal, so {{bround(c, NULL)}} returns null.
* {{F.first(c, None)}}, {{F.approx_percentile(c, p, None)}} and
{{F.from_unixtime(c, None)}} send a null literal in PySpark; the client sends
the default ({{false}}, {{10000}}, {{yyyy-MM-dd HH:mm:ss}}).
* 37 others raise in one client and not in the other.
Treat an explicit {{None}} as PySpark does for each function, for example by
generating the wrappers with PySpark's handling of {{None}}, checked against
its columns.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
