[
https://issues.apache.org/jira/browse/SPARK-60993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60993:
-----------------------------------
Labels: pull-request-available (was: )
> Send the optional arguments of the SQL functions of the Python client as
> PySpark does
> -------------------------------------------------------------------------------------
>
> Key: SPARK-60993
> URL: https://issues.apache.org/jira/browse/SPARK-60993
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, the functions of {{pyspark.sql.functions}} send their
> optional arguments: {{to_date(col, format)}} sends the format, {{log(base,
> col)}} the base, {{sort_array(col, asc=False)}} the order. The Rust-backed
> Python client takes these arguments, as its wrappers have PySpark's
> signatures, but for about 100 functions its dispatch sends only the leading
> ones, or a default instead, so the results are silently wrong. Against a
> Spark 4.3.0 server:
> ||Call||PySpark 4.3.0||Rust-backed client||
> |{{to_date('02/01/2024', 'dd/MM/yyyy')}}|2024-01-02|fails with
> {{CAST_INVALID_INPUT}}|
> |{{log(2.0, 8.0)}}|3.0|0.693, the {{ln}} of 2.0|
> |{{sort_array(array(2, 1, 3), asc=False)}}|[3, 2, 1]|[1, 2, 3]|
> |{{split('a,b,c', ',', 2)}}|['a', 'b,c']|['a', 'b', 'c']|
> |{{substr('a,b,c', lit(1), lit(2))}}|'a,'|'a,b,c'|
> |{{like('a_c', lit('a!_c'), lit('!'))}}|true|false|
> |{{unix_timestamp(timestamp_seconds(1700000000))}}|1700000000|the current
> time|
> |{{months_between('2024-03-31', '2024-02-28',
> roundOff=False)}}|1.096774193548387|1.09677419|
> |{{first(v, ignorenulls=True)}} over NULL and 5|5|NULL|
> The others include the format of {{to_timestamp}} and the other parsing
> functions, the mode, padding, IV and AAD of {{aes_encrypt}} and
> {{aes_decrypt}}, the seed of {{rand}}, {{randn}}, {{shuffle}} and
> {{uniform}}, the delimiter of {{listagg}} and {{string_agg}}, the sketch
> sizes of the {{hll_}}, {{kll_}}, {{theta_}} and {{tuple_}} functions, the
> {{ignoreNulls}} of {{any_value}}, {{first_value}} and {{last_value}}, and the
> {{k}} of {{max_by}} and {{min_by}}. SPARK-60952 fixed four functions. Send
> the optional arguments as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
