[
https://issues.apache.org/jira/browse/SPARK-61043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61043.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 241
[https://github.com/apache/spark-connect-rust/pull/241]
> Take the arguments of count and sum_distinct, and warn of sumDistinct, in the
> Python client as PySpark does
> -----------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61043
> URL: https://issues.apache.org/jira/browse/SPARK-61043
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, {{functions.count(col)}} and
> {{functions.sum_distinct(col)}} take a Column or a column name, and raise
> {{NOT_EXPECTED_TYPE}} for anything else, as their {{_to_col}} does. {{count}}
> of a star Column, as {{count(col("*"))}} or {{count(df["*"])}}, counts the
> rows as {{count(1)}}. {{sumDistinct}}, deprecated, warns {{FutureWarning:
> Deprecated in 3.4, use sum_distinct instead.}}
> The Rust-backed Python client raises a {{TypeError}} for {{count(1)}}, takes
> {{sum_distinct(1)}} as the sum of the distinct values of the literal {{1}},
> sends {{count(col("*"))}} as the count of a star, and does not warn of
> {{sumDistinct}}. Take them as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
