HyukjinKwon opened a new pull request, #241: URL: https://github.com/apache/spark-connect-rust/pull/241
### What changes were proposed in this pull request? [SPARK-61043](https://issues.apache.org/jira/browse/SPARK-61043): `count` and `sum_distinct` now take their arguments, and `sumDistinct` warns, as PySpark Connect's do. - `functions.count(col)` and `functions.sum_distinct(col)` take a Column or a column name, and raise `NOT_EXPECTED_TYPE` for anything else, as PySpark's `_to_col` does. - `count` of a star Column, such as `count(col("*"))` or `count(df["*"])`, counts the rows as `count(1)`. `count("*")` is still the count of a star, as in PySpark. - `sumDistinct`, deprecated, warns `FutureWarning: Deprecated in 3.4, use sum_distinct instead.` ### Why are the changes needed? The client: - raised a `TypeError` for `count(1)` - took `sum_distinct(1)` as the sum of the distinct values of the literal `1` - sent `count(col("*"))` as the count of a star - did not warn of `sumDistinct` A sweep of all 560 functions of `pyspark.sql.functions` with seven kinds of arguments found these as the only differences in the errors and warnings at call time, apart from the exception types of misused `col`, `expr` and `call_function`. Those come from the implementation of PySpark, an `AssertionError` or `AttributeError`, not from its API. ### Does this PR introduce _any_ user-facing change? Yes. These functions take the arguments, and raise the errors and warnings, of PySpark 4.3.0. ### How was this patch tested? - A new offline test checks the columns of `count` and `sum_distinct`, their errors, and the warning of `sumDistinct`. - On a local server, 20 cases of these functions behave as in official PySpark 4.3.0: their columns, errors, warnings and results. - The lib tests of both crates (571 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (1810 passed, 8 skipped), and the live drop-in and Arrow suites (108 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
