HyukjinKwon opened a new pull request, #297: URL: https://github.com/apache/spark-connect-rust/pull/297
### What changes were proposed in this pull request? [SPARK-61093](https://issues.apache.org/jira/browse/SPARK-61093): `Column.try_cast` and `Column.isin` now take their arguments as PySpark Connect's do. As PySpark Connect's: - `try_cast` raises `NOT_EXPECTED_TYPE` (`expected_type` "DataType or str") for a type that is neither a `DataType` nor a str, as `cast` already does. - `isin` unpacks a single list or set argument, but not a tuple. A tuple is a value, so `col.isin((1, 2))` raises the `UNSUPPORTED_DATA_TYPE` of tuple, as `lit((1, 2))` does. ### Why are the changes needed? The client raised a plain `TypeError` for `try_cast(1)`, and took `col.isin((1, 2))` as `col.isin(1, 2)`. ### Does this PR introduce _any_ user-facing change? Yes. `try_cast` and `isin` raise the errors of PySpark 4.3.0. ### How was this patch tested? - A new live test checks the errors of `try_cast(1)` and of `isin` of a tuple, and the rows of `isin` of a list. It passes with official PySpark 4.3.0 and fails without this change. - A new offline test checks these errors, and that `isin` of a list, a set and of separate values is the same column. - On a local server, 52 cases of Column methods with bad and good arguments give the same results as official PySpark 4.3.0. The only exception is the CPython message of a `TypeError` for the arguments of `desc`. - The lib tests of both crates (594 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (2045 passed, 12 skipped), and the live drop-in and Arrow suites (148 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
