HyukjinKwon opened a new pull request, #234: URL: https://github.com/apache/spark-connect-rust/pull/234
### What changes were proposed in this pull request? [SPARK-61030](https://issues.apache.org/jira/browse/SPARK-61030): the bool of a Column, and the arguments of `Column` methods, now behave as PySpark Connect's. - The bool of a Column, as in `if col:` or `col and other`, raises `CANNOT_CONVERT_COLUMN_INTO_BOOL` of the repr of its expression. - `like`, `rlike` and `ilike` take any value, as PySpark's `_bin_op` does: `like(a, 1)`. - `getField` takes any name, as `col[name]` does, with a `FutureWarning` for a Column. - `cast` raises `NOT_EXPECTED_TYPE` for a value that is neither a DataType nor a str. - `withField` raises it for a field name that is not a str and a value that is not a Column, and `dropFields` for a field name that is not a str. `dropFields` raises `CANNOT_BE_EMPTY` without field names. - `over` raises it for a window that is not a `WindowSpec`. ### Why are the changes needed? - The bool of a Column raised `AssertionError: Undefined error message parameter for error class: CANNOT_CONVERT_COLUMN_INTO_BOOL`, as it did not give the column. - `like`, `rlike`, `ilike` and `getField` raised `TypeError`s for values that are not strings. - `withField("x", 1)` was accepted. - `cast`, `withField`, `dropFields` and `over` raised `TypeError`s rather than PySpark's errors. ### Does this PR introduce _any_ user-facing change? Yes. These raise the errors, and take the arguments, of PySpark 4.3.0. ### How was this patch tested? - A new offline test checks the error of the bool of a Column, the arguments of `like`, `rlike`, `ilike` and `getField`, and the errors of `cast`, `withField`, `dropFields` and `over`. - The lib tests of both crates (569 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (1800 passed, 8 skipped), and the live drop-in and Arrow suites (106 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
