HyukjinKwon opened a new pull request, #219: URL: https://github.com/apache/spark-connect-rust/pull/219
### What changes were proposed in this pull request? [SPARK-61015](https://issues.apache.org/jira/browse/SPARK-61015): `unpivot`, `melt` and `replace` now take their arguments as PySpark does. - `unpivot` and `melt`, which is the same function, take `ids` and `values` that are a column name, a Column, or a list or tuple of them. All four arguments are required. - `replace` (and `na.replace`) takes a `subset` that is a column name, or a list or tuple of them. It sends each value as a literal of its type, as PySpark Connect builds the `NAReplace` relation: a bool, a double (an int as well), a string, or `None` as a null. - `replace` raises `ARGUMENT_REQUIRED`, `NOT_EXPECTED_TYPE`, `LENGTH_SHOULD_BE_THE_SAME` and `MIXED_TYPE_REPLACEMENT` where PySpark does, and warns that the `value` of a dict `to_replace` is ignored. In the Rust API, `DataFrame::replace_literals` and `NaFunctions::replace_literals` take the replacements as literals. `DataFrame::replace` of strings is unchanged. ### Why are the changes needed? They took lists only: `df.unpivot("id", ["v"], "var", "val")` and `df.replace("a", "z", "k")` raised a `TypeError`. The values of `replace` were sent as strings, and one that parses as a number as a double. So: - `df.replace("1", "one")` and `df.replace({1: "x"})` failed on the server with `CAST_INVALID_INPUT`. - `df.replace(True, False)` replaced nothing. - `df.replace({"a": None})` replaced `"a"` with the string `'None'`. - `df.replace("a", None, ["k"])` replaced nothing. ### Does this PR introduce _any_ user-facing change? Yes. `unpivot`, `melt` and `replace` take the arguments of PySpark 4.3.0, replace as it does, and raise its errors. ### How was this patch tested? - A new live test replaces strings, numbers, bools and nulls over a subset of a name, a list or a tuple, and unpivots and melts by names and Columns. It passes with official PySpark 4.3.0, and fails without this change. - A new offline test checks the errors of the arguments, and the warning. It passes with official PySpark 4.3.0, and fails without this change. - New lib tests check the literals of the replacements. - A probe of 47 calls matches official PySpark 4.3.0. The exceptions: two messages of `TypeError` that name the function differently. - The lib tests and every integration test, e2e included, against a local Spark 4.3.0 server (673 passed, and 129 of the transport crate), the build of all tests with `wasm-udf`, the offline Python suites (1625 passed, 8 skipped), and the live drop-in and Arrow suites (95 passed) and live cases (10 passed). Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
