HyukjinKwon opened a new pull request, #222: URL: https://github.com/apache/spark-connect-rust/pull/222
### What changes were proposed in this pull request? [SPARK-61019](https://issues.apache.org/jira/browse/SPARK-61019): `explain` and `StreamingQuery.explain` now print with Python's `print`, and `explain` raises the errors of PySpark. - `df.explain(extended, mode)` prints `df._explain_string(extended, mode)` with Python's `print`. So the plan goes to `sys.stdout`, which a notebook or `contextlib.redirect_stdout` captures. - `StreamingQuery.explain(extended)` prints its plan and returns None. - `_explain_string` raises `CANNOT_SET_TOGETHER` for both `extended` and `mode`, and `NOT_EXPECTED_TYPE` for an argument that is neither a bool nor a string. It raises `VALUE_NOT_ALLOWED` for a mode other than `"simple"`, `"extended"`, `"codegen"`, `"cost"` and `"formatted"`, as PySpark Connect's client checks it. An error of a PySpark error class that the client raises is now raised with its error class and message parameters, so that `getCondition()` and `getMessageParameters()` return them. Examples are the `VALUE_NOT_ALLOWED` of `explain`, and the `CANNOT_BE_EMPTY` of `spark.tvf.json_tuple(col)` without fields. An error of a class that PySpark does not know is raised with its message, as before. In the Rust API, `DataFrame::explain_string` returns the plan that `explain_mode` prints, and the mode is checked as PySpark checks it. ### Why are the changes needed? - `explain` wrote the plan to the standard output of the process from Rust, which neither a notebook nor `redirect_stdout` captured. - `StreamingQuery.explain` returned the plan as a string. - `_explain_string` did not exist. - `explain(mode="FORMATTED")` was accepted. - `explain(mode="bogus")`, `explain(True, "simple")` and `explain(1)` raised a `PySparkValueError` without an error class, a `ValueError` and a `TypeError`. ### Does this PR introduce _any_ user-facing change? Yes. `explain` and `StreamingQuery.explain` print as PySpark 4.3.0 does, and raise its errors. The errors of PySpark's error classes that the client raises carry their classes and parameters. ### How was this patch tested? - A new live test captures the plans that `explain` and `StreamingQuery.explain` print with `redirect_stdout`. It passes with official PySpark 4.3.0, and fails without this change. - A new offline test checks the errors of the arguments and of `json_tuple`, and the plan that `explain` prints of the explain string of a mock Connect server and the mode it sends. The mock now answers `Explain` requests. The test fails without this change. - A new lib test checks the explain modes of `explain_string`. - A probe of `explain`, `_explain_string`, `printSchema`, `show` and streaming `explain` matches official PySpark 4.3.0. - The lib tests and every integration test, e2e included, against a local Spark 4.3.0 server (672 passed, and 129 of the transport crate), the build of all tests with `wasm-udf`, the offline Python suites (1786 passed, 8 skipped), and the live drop-in and Arrow suites (95 passed) and live cases (10 passed). Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
