HyukjinKwon opened a new pull request, #194: URL: https://github.com/apache/spark-connect-rust/pull/194
### What changes were proposed in this pull request? [SPARK-60991](https://issues.apache.org/jira/browse/SPARK-60991): `DataFrame.show(n=20, truncate=True, vertical=False)` now sends a `ShowString` relation with the number of rows, the truncation and the vertical flag, as PySpark does. It prints the table that the server formats with Python's `print`. `DataFrame._show_string` returns that table. - Both check their arguments as PySpark's `_show_string` does. They raise `NOT_EXPECTED_TYPE` for an `n` that is not an int (a bool included), a `vertical` that is not a bool, and a `truncate` that `int()` cannot convert. An explicit None is checked as any other value. Too many, unknown or repeated arguments raise Python's `TypeError`s. - The Rust `DataFrame::show` prints the same table, and the new `DataFrame::show_string` returns it, through a new `LogicalPlan::ShowString`. ### Why are the changes needed? `df.show()` collected the first rows and printed each with Rust's `println!`, ignoring `truncate` and `vertical`. For example, `spark.range(2).show()` printed `[0]` and `[1]` instead of the table: ``` +---+ | id| +---+ | 0| | 1| +---+ ``` As it wrote past Python's `sys.stdout`, a notebook or `contextlib.redirect_stdout` did not get the output. ### Does this PR introduce _any_ user-facing change? Yes, `show` prints the table of PySpark 4.3.0. ### How was this patch tested? - Against a local Spark 4.3.0 server, a probe of 18 calls of `show` and `_show_string` gives the same output as official PySpark 4.3.0. It covers the default, `n`, a `truncate` of False, an int, a string and a float, `vertical`, the errors of each argument (None included), and too many, unknown and repeated arguments. - A new live test checks the tables of `show` with each argument, through `contextlib.redirect_stdout`. It also passes with official PySpark 4.3.0, and fails without this change. - A new offline test checks the errors of the arguments, which are raised before any request, and the signature. - The lib tests and every integration test, e2e included, against the local server (647 passed), the build of all tests with `wasm-udf`, the offline Python suites (719 passed, 9 skipped), and the live drop-in and Arrow suites (73 passed) and live cases (10 passed). Clippy reports only one diagnostic in the changed lines: the `result_large_err` of the `py.detach` closure, which every action of the Python `DataFrame` has. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
