HyukjinKwon opened a new pull request, #306: URL: https://github.com/apache/spark-connect-rust/pull/306
### What changes were proposed in this pull request? [SPARK-61110](https://issues.apache.org/jira/browse/SPARK-61110): `DataFrame._repr_html_` is added, as PySpark Connect's. Jupyter calls `_repr_html_` to display a DataFrame. As in PySpark Connect: - With `spark.sql.repl.eagerEval.enabled`, it returns the HTML table of the first `maxNumRows` rows, with values truncated to `truncate` characters. The server formats the table, of an `HtmlString` relation. - Otherwise it returns None. - Once it has been called, `__repr__` of the DataFrame is its `DataFrame[...]` string (PySpark's `_support_repr_html`). So a notebook does not show the table twice. The core gains the `HtmlString` plan and `DataFrame::html_string`, as it has `ShowString` and `show_string`. ### Why are the changes needed? The client lacked `_repr_html_`. Attribute access took it as a column, so `df._repr_html_()` raised `TypeError: 'Column' object is not callable`. A notebook with eager evaluation showed the text table of `__repr__` instead of the HTML table. ### Does this PR introduce _any_ user-facing change? Yes. Notebooks show DataFrames as with PySpark 4.3.0. ### How was this patch tested? - A new live test checks `_repr_html_` without and with eager evaluation, and `__repr__` after it. It passes with official PySpark 4.3.0 and fails without this change. - A new offline test checks, against a mock Connect server: - the `HtmlString` relation sent, of `maxNumRows` and `truncate` - the table returned - `__repr__` before and after `_repr_html_` - On a local server, 41 cases give the same results as official PySpark 4.3.0. They cover `show`, `printSchema`, `explain`, `__repr__`, `_repr_html_` and `describe`/`summary`, with eager evaluation, truncation and maximum rows. The only difference is the expression ids in `explain`. - The lib tests of both crates (589 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (2056 passed, 12 skipped), and the live drop-in and Arrow suites (160 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
