HyukjinKwon opened a new pull request, #193: URL: https://github.com/apache/spark-connect-rust/pull/193
### What changes were proposed in this pull request? [SPARK-60960](https://issues.apache.org/jira/browse/SPARK-60960) has two parts. **`executionInfo` per DataFrame.** `DataFrame.executionInfo` is now the information of the last execution of that DataFrame, and `None` before one, as PySpark's `DataFrame._execution_info` is. Before, it was the session's last execution, of any DataFrame. - These set it, as in PySpark: `collect`, the Arrow and pandas collects, `write`, `writeTo` and `mergeInto`, the temp views, and the checkpoints. - An action that runs a derived DataFrame, such as `count`, `show`, `first` or `take`, sets that DataFrame's, as in PySpark, not the original's. - Exact clones of a DataFrame share the information, and a DataFrame of another plan has its own. - The session still records its last execution, for `profile()`. **Operation ids on errors.** The catalog operations and `writeStream.start()` now run on the reattachable path, as the other `ExecutePlan` requests do. So their errors carry the operation id of the request, as PySpark's do. `start()` now also sends the session's tags. ### Why are the changes needed? - For `a = spark.range(3); a.collect(); b = spark.range(5)`, `b.executionInfo` gave the information of `a`'s execution, where PySpark gives `None`. Any DataFrame showed the metrics and operation id of the last execution in the session, also across threads. - `spark.catalog.setCurrentDatabase("nope")` raised an `AnalysisException` whose `operation_id` was `None`, and so did a failing `writeStream.start()`. ### Does this PR introduce _any_ user-facing change? Yes, `executionInfo` and these errors behave as in PySpark 4.3.0. ### How was this patch tested? - Against a local Spark 4.3.0 server, a probe of 13 cases gives the same results as official PySpark 4.3.0: `executionInfo` before and after `collect`, on other and derived DataFrames, after `count`, `show`, `first` and `take`, temp views, `toArrow`, `toPandas`, a write and distinct operation ids, and the two errors. Before this change, six of them differed. - New live tests check those cases and errors. They also pass with official PySpark 4.3.0, and fail without this change. - The offline tests of `executionInfo` read it from the DataFrame that ran, and check that another one has none. - The lib tests and every integration test, e2e included, against the local server (647 passed), the build of all tests with `wasm-udf`, the offline Python suites (687 passed, 9 skipped), and the live drop-in and Arrow suites (72 passed) and live cases (10 passed). Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
