HyukjinKwon opened a new pull request, #294: URL: https://github.com/apache/spark-connect-rust/pull/294
### What changes were proposed in this pull request? [SPARK-61085](https://issues.apache.org/jira/browse/SPARK-61085): `spark.profile` now collects and shows the results of the UDF profilers, as PySpark Connect does. As PySpark Connect's: - `spark.profile` is a `pyspark.sql.profiler.Profile` of the session's `ConnectProfilerCollector`, also exposed as `spark._profiler_collector`. - The client updates the collector from the `__python_accumulator__` observed metrics of each execution. Each value is a pickled `(id, update)` of `specialAccumulatorSer`. The updates of the UDF profiler (`SQL_UDF_PROFIER_V2`, enabled by `spark.sql.pyspark.udf.profiler`) update the collector. A value that does not unpickle is logged as a warning and skipped. - So `spark.profile.show`, `dump`, `render` and `clear` show, write and clear the perf and memory profiles of the UDFs. `pyspark.sql.connect.profiler` is vendored from Apache Spark v4.3.0, and the drift check covers it. The core hands each pickled update of the Python accumulators to the handler of the new `SparkSession::set_python_accumulator_handler`. It no longer keeps the observed metrics of each execution as text in a `ProfilerCollector`; that type and `SparkSession::profiler()` are removed. ### Why are the changes needed? `spark.profile` returned a `ProfilerCollector` of the client's own, which never held a UDF profile: - it stored the observed metrics of each plan as text instead - its `show()` returned a str rather than printing the profiles - `dump(path)` and `show(type="perf")` raised `TypeError`s ### Does this PR introduce _any_ user-facing change? Yes. `spark.profile` collects and shows the UDF profiles of PySpark 4.3.0. ### How was this patch tested? - A new live test enables the perf UDF profiler, runs a UDF, shows its profile with `spark.profile`, and clears it. It passes with official PySpark 4.3.0 and fails without this change. - A new offline test sends the pickled updates of the UDF profiler, of another accumulator, and a value that does not unpickle, from a mock Connect server. It checks the profiles of the collector, the warning, `show` and `clear`. - A new lib test checks that the handler takes the updates of the Python accumulators, and not the metrics of observations. - On a local server, the profile of a UDF is shown, dumped and cleared as with official PySpark 4.3.0. - The lib tests of both crates (584 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (2040 passed, 12 skipped), and the live drop-in and Arrow suites (145 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
