gaogaotiantian opened a new pull request, #59852: URL: https://github.com/apache/spark/pull/59852
### What changes were proposed in this pull request? This PR adds a sampling profiler for Python UDFs, enabled with `spark.sql.pyspark.udf.profiler=sampling`. - `WorkerSamplingProfiler` (`pyspark/sql/profiler.py`): while the UDF runs, a background thread wakes up every `samplingInterval` ms, calls `sys._current_frames()`, and records each thread's call stack into a per-thread call tree with a sample count on each node. Nodes are keyed by `co_qualname (filename:firstlineno)`. - When the task finishes, the result is sent back through the existing V2 profiler accumulator (`SQL_UDF_PROFIER_V2`) under a `"sampling"` key, using a per-task key (`stage_partition.attempt_taskAttemptId`). `SamplingResultParam` merges these call trees by recursively summing the counts. - `ProfilerCollector._sampling_profile_results` gives access to the collected results. - A new SQL conf, `spark.sql.pyspark.udf.profiler.samplingInterval` (time conf in ms, default 100ms), is passed to the Python worker by `PythonUDFRunner` and `ArrowPythonRunner`. - `spark.sql.pyspark.udf.profiler` now accepts `sampling` in addition to `perf` and `memory`. ### Why are the changes needed? The existing `perf` (cProfile) and `memory` profilers are deterministic. They add overhead to every function call or line, which can distort results and slow down heavy UDFs. A sampling profiler adds little, roughly constant overhead, and gives a call-tree view (usable for flame graphs) of where time is spent, including in threads the UDF starts. ### Does this PR introduce _any_ user-facing change? Yes. There is a new value `sampling` for `spark.sql.pyspark.udf.profiler` and a new conf, `spark.sql.pyspark.udf.profiler.samplingInterval`. This is a work-in-progress draft: user-facing APIs to show or dump sampling results (like `spark.profile.show`/`dump`) are not added yet. ### How was this patch tested? Added `test_sampling_profiler_udf` in `pyspark.sql.tests.test_udf_profiler`. It checks that no results are collected when the profiler is disabled, and that with it enabled there is one result per task and the UDF frame appears in the sampled call tree. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Isaac (PR description) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
