pan3793 opened a new pull request, #58313: URL: https://github.com/apache/spark/pull/58313
### What changes were proposed in this pull request? Add a `none` option to `spark.sql.ui.explainMode`. When it is set, `SparkListenerSQLExecutionStart` and `SparkListenerSQLAdaptiveExecutionUpdate` carry a placeholder string instead of the rendered plan description: ``` No plan description because spark.sql.ui.explainMode=none ``` This skips the explain string generation for the SQL UI entirely. The change touches: - `SQLConf.UI_EXPLAIN_MODE`: accepts `none` in addition to `simple`, `extended`, `codegen`, `cost`, `formatted` (doc and validation updated) - `SQLExecution`: skips `QueryExecution.explainString` when the mode is `none` - `AdaptiveSparkPlanExec`: same for the plan description posted with adaptive execution updates ### Why are the changes needed? Rendering the plan description is done solely for the SQL UI. For workloads with large plans, or with adaptive execution enabled (where the plan description is re-rendered on every adaptive update), this generation is non-trivial overhead that users who do not use the SQL UI cannot avoid today, since every valid mode renders something. `none` lets them opt out of the cost. For example, we have a customer job that constructs a huge plan whose `treeString` exceeds 280,000 lines. Rendering the plan description takes more than 3 minutes per iteration, and with AQE enabled, assembling the plan tree strings takes more than 40 minutes across the query execution. A string this large also pressures driver memory (we have observed driver OOM), and the browser becomes unresponsive when the SQL UI tries to render it. ### Does this PR introduce _any_ user-facing change? Yes. `spark.sql.ui.explainMode` now accepts `none`. When set, the SQL UI shows the placeholder text above instead of the query plan description for SQL executions. The default value (`formatted`) is unchanged. ### How was this patch tested? The existing `control a plan explain mode in listeners via SQLConf` tests are extended with a `none` case, asserting the placeholder is carried by the events: - `AdaptiveQueryExecSuite` (covers `SparkListenerSQLAdaptiveExecutionUpdate`) - `SQLAppStatusListenerSuite` (covers `SparkListenerSQLExecutionStart`) ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Qwen3.8 Max -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
