Shreyas created SPARK-53041:
-------------------------------
Summary: Inconsistent behavior between SQL SET and
SparkContext.setJobDescription for job descriptions in Web UI
Key: SPARK-53041
URL: https://issues.apache.org/jira/browse/SPARK-53041
Project: Spark
Issue Type: Improvement
Components: SQL, Web UI
Affects Versions: 3.4.4
Reporter: Shreyas
There is inconsistent behavior between setting job descriptions via SQL {{SET}}
commands versus {{{}SparkContext.setJobDescription(){}}}, causing confusion in
the Web UI display.
*Current Behavior:*
*
{{SET spark.job.description = 'value'}} sets a global configuration property
that appears in Jobs and Stages tabs but NOT in the SQL/DataFrame tab
*
{{sc.setJobDescription('value')}} sets a thread-local property that appears in
Jobs, Stages, AND SQL/DataFrame tabs
*Expected Behavior:*
Both methods should have consistent behavior and appear in the same UI
locations.
*Steps to Reproduce:*
{{// Method 1: SQL SET command}}
{{spark.sql("SET spark.job.description = 'SQL Set Method'")
spark.sql("SELECT * FROM table").collect()}}
{{// Result: Appears in Jobs/Stages tabs only}}
{{}}
{{// Method 2: SparkContext method
spark.sparkContext.setJobDescription("SparkContext Method")
spark.sql("SELECT * FROM table").collect()}}
{{// Result: Appears in Jobs/Stages AND SQL/DataFrame tabs}}
CopyInsert at cursorscala
*Root Cause:*
* SQL {{SET}} calls
{{sparkSession.conf.set()}} (global config)
*
{{setJobDescription()}} calls
{{setLocalProperty()}} (thread-local property)
* Web UI SQL/DataFrame tab only reads thread-local properties
* Jobs/Stages tabs read both global config and local properties
*Proposed Solutions:*
# {*}Bridge approach{*}: Modify {{SetCommand.scala}} to call
{{setLocalProperty()}} when setting {{spark.job.description}}
# {*}Unification approach{*}: Make Web UI consistently read from both property
sources
# {*}Documentation approach{*}: Clearly document the difference and deprecate
SQL SET for job metadata
*Files Involved:*
{{sql/core/src/main/scala/org/apache/spark/sql/execution/command/SetCommand.scala}}
{{core/src/main/scala/org/apache/spark/SparkContext.scala}}
*Impact:*
* User confusion about inconsistent UI behavior
* Affects job monitoring and debugging workflows
* No breaking changes expected with proposed fixes
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
