Navdeep Poonia created SPARK-29340: -------------------------------------- Summary: Spark Sql executions do not use thread local jobgroup Key: SPARK-29340 URL: https://issues.apache.org/jira/browse/SPARK-29340 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 2.4.4 Reporter: Navdeep Poonia
val sparkThreadLocal: SparkSession = DataCurator.spark.newSession() sparkThreadLocal.sparkContext.setJobGroup("<id>", "<description>") OR sparkThreadLocal.sparkContext.setLocalProperty("spark.job.description", "<id>") sparkThreadLocal.sparkContext.setLocalProperty("spark.jobGroup.id", "<description>") The jobgroup property works fine for spark jobs/stages created by spark dataframe operations but in case of sparksql the jobgroup is randomly assigned to stages or is null sometimes. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org