[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #35856: [SPARK-38550][SQL][CORE] Use a disk-based store to save more debug information for live UI

[GitBox]

dongjoon-hyun commented on code in PR #35856:
URL: https://github.com/apache/spark/pull/35856#discussion_r841422549


##
sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala:
##
@@ -118,6 +119,12 @@ private[sql] class SharedState(
 statusStore
   }
 
+  sparkContext.statusStore.diskStore.foreach { kvStore =>
+sparkContext.listenerBus.addToQueue(
+  new DiagnosticListener(conf, kvStore.asInstanceOf[ElementTrackingStore]),

Review Comment:
   Why do we need to share the same kvStore?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #35856: [SPARK-38550][SQL][CORE] Use a disk-based store to save more debug information for live UI

[GitBox]

dongjoon-hyun commented on code in PR #35856:
URL: https://github.com/apache/spark/pull/35856#discussion_r841421049


##
core/src/main/scala/org/apache/spark/internal/config/Status.scala:
##
@@ -70,4 +70,11 @@ private[spark] object Status {
   .version("3.0.0")
   .booleanConf
   .createWithDefault(false)
+
+  val DISK_STORE_DIR_FOR_STATUS =
+ConfigBuilder("spark.appStatusStore.diskStore.dir")

Review Comment:
   If there is no other config, Apache Spark community's configuration naming 
guide is not to introduce a namespace by removing `.`. In this case,
   ```
   - spark.appStatusStore.diskStore.dir
   + spark.appStatusStore.diskStoreDir
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org