(spark) branch branch-3.5 updated: [SPARK-46330] Loading of Spark UI blocks for a long time when HybridStore enabled

[yao]

This is an automated email from the ASF dual-hosted git repository.

yao pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.5 by this push:
     new 0c00c54583fe [SPARK-46330] Loading of Spark UI blocks for a long time 
when HybridStore enabled
0c00c54583fe is described below

commit 0c00c54583fe3e56f940425aac6e0e4f05c4b9db
Author: zhouyifan279 <zhouyifan...@gmail.com>
AuthorDate: Wed Dec 20 16:50:38 2023 +0800

    [SPARK-46330] Loading of Spark UI blocks for a long time when HybridStore 
enabled
    
    ### What changes were proposed in this pull request?
    Move `LoadedAppUI` invalidate operation out of `FsHistoryProvider` 
synchronized block.
    
    ### Why are the changes needed?
    When closing a HybridStore of a `LoadedAppUI` with a lot of data waiting to 
be written to disk, loading of other Spark UIs will be blocked for a long time.
    
    See more details at https://issues.apache.org/jira/browse/SPARK-46330
    
    ### Does this PR introduce _any_ user-facing change?
    No.
    
    ### How was this patch tested?
    Passed existing tests.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    No.
    
    Closes #44260 from zhouyifan279/SPARK-46330.
    
    Authored-by: zhouyifan279 <zhouyifan...@gmail.com>
    Signed-off-by: Kent Yao <y...@apache.org>
    (cherry picked from commit cf54e8f9a51bf54e8fa3e1011ac370e46134b134)
    Signed-off-by: Kent Yao <y...@apache.org>
---
 .../org/apache/spark/deploy/history/FsHistoryProvider.scala   | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git 
a/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala 
b/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
index 49b479f3124e..387bc7d9e45b 100644
--- 
a/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
+++ 
b/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
@@ -925,11 +925,12 @@ private[history] class FsHistoryProvider(conf: SparkConf, 
clock: Clock)
    * UI lifecycle.
    */
   private def invalidateUI(appId: String, attemptId: Option[String]): Unit = {
-    synchronized {
-      activeUIs.get((appId, attemptId)).foreach { ui =>
-        ui.invalidate()
-        ui.ui.store.close()
-      }
+    val uiOption = synchronized {
+      activeUIs.get((appId, attemptId))
+    }
+    uiOption.foreach { ui =>
+      ui.invalidate()
+      ui.ui.store.close()
     }
   }
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org