[incubator-kyuubi] branch branch-1.6 updated: [KYUUBI #3600] [SPARK][K8S] Respect default static port for web UI on Spark K8s cluster mode

[chengpan]

This is an automated email from the ASF dual-hosted git repository.

chengpan pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/incubator-kyuubi.git


The following commit(s) were added to refs/heads/branch-1.6 by this push:
     new 8b41223a1 [KYUUBI #3600] [SPARK][K8S] Respect default static port for 
web UI on Spark K8s cluster mode
8b41223a1 is described below

commit 8b41223a123a3f3afb020f4654971b649f91bd51
Author: zwangsheng <2213335...@qq.com>
AuthorDate: Mon Oct 10 19:59:08 2022 +0800

    [KYUUBI #3600] [SPARK][K8S] Respect default static port for web UI on Spark 
K8s cluster mode
    
    ### _Why are the changes needed?_
    
    Now kyuubi will help set web ui port to 0 while scalaing up spark sql 
engine, which is good for the general situation, help avoid port conflicts.
    
    But on k8s case, kyuubi should respect the user's default settings, here 
means web ui should be `4040`.
    
    ### _How was this patch tested?_
    - [ ] Add some test cases that check the changes thoroughly including 
negative and positive cases if possible
    
    - [ ] Add screenshots for manual tests if appropriate
    
    - [ ] [Run 
test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests)
 locally before make a pull request
    
    Closes #3600 from zwangsheng/improve/fix_spark_sql_engine_web_ui_on_k8s.
    
    Closes #3600
    
    0174e22f [zwangsheng] comment
    ac2dbedc [zwangsheng] fix
    1a2bff70 [zwangsheng] fix
    bd4c35c2 [zwangsheng] fix
    
    Authored-by: zwangsheng <2213335...@qq.com>
    Signed-off-by: Cheng Pan <cheng...@apache.org>
    (cherry picked from commit e80a7db146095f383e2fce9b6fc16dfcb504323c)
    Signed-off-by: Cheng Pan <cheng...@apache.org>
---
 .../scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git 
a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
 
b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
index ea44408f3..828af1adb 100644
--- 
a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
+++ 
b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
@@ -147,7 +147,6 @@ object SparkSQLEngine extends Logging {
     _sparkConf.setIfMissing("spark.sql.execution.topKSortFallbackThreshold", 
"10000")
     
_sparkConf.setIfMissing("spark.sql.legacy.castComplexTypesToString.enabled", 
"true")
     _sparkConf.setIfMissing("spark.master", "local")
-    _sparkConf.setIfMissing("spark.ui.port", "0")
     _sparkConf.set(
       "spark.redaction.regex",
       _sparkConf.get("spark.redaction.regex", 
"(?i)secret|password|token|access[.]key")
@@ -171,6 +170,11 @@ object SparkSQLEngine extends Logging {
       kyuubiConf.setIfMissing(FRONTEND_CONNECTION_URL_USE_HOSTNAME, false)
     }
 
+    // Set web ui port 0 to avoid port conflicts during non-k8s cluster mode
+    if (!isOnK8sClusterMode) {
+      _sparkConf.setIfMissing("spark.ui.port", "0")
+    }
+
     // Pass kyuubi config from spark with `spark.kyuubi`
     val sparkToKyuubiPrefix = "spark.kyuubi."
     _sparkConf.getAllWithPrefix(sparkToKyuubiPrefix).foreach { case (k, v) =>
@@ -309,4 +313,9 @@ object SparkSQLEngine extends Logging {
       },
       "CreateSparkTimeoutChecker").start()
   }
+
+  private def isOnK8sClusterMode: Boolean = {
+    // only spark driver pod will build with `SPARK_APPLICATION_ID` env.
+    Utils.isOnK8s && sys.env.contains("SPARK_APPLICATION_ID")
+  }
 }