[zeppelin] branch master updated: [ZEPPELIN-5446] SparkInterpreterLauncher#detectSparkScalaVersion may return wrong scala version

[zjffdu]

This is an automated email from the ASF dual-hosted git repository.

zjffdu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/zeppelin.git


The following commit(s) were added to refs/heads/master by this push:
     new 80228c3  [ZEPPELIN-5446] 
SparkInterpreterLauncher#detectSparkScalaVersion may return wrong scala version
80228c3 is described below

commit 80228c35905ebd16b7c80ce40f5b2d4f67180160
Author: Jeff Zhang <zjf...@apache.org>
AuthorDate: Sat Jul 10 15:16:20 2021 +0800

    [ZEPPELIN-5446] SparkInterpreterLauncher#detectSparkScalaVersion may return 
wrong scala version
    
    ### What is this PR for?
    
    This issue happens when user set `SPARK_HOME` in zeppelin spark interpreter 
setting, but zeppelin server host has `SPARK_CONF_DIR` which point to another 
version of spark.  In this case, even when user set `SPARK_CONF_DIR` 
explicitly, `SparkInterpreterLauncher#detectSparkScalaVersion` would still 
return the scala version of the spark version of `SPARK_CONF_DIR` point to.
    
    This PR would pass the envs built from interpreter setting to the process 
of `detectSparkScalaVersion`
    
    ### What type of PR is it?
    [Bug Fix ]
    
    ### Todos
    * [ ] - Task
    
    ### What is the Jira issue?
    * https://issues.apache.org/jira/browse/ZEPPELIN-5446
    
    ### How should this be tested?
    * Manually tested
    
    ### Screenshots (if appropriate)
    
    ### Questions:
    * Does the licenses files need update? No
    * Is there breaking changes for older versions? No
    * Does this needs documentation? No
    
    Author: Jeff Zhang <zjf...@apache.org>
    
    Closes #4161 from zjffdu/ZEPPELIN-5446 and squashes the following commits:
    
    ce453d8f0e [Jeff Zhang] rename envs to env
    95aa5a6393 [Jeff Zhang] [ZEPPELIN-5446] 
SparkInterpreterLauncher#detectSparkScalaVersion may return wrong scala version
---
 .../zeppelin/interpreter/launcher/SparkInterpreterLauncher.java    | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git 
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/launcher/SparkInterpreterLauncher.java
 
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/launcher/SparkInterpreterLauncher.java
index 440b249..d0a9aa8 100644
--- 
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/launcher/SparkInterpreterLauncher.java
+++ 
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/launcher/SparkInterpreterLauncher.java
@@ -133,7 +133,7 @@ public class SparkInterpreterLauncher extends 
StandardInterpreterLauncher {
           }
         }
 
-        String scalaVersion = detectSparkScalaVersion(getEnv("SPARK_HOME"));
+        String scalaVersion = detectSparkScalaVersion(getEnv("SPARK_HOME"), 
env);
         Path scalaFolder =  Paths.get(zConf.getZeppelinHome(), 
"/interpreter/spark/scala-" + scalaVersion);
         if (!scalaFolder.toFile().exists()) {
           throw new IOException("spark scala folder " + scalaFolder.toFile() + 
" doesn't exist");
@@ -241,11 +241,12 @@ public class SparkInterpreterLauncher extends 
StandardInterpreterLauncher {
     }
     LOGGER.info("buildEnvFromProperties: {}", env);
     return env;
-
   }
 
-  private String detectSparkScalaVersion(String sparkHome) throws Exception {
+  private String detectSparkScalaVersion(String sparkHome, Map<String, String> 
env) throws Exception {
+    LOGGER.info("Detect scala version from SPARK_HOME: {}", sparkHome);
     ProcessBuilder builder = new ProcessBuilder(sparkHome + 
"/bin/spark-submit", "--version");
+    builder.environment().putAll(env);
     File processOutputFile = File.createTempFile("zeppelin-spark", ".out");
     builder.redirectError(processOutputFile);
     Process process = builder.start();