[spark] branch master updated: [SPARK-36193][CORE] Recover SparkSubmit.runMain not to stop SparkContext in non-K8s env

[dongjoon]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new fd3e9ce  [SPARK-36193][CORE] Recover SparkSubmit.runMain not to stop 
SparkContext in non-K8s env
fd3e9ce is described below

commit fd3e9ce0b9ee09c7dce9f2e029fe96eac51eab96
Author: Dongjoon Hyun <dongj...@apache.org>
AuthorDate: Sun Jul 18 22:26:23 2021 -0700

    [SPARK-36193][CORE] Recover SparkSubmit.runMain not to stop SparkContext in 
non-K8s env
    
    ### What changes were proposed in this pull request?
    
    According to the discussion on https://github.com/apache/spark/pull/32283 , 
this PR aims to limit the feature of SPARK-34674 to K8s environment only.
    
    ### Why are the changes needed?
    
    To reduce the behavior change in non-K8s environment.
    
    ### Does this PR introduce _any_ user-facing change?
    
    The change behavior is consistent with 3.1.1 and older Spark releases.
    
    ### How was this patch tested?
    
    N/A
    
    Closes #33403 from dongjoon-hyun/SPARK-36193.
    
    Authored-by: Dongjoon Hyun <dongj...@apache.org>
    Signed-off-by: Dongjoon Hyun <dongj...@apache.org>
---
 core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala 
b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
index a65be54..8124650 100644
--- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
@@ -957,8 +957,8 @@ private[spark] class SparkSubmit extends Logging {
       case t: Throwable =>
         throw findCause(t)
     } finally {
-      if (!isShell(args.primaryResource) && !isSqlShell(args.mainClass) &&
-        !isThriftServer(args.mainClass)) {
+      if (args.master.startsWith("k8s") && !isShell(args.primaryResource) &&
+          !isSqlShell(args.mainClass) && !isThriftServer(args.mainClass)) {
         try {
           SparkContext.getActive.foreach(_.stop())
         } catch {

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org