[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-12-04 Thread vanzin
Github user vanzin closed the pull request at: https://github.com/apache/spark/pull/19631 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-12-01 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r154403001 --- Diff: project/MimaExcludes.scala --- @@ -36,6 +36,12 @@ object MimaExcludes { // Exclude rules for 2.3.x lazy val v23excludes = v22exc

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-12-01 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r154378257 --- Diff: project/MimaExcludes.scala --- @@ -36,6 +36,12 @@ object MimaExcludes { // Exclude rules for 2.3.x lazy val v23excludes = v22

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-12-01 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r154373635 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -542,7 +496,54 @@ private[spark] class SecurityManager( * Gets the secr

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-30 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r154240414 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -542,7 +496,54 @@ private[spark] class SecurityManager( * Gets the secret

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-30 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r154218810 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -542,7 +496,54 @@ private[spark] class SecurityManager( * Gets the secret

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-30 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r154218236 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -58,18 +58,14 @@ import org.apache.spark.util.{CallerContex

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-28 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r153510604 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -542,7 +496,54 @@ private[spark] class SecurityManager( * Gets the secret

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-20 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r152106333 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -824,6 +821,11 @@ private[spark] class Client(

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-20 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r152117915 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -542,7 +496,54 @@ private[spark] class SecurityManager( * Gets the secr

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-20 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r152084447 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -58,18 +58,14 @@ import org.apache.spark.util.{CallerCon

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-20 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r152093236 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala --- @@ -56,11 +56,38 @@ private[spark] class Applica

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-16 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r151490038 --- Diff: core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala --- @@ -216,7 +216,9 @@ private[spark] object CoarseGrainedExecut

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-15 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r151320052 --- Diff: core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala --- @@ -216,7 +216,9 @@ private[spark] object CoarseGrainedExe

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-14 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r151020385 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -745,15 +739,20 @@ private[spark] class Client(

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-14 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r151020337 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -412,8 +412,6 @@ class SparkContext(config: SparkConf) extends Logging { }

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-14 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r151015745 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -412,8 +412,6 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-14 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r151017494 --- Diff: core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala --- @@ -216,7 +216,9 @@ private[spark] object CoarseGrainedExe

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-14 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r151018454 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -745,15 +739,20 @@ private[spark] class Client(

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-14 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r150916635 --- Diff: core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala --- @@ -216,7 +216,9 @@ private[spark] object CoarseGrainedExecut

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-14 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r150916112 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -92,6 +92,11 @@ object SparkSubmit extends CommandLineUtils with Logging {

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-14 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r150916079 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -551,13 +553,10 @@ private[spark] class SecurityManager( private[spark]

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-13 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r150751268 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -551,13 +553,10 @@ private[spark] class SecurityManager( private[spa

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-13 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r150751761 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -92,6 +92,11 @@ object SparkSubmit extends CommandLineUtils with Logging {

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-13 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r150752055 --- Diff: core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala --- @@ -216,7 +216,9 @@ private[spark] object CoarseGrainedExe

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-07 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r149466183 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -542,7 +496,55 @@ private[spark] class SecurityManager( * Gets the secret

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-07 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r149459107 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -542,7 +496,55 @@ private[spark] class SecurityManager( * Gets the secret

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-07 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r149456772 --- Diff: core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala --- @@ -398,9 +399,20 @@ private[spark] object RestSubmissionClient {

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-07 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r149455493 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -542,7 +496,55 @@ private[spark] class SecurityManager( * Gets the secret

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-07 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r149454492 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -542,7 +496,55 @@ private[spark] class SecurityManager( * Gets the secret

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-07 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r149451751 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -542,7 +496,55 @@ private[spark] class SecurityManager( * Gets the secret

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-07 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r149451263 --- Diff: core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala --- @@ -398,9 +399,20 @@ private[spark] object RestSubmissionClient {

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-07 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r149375998 --- Diff: core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala --- @@ -398,9 +399,20 @@ private[spark] object RestSubmissionClient {

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-07 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r149375879 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -365,22 +370,21 @@ object SparkSubmit extends CommandLineUtils with Logging {

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-07 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r149375835 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -542,7 +496,55 @@ private[spark] class SecurityManager( * Gets the secret

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-07 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r149361297 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -542,7 +496,55 @@ private[spark] class SecurityManager( * Gets the secret

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-01 Thread vanzin
GitHub user vanzin opened a pull request: https://github.com/apache/spark/pull/19631 [SPARK-22372][core, yarn] Make cluster submission use SparkApplication. The main goal of this change is to allow multiple cluster-mode submissions from the same JVM, without having them end up wi