Github user lianhuiwang commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-85542318
@tgravescs thanks, i know what you means. i will take a look at it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on G
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-85536800
filed https://issues.apache.org/jira/browse/SPARK-6506 for this
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub a
Github user lianhuiwang commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-85531984
@tgravescs
does this mean that spark has to be installed on all the nodes? That
shouldn't be needed on YARN.
yes, now we need to put spark_home dir to all no
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-85502310
so it appears setting SPARK_HOME to anything on AM and executors works
around this error:
spark.yarn.appMasterEnv.SPARK_HOME /bogus
spark.executorEnv.SPARK_HOME
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-85498686
> @andrewor14 for python application, if SPARK_HOME of submission node is
different from the nodeManager, so it can not work in my test.
example:submission node's versi
Github user lianhuiwang commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-73541324
@tgravescs your thought is right. but there is just different at Yarn
internal's Client. it is same way on spark-submit with Yarn client and Yarn
cluster.so i think t
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-73538842
I thought we had it documented somewhere in what modes different things ran
but I'm not seeing it in the documentation. I think I will file jira to add
some documentat
Github user lianhuiwang commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72578648
@andrewor14 thank you, About SPARK_HOME, we need to consider two places of
compatible. The first is communication between python context and Scala
context. The secon
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/3976
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enab
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72532026
Ok LGTM thanks a lot @lianhuiwang I'm merging this into master. We can fix
the SPARK_HOME issue separately later. Could you file a JIRA for it?
---
If your project is
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72531678
Alright, LGTM thanks a lot @lianhuiwang I'm merging this into master. We
can fix the SPARK_HOME issue separately.
---
If your project is set up for it, you can reply
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72437808
[Test build #26508 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26508/consoleFull)
for PR 3976 at commit
[`28a8a58`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72437821
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72428343
[Test build #26508 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26508/consoleFull)
for PR 3976 at commit
[`28a8a58`](https://githu
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72415530
[Test build #26490 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26490/consoleFull)
for PR 3976 at commit
[`67f8cee`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72415533
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72410777
[Test build #26490 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26490/consoleFull)
for PR 3976 at commit
[`67f8cee`](https://githu
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72403008
[Test build #26482 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26482/consoleFull)
for PR 3976 at commit
[`0319ae3`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72403015
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72402028
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72402026
[Test build #26480 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26480/consoleFull)
for PR 3976 at commit
[`2385ef6`](https://gith
Github user lianhuiwang commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72401201
for python application, if SPARK_HOME of submission node is different from
the nodeManager, so it can not work in my test. example:submission node's
version is 1.2, b
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23905022
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -134,12 +136,29 @@ object SparkSubmit {
}
}
+
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72399690
@lianhuiwang what happens now if the submission node uses a different
SPARK_HOME from the machines? Does it still work?
---
If your project is set up for it, you can
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23904936
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala ---
@@ -185,6 +192,7 @@ private[spark] class ClientArguments(args:
Array[
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23904930
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMasterArguments.scala
---
@@ -81,6 +91,9 @@ class ApplicationMasterArguments(val arg
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23904922
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -134,12 +136,29 @@ object SparkSubmit {
}
}
+
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23904929
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
@@ -430,6 +430,10 @@ private[spark] class ApplicationMaster(args:
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23904915
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -134,12 +136,29 @@ object SparkSubmit {
}
}
+
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72399545
[Test build #26482 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26482/consoleFull)
for PR 3976 at commit
[`0319ae3`](https://githu
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23904909
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -134,12 +136,29 @@ object SparkSubmit {
}
}
+
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23904882
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -267,10 +292,20 @@ object SparkSubmit {
// In yarn-cluster mode, use
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23904843
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -134,12 +136,29 @@ object SparkSubmit {
}
}
+
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23904826
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -134,12 +136,29 @@ object SparkSubmit {
}
}
+
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23904808
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -134,12 +136,29 @@ object SparkSubmit {
}
}
+
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23904612
--- Diff:
yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnClusterSuite.scala ---
@@ -98,6 +121,9 @@ class YarnClusterSuite extends FunSuite with
Bef
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72398202
[Test build #26480 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26480/consoleFull)
for PR 3976 at commit
[`2385ef6`](https://githu
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23904494
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -267,10 +292,20 @@ object SparkSubmit {
// In yarn-cluster mode, us
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23904474
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -134,12 +136,29 @@ object SparkSubmit {
}
}
+
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23904387
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
@@ -430,6 +430,10 @@ private[spark] class ApplicationMaster(args:
A
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72377952
Thanks for adding the test and getting it to work @lianhuiwang. Had a few
more comments, but this is looking close to me.
---
If your project is set up for it, you can rep
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23899955
--- Diff:
yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnClusterSuite.scala ---
@@ -98,6 +121,9 @@ class YarnClusterSuite extends FunSuite with
BeforeAnd
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23899908
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -267,10 +292,22 @@ object SparkSubmit {
// In yarn-cluster mode, use yarn
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23899881
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -267,10 +292,22 @@ object SparkSubmit {
// In yarn-cluster mode, use yarn
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23899688
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -267,10 +292,22 @@ object SparkSubmit {
// In yarn-cluster mode, use yarn
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23899671
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -134,12 +136,29 @@ object SparkSubmit {
}
}
+val
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72374357
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72374353
[Test build #26474 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26474/consoleFull)
for PR 3976 at commit
[`03640ab`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72371273
[Test build #26474 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26474/consoleFull)
for PR 3976 at commit
[`03640ab`](https://githu
Github user lianhuiwang closed the pull request at:
https://github.com/apache/spark/pull/3976
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
GitHub user lianhuiwang reopened a pull request:
https://github.com/apache/spark/pull/3976
[SPARK-5173]support python application running on yarn cluster mode
now when we run python application on yarn cluster mode through
spark-submit, spark-submit does not support python applicati
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72349756
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72349753
[Test build #26470 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26470/consoleFull)
for PR 3976 at commit
[`47d2fc3`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72348373
[Test build #26470 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26470/consoleFull)
for PR 3976 at commit
[`47d2fc3`](https://githu
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72327239
[Test build #26466 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26466/consoleFull)
for PR 3976 at commit
[`2adc8f5`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72327242
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72324906
[Test build #26466 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26466/consoleFull)
for PR 3976 at commit
[`2adc8f5`](https://githu
Github user lianhuiwang commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72323941
@JoshRosen can you help me? now i add a unit test for python applicaiton on
yarn cluster mode. but now there is a failed. i think its reason is environment
of pyspar
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72321234
[Test build #26464 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26464/consoleFull)
for PR 3976 at commit
[`d60bc60`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72321243
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72319231
[Test build #26464 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26464/consoleFull)
for PR 3976 at commit
[`d60bc60`](https://githu
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72317452
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72317446
[Test build #26462 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26462/consoleFull)
for PR 3976 at commit
[`5b30064`](https://gith
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23888565
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
@@ -430,6 +430,10 @@ private[spark] class ApplicationMaster(args:
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72315747
[Test build #26462 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26462/consoleFull)
for PR 3976 at commit
[`5b30064`](https://githu
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72304811
[Test build #26453 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26453/consoleFull)
for PR 3976 at commit
[`097a5ec`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72304815
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72303459
[Test build #26453 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26453/consoleFull)
for PR 3976 at commit
[`097a5ec`](https://githu
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72302649
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72302648
[Test build #26452 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26452/consoleFull)
for PR 3976 at commit
[`905a106`](https://gith
Github user lianhuiwang commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72302620
@sryza @andrewor14 thank for your reviews. I have updated with your
comments. Later i will add YarnClusterSuite for a test.
---
If your project is set up for it, yo
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72302619
[Test build #26452 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26452/consoleFull)
for PR 3976 at commit
[`905a106`](https://githu
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23885537
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -172,7 +172,8 @@ private[spark] class SparkSubmitArguments(args:
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23885383
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -165,6 +168,13 @@ object SparkSubmit {
}
}
+
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23884756
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -267,10 +277,22 @@ object SparkSubmit {
// In yarn-cluster mode, us
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23857919
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -267,10 +277,22 @@ object SparkSubmit {
// In yarn-cluster mode, use yarn
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23857876
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -267,10 +277,22 @@ object SparkSubmit {
// In yarn-cluster mode, use yarn
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23857834
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -148,7 +151,7 @@ object SparkSubmit {
}
// If we're runnin
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23857534
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala ---
@@ -103,11 +104,15 @@ private[spark] class ClientArguments(args:
Array[Str
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23856753
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -267,10 +277,22 @@ object SparkSubmit {
// In yarn-cluster mode, use yarn
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23856668
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala ---
@@ -103,11 +104,15 @@ private[spark] class ClientArguments(args:
Array[Str
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23856550
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -165,6 +168,13 @@ object SparkSubmit {
}
}
+// I
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23856503
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -138,8 +140,9 @@ object SparkSubmit {
(clusterManager, deployMode) match
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23862207
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -165,6 +168,13 @@ object SparkSubmit {
}
}
+
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23862118
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala ---
@@ -103,11 +104,15 @@ private[spark] class ClientArguments(args:
Arra
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23862088
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMasterArguments.scala
---
@@ -48,6 +50,14 @@ class ApplicationMasterArguments(val ar
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23865157
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
@@ -430,6 +430,10 @@ private[spark] class ApplicationMaster(args:
A
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23861918
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -172,7 +172,8 @@ private[spark] class SparkSubmitArguments(args:
S
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23865116
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
@@ -430,6 +430,10 @@ private[spark] class ApplicationMaster(args:
A
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23861868
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -172,7 +172,8 @@ private[spark] class SparkSubmitArguments(args:
S
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23864907
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -267,10 +277,22 @@ object SparkSubmit {
// In yarn-cluster mode, use
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23861584
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -267,10 +277,22 @@ object SparkSubmit {
// In yarn-cluster mode, use
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72240509
This looks like the right approach. Added some comments inline. Are you
able to add a test for this in `YarnClusterSuite`?
Also, one last small thing: in `PythonR
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23861241
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -138,8 +140,9 @@ object SparkSubmit {
(clusterManager, deployMode) m
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23858269
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -267,10 +277,22 @@ object SparkSubmit {
// In yarn-cluster mode, use yarn
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23858127
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -267,10 +277,22 @@ object SparkSubmit {
// In yarn-cluster mode, use yarn
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3976#discussion_r23857948
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -267,10 +277,22 @@ object SparkSubmit {
// In yarn-cluster mode, use yarn
Github user lianhuiwang commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72174543
@tgravescs or @andrewor14 Can you help to review this PR? thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on Gi
Github user voukka commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72173406
@lianhuiwang thank you for non-local files feature.
Could admins review this PR and merge it, please?
---
If your project is set up for it, you can reply to this
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3976#issuecomment-72016569
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26
1 - 100 of 113 matches
Mail list logo
