[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-09 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-69426979 Oh gosh it is merged finally. Thanks guys for persistent comments. @andrewor14 @tgravescs @vanzin @sryza --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22743735 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -87,6 +91,21 @@ private[spark] class ClientArguments(args:

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-09 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-69400682 Ok I'm merging this into master thanks @WangTaoTheTonic for your repeated updates. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-09 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3607 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-69286150 [Test build #25284 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25284/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-69286154 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-08 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22697617 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -87,6 +92,21 @@ private[spark] class ClientArguments(args:

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-69286990 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-69286987 [Test build #25285 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25285/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-08 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22697722 --- Diff: yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala --- @@ -68,8 +68,6 @@ private[spark] class

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-69282417 [Test build #25285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25285/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-69281668 [Test build #25284 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25284/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-08 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22697386 --- Diff: yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala --- @@ -68,8 +68,6 @@ private[spark] class

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-08 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-69148864 @tgravescs @andrewor14 Sorry to produce so much issues in last rebase. I've fixed them and tested again on my cluster. Here is the

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-08 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22640199 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -38,24 +38,30 @@ private[spark] class ClientArguments(args:

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-08 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22640201 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -38,24 +38,30 @@ private[spark] class ClientArguments(args:

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-69152255 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-69152244 [Test build #25208 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25208/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-08 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22671487 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -87,6 +92,21 @@ private[spark] class ClientArguments(args:

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-08 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22671504 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -87,6 +92,21 @@ private[spark] class ClientArguments(args:

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-08 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22671524 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -87,6 +92,21 @@ private[spark] class ClientArguments(args:

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-08 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22671592 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -87,6 +92,21 @@ private[spark] class ClientArguments(args:

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-08 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22672128 --- Diff: yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala --- @@ -68,8 +68,6 @@ private[spark] class

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-08 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-69229012 Hi @WangTaoTheTonic the latest changes look pretty close. I believe the semantics of what should be set in what mode is as discussed. It would be good if other

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22680114 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -87,6 +92,21 @@ private[spark] class ClientArguments(args:

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22680079 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -87,6 +92,21 @@ private[spark] class ClientArguments(args:

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-07 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22587969 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -38,24 +38,30 @@ private[spark] class ClientArguments(args:

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-07 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22588093 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -38,24 +38,30 @@ private[spark] class ClientArguments(args:

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-07 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-69028757 @WangTaoTheTonic have you manually tested this on a real cluster? If not could you please and make sure both the overhead and memory work in both cluster and client

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-07 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-69146835 [Test build #25208 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25208/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-07 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22593396 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -87,6 +93,24 @@ private[spark] class ClientArguments(args:

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-07 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22593879 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -87,6 +93,24 @@ private[spark] class ClientArguments(args:

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-07 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-69042147 Hey @WangTaoTheTonic I believe the latest changes correctly addresses my last concern (that in client mode the `amMemory` shouldn't depend on `--driver-memory` at

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-07 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22593149 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -38,24 +38,30 @@ private[spark] class ClientArguments(args:

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-07 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22593068 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -38,24 +38,30 @@ private[spark] class ClientArguments(args:

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-07 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22593506 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -38,24 +38,30 @@ private[spark] class ClientArguments(args:

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-07 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-69041998 Hey @WangTaoTheTonic I believe the latest changes correctly addresses my last concern (that in client mode the `amMemory` shouldn't depend on `--driver-memory` at

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-05 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-68733624 @andrewor14 What to do now? @vanzin @sryza @tgravescs Someone has any better idea? --- If your project is set up for it, you can reply to this email and

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-68825511 [Test build #25087 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25087/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-68825514 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-68821743 [Test build #25087 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25087/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-05 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-68821904 @vanzin Oh thanks for that. @andrewor14 Updated. Could you check if it is ok now? --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-05 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-68750750 I'm not sure I understand the question? You can modify `amMemory` after `parseArgs()` is done, at which point you have all the info you need to make the right decision.

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-25 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-68118403 Actually we cann't confirm that we will get `--class`(the one indicates cluster or client mode) before `--driver-memory', can we? --- If your project is set up

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-23 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-68033344 @WangTaoTheTonic what I mean is more like the following. Currently we set `amMemory` to the value of `--driverMemory` in `parseArgs`. We should do this only if this

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-22 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67816606 @andrewor14 Ok, I got what you mean. I think I have a misunderstanding before. To solve this problem, should we just delete `(--driver-memory,

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-22 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67881730 @WangTaoTheTonic that would fix it, but I think in addition to that we should also add a check in `ClientArguments` itself in case the user calls into the `Client`

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67918378 [Test build #24723 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24723/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-22 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67918563 @andrewor14 I use a hack(if `amMemory` equals default value 512) to determine whether `amMemory` is assigned by `--driver-memory`. Call me if you have any

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67921642 [Test build #24723 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24723/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67921645 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-19 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67705762 I think it is not correct to say the client mode's AM memory will use --driver-memory if saprk.yarn.am.memory is not set. Actually it use the default value(512m).

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22050878 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -39,23 +39,37 @@ private[spark] class ClientArguments(args:

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22051045 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -39,23 +39,37 @@ private[spark] class ClientArguments(args:

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22051552 --- Diff: docs/running-on-yarn.md --- @@ -22,6 +22,14 @@ Most of the configs are the same for Spark on YARN as for other deployment modes table

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22051716 --- Diff: docs/running-on-yarn.md --- @@ -92,6 +100,13 @@ Most of the configs are the same for Spark on YARN as for other deployment modes /td

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22063984 --- Diff: docs/running-on-yarn.md --- @@ -92,6 +100,13 @@ Most of the configs are the same for Spark on YARN as for other deployment modes /td

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22065439 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -39,23 +39,37 @@ private[spark] class ClientArguments(args:

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22065568 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -39,23 +39,37 @@ private[spark] class ClientArguments(args:

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22065666 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -39,23 +39,37 @@ private[spark] class ClientArguments(args:

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67544093 Hey @WangTaoTheTonic @tgravescs I just thought of the following: In client mode, the AM memory should never even depend on driver memory at all. Currently,

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67586970 [Test build #24616 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24616/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22087678 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -39,23 +39,37 @@ private[spark] class ClientArguments(args:

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67592294 Now I assign value to `isClusterMode` and log the warning when non-used configs set in cluster/client mode in validateArgs()。 @andrewor14 I think it is

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67592484 [Test build #24616 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24616/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67592488 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67592747 [Test build #24621 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24621/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67593144 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67593138 [Test build #24621 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24621/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67593348 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67593578 [Test build #24622 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24622/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67593976 [Test build #24622 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24622/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67593977 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67596094 [Test build #24623 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24623/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67599686 [Test build #24623 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24623/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67599691 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22090369 --- Diff: docs/running-on-yarn.md --- @@ -22,6 +22,14 @@ Most of the configs are the same for Spark on YARN as for other deployment modes table

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r22090815 --- Diff: docs/running-on-yarn.md --- @@ -22,6 +22,14 @@ Most of the configs are the same for Spark on YARN as for other deployment modes table

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67601463 [Test build #24628 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24628/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67606019 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-18 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67606014 [Test build #24628 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24628/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67317284 [Test build #24545 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24545/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-17 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67318029 I have tested in my cluster(3 NM nodes with 8GB memory each), with configuration: `spark.driver.memory=4G spark.yarn.am.memoryOverhead=1024

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67324257 [Test build #24545 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24545/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67324262 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-17 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67339282 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67339696 [Test build #24548 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24548/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67354665 [Test build #24548 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24548/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67354684 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-16 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67125228 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-16 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67125457 [Test build #24489 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24489/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-16 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67133181 [Test build #24489 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24489/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67133187 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-13 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-66870555 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-66870575 [Test build #24436 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24436/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-13 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-66870621 The failed cases are all about streaming, something wrong? @andrewor14 --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-66872007 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-66872004 [Test build #24436 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24436/consoleFull) for PR 3607 at commit

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-12 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3607#discussion_r21763581 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -39,23 +39,34 @@ private[spark] class ClientArguments(args:

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-12 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-66815050 Hey @WangTaoTheTonic this LGTM minus the warning comment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

  1   2   >