[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-12-29 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-167899082 > Earlier, you had also suggested offering an option for the amount of memory per executor. Is that still valid in your proposal? What do you mean? You can

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-12-29 Thread rnowling
Github user rnowling commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-167891616 @andrewor14 I wasn't clear about which of your posts I was referring to -- my apologies. The `spark.executor.cores` suggestion to set the number of cores per executor

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-12-28 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-167643240 > It's much easier (for me) to think about the number of cores and amount of memory per NODE You did not understand my suggestion. That's exactly what I'm

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-12-25 Thread maver1ck
Github user maver1ck commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-167260638 I agree. In YARN mode we have configuration per node ``` YARN: The --num-executors option to the Spark YARN client controls how many executors it will

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-12-24 Thread rnowling
Github user rnowling commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-167166741 It's really unfortunate that this patch was closed without merging. I disagree with @andrewor14 and others that it exposes to much to the average user. It's much

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-12-14 Thread tnachen
Github user tnachen closed the pull request at: https://github.com/apache/spark/pull/4027 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-12-14 Thread 3ourroom
Github user 3ourroom commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-164677869 NAVER - http://www.naver.com/ 3ourr...@naver.com 님께 보내신 메일 이 다음과 같은 이유로

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-12-12 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-164125291 Yes at this moment I'll be closing this PR and move forward with another proposal that is to use spark.executor.cores and have a new coarse grain scheduler. >

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-12-11 Thread dragos
Github user dragos commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-163952266 @tnachen @andrewor14 what should be the way forward? I know about a more ambitious plan to rewrite the Mesos scheduler. Should we still push this in the interim, or just

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-12-11 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-164037332 I still think the configs introduced in this patch expose too much to the average user. It's not straightforward even to me to think about how "max cores" and "max

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-11-21 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-158668110 @andrewor14 I've updated the patch now. Originally you suggested me to look at deploy/master.scala to try to use the same configurations like spark.executor.cores. But

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-11-21 Thread dragos
Github user dragos commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-158686949 @tnachen I think this trade-off has been discussed in [this comment](https://github.com/apache/spark/pull/4027#issuecomment-92553493) and the following three. Since

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-11-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-158614042 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-11-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-158614041 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-11-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-158614020 **[Test build #46472 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46472/consoleFull)** for PR 4027 at commit

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-11-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-158571748 **[Test build #46465 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46465/consoleFull)** for PR 4027 at commit

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-11-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-158572755 **[Test build #46465 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46465/consoleFull)** for PR 4027 at commit

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-11-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-158572765 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-11-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-158572768 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-11-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-158598845 **[Test build #46472 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46472/consoleFull)** for PR 4027 at commit

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-10-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-152669398 **[Test build #44709 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44709/consoleFull)** for PR 4027 at commit

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-10-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-152669425 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-10-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-152669426 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-10-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-152668094 **[Test build #44709 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44709/consoleFull)** for PR 4027 at commit

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-10-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-152667083 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-10-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-152667103 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-19 Thread SleepyThread
Github user SleepyThread commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-141634118 Yes. Defaults in given patch will achieve the configurations. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-18 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-141617352 I see, I can see having a minimum CPU per executor useful, that's a easy change in this patch, other than that i think this patch can achieve what you want --- If

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-18 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-141582708 The thought of this patch is give user the flexibility to provide a max and min, and also optionally allow multiple executors or not. So in your case you just need to

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-18 Thread SleepyThread
Github user SleepyThread commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-141579731 @andrewor14 @pwendell @tnachen @dragos. In my company we have patched the Spark to specify minimum and maximum no of core per executor with only one executor

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-18 Thread SleepyThread
Github user SleepyThread commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-141590430 There is no minimum core per executor cap in this patch, only maximum cores. Also i think we will need to add more documentation around it to make these parameter

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-10 Thread rnowling
Github user rnowling commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-139331885 Being able to explicitly control max memory and cores per executor and executors per slave in Mesos coarse-grained mode would be awesome. Very much looking forward to

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136957034 [Test build #41916 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41916/consoleFull) for PR 4027 at commit

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136946464 [Test build #41912 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41912/console) for PR 4027 at commit

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136956654 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136956668 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136946511 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136946510 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-137011910 [Test build #41916 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41916/console) for PR 4027 at commit

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-137012152 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-137012148 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136605185 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136605170 [Test build #41859 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41859/console) for PR 4027 at commit

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136605181 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136619633 [Test build #41862 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41862/consoleFull) for PR 4027 at commit

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136602947 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136602918 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136617780 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136617747 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136604131 [Test build #41859 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41859/consoleFull) for PR 4027 at commit

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136920879 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136920924 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136921658 [Test build #41908 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41908/consoleFull) for PR 4027 at commit

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136620416 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136620412 [Test build #41862 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41862/console) for PR 4027 at commit

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread dragos
Github user dragos commented on a diff in the pull request: https://github.com/apache/spark/pull/4027#discussion_r38396413 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala --- @@ -357,30 +380,34 @@ private[spark] class

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136632132 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136674288 [Test build #41864 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41864/console) for PR 4027 at commit

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136632064 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136634507 [Test build #41864 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41864/consoleFull) for PR 4027 at commit

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136620419 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread dragos
Github user dragos commented on a diff in the pull request: https://github.com/apache/spark/pull/4027#discussion_r38395833 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala --- @@ -64,16 +63,32 @@ private[spark] class

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136674350 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136674347 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136928560 [Test build #41912 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41912/consoleFull) for PR 4027 at commit

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136928429 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136928439 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/4027#discussion_r38496861 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala --- @@ -357,30 +380,34 @@ private[spark] class

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136939354 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136939353 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-136939312 [Test build #41908 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41908/console) for PR 4027 at commit

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread dragos
Github user dragos commented on a diff in the pull request: https://github.com/apache/spark/pull/4027#discussion_r38499472 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala --- @@ -357,35 +379,39 @@ private[spark] class

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-07-18 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-122616819 OK, sounds good. It might make sense to just patch up #7195 quickly and merge that one first in the mean time. --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-07-17 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-122458457 @andrewor14 that's true, I'm thinking I should work on the removing the shuffle data first since that can help make sure we tie up the dynamic allocation feature much

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-07-17 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-122435597 @tnachen is this the next on your list to straighten out? I'd just like to remind you that the dynamic allocation patch made some assumptions that there can only be

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-07-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-120792668 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-07-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-120792660 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-07-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-120792946 [Test build #37115 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37115/consoleFull) for PR 4027 at commit

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-07-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-120793071 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-07-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-120793068 [Test build #37115 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37115/console) for PR 4027 at commit

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-07-06 Thread dragos
Github user dragos commented on a diff in the pull request: https://github.com/apache/spark/pull/4027#discussion_r33919242 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala --- @@ -109,6 +123,13 @@ private[spark] class

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-07-02 Thread dragos
Github user dragos commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-117948614 @andrewor14 if this one is smaller, wouldn't it make more that conflicts are easier to fix on this one, than the other one? Anyhow, up to you. I'm used to fixing merge

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-07-02 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-118185453 We can do it either way. I just didn't want to block the progress of this patch if #4984 still has some progress to make. If it turns out that #4984 is ready much

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-07-01 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-117832235 @tnachen can you address the comments to simplify this patch using `spark.executor.cores` instead? As discussed above this is an optional setting and reuses the same

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-07-01 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4027#discussion_r33728044 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala --- @@ -109,6 +123,13 @@ private[spark] class

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-07-01 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4027#discussion_r33728151 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala --- @@ -211,35 +226,43 @@ private[spark] class

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-07-01 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-117835204 By the way this patch will likely conflict with #4984 quite a bit. I'm more inclined to merge this one first because this is much smaller and the conflicts will be

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-05-05 Thread maasg
Github user maasg commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-99211900 @tdas Would you have an opinion on this? Are there alternatives to warranty the even spread of Streaming receivers over nodes of a Mesos cluster ? --- If your project is

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-04-14 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-92653950 The trade off is in predictability and how easily users can interpret the implications of the options they are passing. I think it could be fine later on to add

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-04-14 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-92631828 @andrewor14 What's the rationale to have exact cores per executor vs allowing any cores that is between maxCoresPerExecutor and executor cores? It seems like the cookie

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-04-14 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-92657307 I see, only concern for me is that if we enforce this setting in mesos coarse Grain mode, then all existing users with most likely see a decrease of performance

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-04-14 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-92997768 Yes `spark.executor.cores` is definitely an optional setting. By default we will continue to launch 1 executor per slave as before, where the executor grabs all the

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-04-14 Thread maasg
Github user maasg commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-93029751 One of the issues this PR is solving is ensuring jobs could be forced to spread over several nodes. This is particularly important for Spark Streaming as parallelizing the

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-04-13 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-92553493 By the way, just an update on this. @pwendell and I think we should use the same approach on Mesos as we do on YARN and will do so on standalone mode as in #731.

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-04-10 Thread nickpoorman
Github user nickpoorman commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-91735401 +1 Would love to see this much needed feature added. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-03-24 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-85718866 [Test build #29110 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29110/consoleFull) for PR 4027 at commit

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-03-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-85718873 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-03-24 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4027#discussion_r27091956 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala --- @@ -63,20 +63,34 @@ private[spark] class

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-03-24 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4027#discussion_r27091981 --- Diff: core/src/test/scala/org/apache/spark/scheduler/mesos/CoarseMesosSchedulerBackendSuite.scala --- @@ -0,0 +1,139 @@ +/* + * Licensed to

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-03-24 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4027#discussion_r27091990 --- Diff: core/src/test/scala/org/apache/spark/scheduler/mesos/CoarseMesosSchedulerBackendSuite.scala --- @@ -0,0 +1,139 @@ +/* + * Licensed to

  1   2   >