Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-136956668
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-136956654
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-136957034
[Test build #41916 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41916/consoleFull)
for PR 4027 at commit
[`fd0267f`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-137011910
[Test build #41916 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41916/console)
for PR 4027 at commit
[`fd0267f`](https://github.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-137012152
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-137012148
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user rnowling commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-139331885
Being able to explicitly control max memory and cores per executor and
executors per slave in Mesos coarse-grained mode would be awesome. Very much
looking forward to
Github user SleepyThread commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-141579731
@andrewor14 @pwendell @tnachen @dragos. In my company we have patched the
Spark to specify minimum and maximum no of core per executor with only one
executor runnin
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-141582708
The thought of this patch is give user the flexibility to provide a max and
min, and also optionally allow multiple executors or not. So in your case you
just need to sp
Github user SleepyThread commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-141590430
There is no minimum core per executor cap in this patch, only maximum
cores. Also i think we will need to add more documentation around it to make
these parameter m
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-141617352
I see, I can see having a minimum CPU per executor useful, that's a easy
change in this patch, other than that i think this patch can achieve what you
want
---
If you
Github user SleepyThread commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-141634118
Yes. Defaults in given patch will achieve the configurations.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHu
Github user nickpoorman commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-91735401
+1 Would love to see this much needed feature added.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. I
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-92553493
By the way, just an update on this. @pwendell and I think we should use the
same approach on Mesos as we do on YARN and will do so on standalone mode as in
#731. There
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-92631828
@andrewor14 What's the rationale to have exact cores per executor vs
allowing any cores that is between maxCoresPerExecutor and executor cores? It
seems like the cookie c
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-92653950
The trade off is in predictability and how easily users can interpret the
implications of the options they are passing. I think it could be fine
later on to add so
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-92657307
I see, only concern for me is that if we enforce this setting in mesos
coarse
Grain mode, then all existing users with most likely see a decrease of
performance given
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-92997768
Yes `spark.executor.cores` is definitely an optional setting. By default we
will continue to launch 1 executor per slave as before, where the executor
grabs all the co
Github user maasg commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-93029751
One of the issues this PR is solving is ensuring jobs could be forced to
spread over several nodes. This is particularly important for Spark Streaming
as parallelizing the
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-120792668
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-120792660
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-120792946
[Test build #37115 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37115/consoleFull)
for PR 4027 at commit
[`85362a3`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-120793071
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-120793068
[Test build #37115 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37115/console)
for PR 4027 at commit
[`85362a3`](https://github.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-122435597
@tnachen is this the next on your list to straighten out? I'd just like to
remind you that the dynamic allocation patch made some assumptions that there
can only be 1
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-122458457
@andrewor14 that's true, I'm thinking I should work on the removing the
shuffle data first since that can help make sure we tie up the dynamic
allocation feature much mo
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-122616819
OK, sounds good. It might make sense to just patch up #7195 quickly and
merge that one first in the mean time.
---
If your project is set up for it, you can reply to
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r25310704
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
---
@@ -63,20 +63,25 @@ private[spark] class Coar
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-76519231
[Test build #28124 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28124/consoleFull)
for PR 4027 at commit
[`d2d5cb2`](https://githu
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-76520378
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-76520377
[Test build #28124 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28124/consoleFull)
for PR 4027 at commit
[`d2d5cb2`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-76536787
[Test build #28127 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28127/consoleFull)
for PR 4027 at commit
[`4ec8b31`](https://githu
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-76538542
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-76538537
[Test build #28127 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28127/consoleFull)
for PR 4027 at commit
[`4ec8b31`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-85228821
[Test build #29034 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29034/consoleFull)
for PR 4027 at commit
[`6d04da1`](https://githu
Github user tnachen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r26987588
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
---
@@ -63,20 +63,25 @@ private[spark] class CoarseM
Github user tnachen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r26988123
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
---
@@ -204,35 +209,43 @@ private[spark] class Coars
Github user tnachen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r26988213
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
---
@@ -204,35 +209,43 @@ private[spark] class Coars
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-85239717
[Test build #29034 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29034/consoleFull)
for PR 4027 at commit
[`6d04da1`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-85239748
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-85652752
@andrewor14 it's true if you don't cap the maximum cores then the first one
will freely grab whatever it can. I think what I can do is that to ensure a
user that set the
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-85689685
[Test build #29110 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29110/consoleFull)
for PR 4027 at commit
[`c3cd8ce`](https://githu
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-85718866
[Test build #29110 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29110/consoleFull)
for PR 4027 at commit
[`c3cd8ce`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-85718873
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r27091956
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
---
@@ -63,20 +63,34 @@ private[spark] class Coar
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r27091981
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/mesos/CoarseMesosSchedulerBackendSuite.scala
---
@@ -0,0 +1,139 @@
+/*
+ * Licensed to th
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r27091990
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/mesos/CoarseMesosSchedulerBackendSuite.scala
---
@@ -0,0 +1,139 @@
+/*
+ * Licensed to th
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-85810614
@tnachen I just thought of something while I was reviewing the standalone
mode equivalent of this patch at #731. I wonder if it makes sense to allow the
user to specif
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-117832235
@tnachen can you address the comments to simplify this patch using
`spark.executor.cores` instead? As discussed above this is an optional setting
and reuses the same
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r33728044
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
---
@@ -109,6 +123,13 @@ private[spark] class Coa
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r33728151
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
---
@@ -211,35 +226,43 @@ private[spark] class Co
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-117835204
By the way this patch will likely conflict with #4984 quite a bit. I'm more
inclined to merge this one first because this is much smaller and the conflicts
will be ea
Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-117948614
@andrewor14 if this one is smaller, wouldn't it make more that conflicts
are easier to fix on this one, than the other one? Anyhow, up to you. I'm used
to fixing merge co
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-118185453
We can do it either way. I just didn't want to block the progress of this
patch if #4984 still has some progress to make. If it turns out that #4984 is
ready much soo
Github user dragos commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r33919242
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
---
@@ -109,6 +123,13 @@ private[spark] class CoarseM
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-75162291
Ok updated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
ena
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r25096773
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
---
@@ -63,20 +63,25 @@ private[spark] class Coar
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r25096822
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
---
@@ -63,20 +63,25 @@ private[spark] class Coar
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r25096910
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
---
@@ -63,20 +63,25 @@ private[spark] class Coar
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r25097455
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
---
@@ -63,20 +63,25 @@ private[spark] class Coar
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r25099491
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
---
@@ -204,35 +209,43 @@ private[spark] class Co
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r25099936
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
---
@@ -204,35 +209,43 @@ private[spark] class Co
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r25099735
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
---
@@ -63,20 +63,25 @@ private[spark] class Coar
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r25100067
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
---
@@ -204,35 +209,43 @@ private[spark] class Co
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r25100341
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
---
@@ -204,35 +209,43 @@ private[spark] class Co
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r25100554
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/mesos/CoarseMesosSchedulerBackendSuite.scala
---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to th
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r25100423
--- Diff: docs/running-on-mesos.md ---
@@ -226,6 +226,20 @@ See the [configuration page](configuration.html) for
information on Spark config
The f
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r25100490
--- Diff: docs/running-on-mesos.md ---
@@ -226,6 +226,20 @@ See the [configuration page](configuration.html) for
information on Spark config
The f
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r25100463
--- Diff: docs/running-on-mesos.md ---
@@ -226,6 +226,20 @@ See the [configuration page](configuration.html) for
information on Spark config
The f
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r25100639
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/mesos/CoarseMesosSchedulerBackendSuite.scala
---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to th
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-75314717
Hi @tnachen this looks like a reasonably straight-forward change. My inline
comments have mostly to do with naming and docs. One high level question: does
it make sens
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r25112596
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
---
@@ -63,20 +63,25 @@ private[spark] class Coar
Github user CodingCat commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r25138364
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
---
@@ -63,20 +64,25 @@ private[spark] class Coars
Github user CodingCat commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r25138507
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
---
@@ -63,20 +63,25 @@ private[spark] class Coars
Github user CodingCat commented on a diff in the pull request:
https://github.com/apache/spark/pull/4027#discussion_r25138542
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
---
@@ -204,35 +209,43 @@ private[spark] class Coa
Github user maasg commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-99211900
@tdas Would you have an opinion on this? Are there alternatives to
warranty the even spread of Streaming receivers over nodes of a Mesos cluster ?
---
If your project is
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-152667083
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-152667103
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-152668094
**[Test build #44709 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44709/consoleFull)**
for PR 4027 at commit
[`3dc1383`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-152669426
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-152669398
**[Test build #44709 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44709/consoleFull)**
for PR 4027 at commit
[`3dc1383`](https://git
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-152669425
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-158571748
**[Test build #46465 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46465/consoleFull)**
for PR 4027 at commit
[`08723f8`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-158572755
**[Test build #46465 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46465/consoleFull)**
for PR 4027 at commit
[`08723f8`](https://git
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-158572765
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-158572768
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/4
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-158598845
**[Test build #46472 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46472/consoleFull)**
for PR 4027 at commit
[`8a7a735`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-158614020
**[Test build #46472 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46472/consoleFull)**
for PR 4027 at commit
[`8a7a735`](https://git
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-158614041
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-158614042
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/4
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-158668110
@andrewor14 I've updated the patch now. Originally you suggested me to look
at deploy/master.scala to try to use the same configurations like
spark.executor.cores. But i
Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-158686949
@tnachen I think this trade-off has been discussed in [this
comment](https://github.com/apache/spark/pull/4027#issuecomment-92553493) and
the following three. Since there
Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-163952266
@tnachen @andrewor14 what should be the way forward? I know about a more
ambitious plan to rewrite the Mesos scheduler. Should we still push this in the
interim, or just
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-164037332
I still think the configs introduced in this patch expose too much to the
average user. It's not straightforward even to me to think about how "max
cores" and "max ex
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-164125291
Yes at this moment I'll be closing this PR and move forward with another
proposal that is to use spark.executor.cores and have a new coarse grain
scheduler.
> O
Github user tnachen closed the pull request at:
https://github.com/apache/spark/pull/4027
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is ena
Github user 3ourroom commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-164677869
NAVER - http://www.naver.com/
3ourr...@naver.com ëê» ë³´ë´ì ë©ì¼ ì´
ë¤ìê³¼ ê°ì ì´ì ë¡ ì
Github user rnowling commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-167166741
It's really unfortunate that this patch was closed without merging. I
disagree with @andrewor14 and others that it exposes to much to the average
user. It's much easi
Github user maver1ck commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-167260638
I agree.
In YARN mode we have configuration per node
```
YARN: The --num-executors option to the Spark YARN client controls how many
executors it will alloca
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4027#issuecomment-167643240
> It's much easier (for me) to think about the number of cores and amount
of memory per NODE
You did not understand my suggestion. That's exactly what I'm sug
1 - 100 of 136 matches
Mail list logo
