Github user njwhite commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-97641
Thanks @markhamstra! The Jenkins build failed because a single test,
`ExternalAppendOnlyMapSuite#spilling with compression`, failed. It seems
unrelated (and passes
Github user markhamstra commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-30643
Added my comments to the JIRA. In short, I think there is a legitimate use
case for this, and there is a significant gap in our current fair-scheduling
pool API.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-11649
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-11646
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-11467
**[Test build #59501 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59501/consoleFull)**
for PR 12951 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-07408
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-07406
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-07170
**[Test build #59498 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59498/consoleFull)**
for PR 12951 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-222190773
**[Test build #59501 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59501/consoleFull)**
for PR 12951 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-222180851
**[Test build #59498 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59498/consoleFull)**
for PR 12951 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-222175839
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-222175830
**[Test build #59497 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59497/consoleFull)**
for PR 12951 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-222175841
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-222175470
**[Test build #59497 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59497/consoleFull)**
for PR 12951 at commit
Github user njwhite commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-222175368
Thanks @squito; I've renamed the setting to `maxRunningTasks` and added the
tests you asked for to `TaskSetManagerSuite`. I've also added support (& tests)
for
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-220742174
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-220742176
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-220741994
**[Test build #59030 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59030/consoleFull)**
for PR 12951 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-220720729
**[Test build #59030 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59030/consoleFull)**
for PR 12951 at commit
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-220720199
Jenkins, ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-220615364
ah, I completely overlooked `Pool.maxShare` referencing `runningTasks`,
makes more sense now.
The added tests verify that `maxShare` is getting propagated
Github user njwhite commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-220555299
Thanks for the review @squito - I've commented on the JIRA about why this
feature would be useful. As for the implementation - maybe "maxShare" is the
wrong word, as
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/12951#discussion_r63911572
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala ---
@@ -90,10 +92,16 @@ private[spark] class FairSchedulableBuilder(val
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-220379391
Hi @njwhite, I'm not sure I see a strong need for this -- I posted a msg on
jira (as Kay had earlier). We should keep discussion about the feature in
general there,
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-220379290
I commented on the JIRA.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-220276101
cc @kayousterhout for review
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ash211 commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-219463293
Ping, anything more needed on this PR before merging?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-217597424
@njwhite I am not a committer but just one of contributors. I guess most of
codes were written by @kayousterhout.
---
If your project is set up for it, you can
Github user njwhite commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-217498019
@HyukjinKwon I've run `run-tests` and fixed all the style issues. Could you
take another look? Thanks -
---
If your project is set up for it, you can reply to this
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/12951#discussion_r62332468
--- Diff: core/src/main/scala/org/apache/spark/scheduler/Pool.scala ---
@@ -21,6 +21,7 @@ import java.util.concurrent.{ConcurrentHashMap,
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/12951#discussion_r62331312
--- Diff: core/src/main/scala/org/apache/spark/scheduler/Pool.scala ---
@@ -21,6 +21,7 @@ import java.util.concurrent.{ConcurrentHashMap,
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/12951#discussion_r62330957
--- Diff: core/src/main/scala/org/apache/spark/scheduler/Pool.scala ---
@@ -47,6 +49,15 @@ private[spark] class Pool(
var name = poolName
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/12951#discussion_r62330929
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -98,6 +98,14 @@ private[spark] class TaskSetManager(
var
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/12951#discussion_r62330358
--- Diff: core/src/main/scala/org/apache/spark/scheduler/Pool.scala ---
@@ -21,6 +21,7 @@ import java.util.concurrent.{ConcurrentHashMap,
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/12951#discussion_r62329769
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala ---
@@ -90,10 +92,11 @@ private[spark] class
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-217399531
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
GitHub user njwhite opened a pull request:
https://github.com/apache/spark/pull/12951
[SPARK-15176][Core] Add maxShares setting to Pools
## What changes were proposed in this pull request?
Help guarantee resource availablity by (hierarchically) limiting the amount
of tasks
37 matches
Mail list logo
