Github user njwhite commented on the issue:
https://github.com/apache/spark/pull/12951
Actually, @kayousterhout - I'm not entirely sure what you expect for the
semantics of maxShares in general. Maybe a worked example would help: if I have
a pool X with 5 running tasks from Taskset
Github user njwhite commented on the issue:
https://github.com/apache/spark/pull/12951
@kayousterhout minShares is a configuration parameter for the fair
scheduler algorithm only - what would the semantics of a maxShares setting for
the FIFO algorithm be?
---
If your project is set
Github user njwhite commented on the issue:
https://github.com/apache/spark/pull/12951
ping?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user njwhite commented on the issue:
https://github.com/apache/spark/pull/12951
Hi @kayousterhout - I've renamed all references to `maxRunningTasks` and
updated the Markdown documentation in the repo. Is this OK? Thanks -
---
If your project is set up for it, you can reply
Github user njwhite commented on the issue:
https://github.com/apache/spark/pull/12951
@squito is this OK?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user njwhite commented on the issue:
https://github.com/apache/spark/pull/12951
@squito thanks - I've expanded the `Scheduler respects maxRunningTasks
setting of its pool` test to cover the cases you mention (and a couple of
others).
---
If your project is set up for it, you
Github user njwhite commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-97641
Thanks @markhamstra! The Jenkins build failed because a single test,
`ExternalAppendOnlyMapSuite#spilling with compression`, failed. It seems
unrelated (and passes
Github user njwhite commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-222175368
Thanks @squito; I've renamed the setting to `maxRunningTasks` and added the
tests you asked for to `TaskSetManagerSuite`. I've also added support (& t
Github user njwhite commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-220555299
Thanks for the review @squito - I've commented on the JIRA about why this
feature would be useful. As for the implementation - maybe "maxShare" is the
Github user njwhite commented on the pull request:
https://github.com/apache/spark/pull/12951#issuecomment-217498019
@HyukjinKwon I've run `run-tests` and fixed all the style issues. Could you
take another look? Thanks -
---
If your project is set up for it, you can reply
GitHub user njwhite opened a pull request:
https://github.com/apache/spark/pull/12951
[SPARK-15176][Core] Add maxShares setting to Pools
## What changes were proposed in this pull request?
Help guarantee resource availablity by (hierarchically) limiting the amount
of tasks
Github user njwhite commented on the pull request:
https://github.com/apache/spark/pull/12620#issuecomment-214210492
@davies I'm using this to use the "dill" serializer, as it can pickle more
things (and allows more fine-grained control) than the cloud-pickle serializer.
GitHub user njwhite opened a pull request:
https://github.com/apache/spark/pull/12620
[SPARK-14859][PYSPARK] Make Lambda Serializer Configurable
## What changes were proposed in this pull request?
Store the serializer that we should use to serialize RDD transformation
13 matches
Mail list logo
