[
https://issues.apache.org/jira/browse/YARN-45?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644664#comment-13644664
]
Carlo Curino commented on YARN-45:
----------------------------------
[~acmurthy] I see your point, which was in fact reflected more clearly in our
initial proposal. The only caveat is not to make this a capacity-only protocol
(which you are not, but I wanted to reiterate that there are other use cases).
I like [~bikassaha] and [~chris.douglas] spin on it (i.e., using
ResourceRequest), as it gives us the immediate "capacity angle", but will
eventually allow to evolve the implementations towards something richer (e.g.,
the preempt on behalf of a specific request
that Bikas considered before) without impact to the protocols.
I think there is a slightly cleaner version of Chris's proposal:
use ResourceRequest and to represent a request that only cares about overall
capacity we could express the ResourceRequest as a multiple of the minimum
allocation (i.e., if we want 100GB of RAM back and min_container size is 1GB we
ask for 100 x 1GB containers). This achieves Chris's proposal with a slightly
prettier use of ResourceRequest. Note that there are size-matching issues
(e.g., you have 1.5GB containers and I ask for 1x1GB containers, but we have
very similar problems with Resource).
I would say that as Chris pointed out [these semantics |
https://issues.apache.org/jira/browse/YARN-45?focusedCommentId=13628950&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13628950]
plus the use of ResourceRequest I propose here as a minor variation on Chris's
take should cover Arun's and Bika's comments (and I believe also the prior 45+
messages).
Thoughts?
> Scheduler feedback to AM to release containers
> ----------------------------------------------
>
> Key: YARN-45
> URL: https://issues.apache.org/jira/browse/YARN-45
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
> Reporter: Chris Douglas
> Assignee: Carlo Curino
> Attachments: YARN-45.patch, YARN-45.patch, YARN-45.patch,
> YARN-45.patch, YARN-45.patch, YARN-45_summary_of_alternatives.pdf
>
>
> The ResourceManager strikes a balance between cluster utilization and strict
> enforcement of resource invariants in the cluster. Individual allocations of
> containers must be reclaimed- or reserved- to restore the global invariants
> when cluster load shifts. In some cases, the ApplicationMaster can respond to
> fluctuations in resource availability without losing the work already
> completed by that task (MAPREDUCE-4584). Supplying it with this information
> would be helpful for overall cluster utilization [1]. To this end, we want to
> establish a protocol for the RM to ask the AM to release containers.
> [1] http://research.yahoo.com/files/yl-2012-003.pdf
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
