[
https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13785365#comment-13785365
]
Bikas Saha commented on YARN-1197:
----------------------------------
Alejandro, we have already iterated through the create new container and merge
with old container approach and saw no advantage over a cleaner api that asks
for increase in an existing containers request. Internal implementations of
schedulers can choose to model this as a new allocation followed by an internal
merge instead of exposing a new container to the external environment only to
have it merged back by the AM. This is more burden on the AM, NM and confusing
to tracking services like logs/AHS. Also the delta container request would have
to specify specific locality to place it exactly where the existing container
is or else the RM can allocate the delta anywhere. This should just happen
automatically if the request was to increase an existing containers capacity.
Given the above, here is the current approach based on all the comments above
To increase container resource
1) AM ask RM to increase resources
2) RM increases the resources and provides a new token to sign that increase
3) AM calls new NM API to increase the resources, signed with above token
To decrease container resource
1) AM ask NM to decrease the resource
2) NM tell RM about this
3) RM informs the AM about the change and allocates free space somewhere else
The up side of this is that it avoids race conditions. The down side is that
the flow is different in both cases. Another downside is that the NM does not
know if the reduced size is going to be unsupported by the scheduler. IMO, its
not a good idea for RM scheduler internals to leak into the NM and NM should
not get involved in those kind of decisions. One simple solution would be that
when the NM tells the RM about reduced resources, then the RM only reduces its
internal book-keeping by the amount until min-allocation is reached. The extra
freed resources would sit unused in the node but they would anyways be unused
if min-allocation was being enforced by the NM. So we are not worse off. As far
as the AM is concerned it already promised to use stay within the reduced
capacity and so this shouldnt affect it.
> Support changing resources of an allocated container
> ----------------------------------------------------
>
> Key: YARN-1197
> URL: https://issues.apache.org/jira/browse/YARN-1197
> Project: Hadoop YARN
> Issue Type: Task
> Components: api, nodemanager, resourcemanager
> Affects Versions: 2.1.0-beta
> Reporter: Wangda Tan
> Attachments: yarn-1197.pdf
>
>
> Currently, YARN cannot support merge several containers in one node to a big
> container, which can make us incrementally ask resources, merge them to a
> bigger one, and launch our processes. The user scenario is described in the
> comments.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
