[
https://issues.apache.org/jira/browse/YARN-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15155692#comment-15155692
]
Karthik Kambatla commented on YARN-1011:
----------------------------------------
Had offline discussions with [~jlowe], [~nroberts], [~elgoiri], [~kkaranasos]
and [~asuresh]. Take-aways:
# To ensure the guaranteed containers continue to be allocated exactly the same
way as today, we leave that scheduling logic as is.
# A "second scheduler" is responsible for allocating opportunistic containers.
## This "second scheduler" could be another method that is called during node
update, or just another thread that runs asynchronously.
## Using an asynchronous thread allows us to process the nodes in the order of
unused resources instead of node heartbeat.
## Opportunistic scheduling could trigger only after the cluster allocation is
over a threshold - initially, we could hard code it to 80% of cluster capacity.
# When the scheduler comes around to allocate a guaranteed container for a
previously allocated opportunistic container, that container is promoted.
## Promotion on the same node is straight-forward and always desirable.
## Promotion across nodes is more complicated and leads to resource wastage.
However, not promoting could lead to an application getting resources later
than what it would have with oversubscription turned off. Accordingly, we could
have a policy to enable/disable cross-node promotion. To begin with, it would
be disabled. We could always add the option of enabling it in the future.
> [Umbrella] Schedule containers based on utilization of currently allocated
> containers
> -------------------------------------------------------------------------------------
>
> Key: YARN-1011
> URL: https://issues.apache.org/jira/browse/YARN-1011
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Arun C Murthy
> Attachments: yarn-1011-design-v0.pdf, yarn-1011-design-v1.pdf,
> yarn-1011-design-v2.pdf
>
>
> Currently RM allocates containers and assumes resources allocated are
> utilized.
> RM can, and should, get to a point where it measures utilization of allocated
> containers and, if appropriate, allocate more (speculative?) containers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
