[
https://issues.apache.org/jira/browse/YARN-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14217207#comment-14217207
]
Sriram Rao commented on YARN-2877:
----------------------------------
The proposal:
# Extend the NM to support task queueing. AM's can queue tasks directly at
the NM's and the NM's will execute those tasks opportunistically.
# Extend the type of containers that YARN exposes:
#* CONSERVATIVE: This corresponds to containers allocated by YARN today.
#* OPTIMISTIC: This corresponds to a new class of containers, which will be
queued for execution at the NM.
This extension allows AM's to control what type of container they are
requesting from the RM framework.
# Extend the NM with a "local RM" (i.e., a local Resource Manager) which uses
local policies for deciding when an "OPTIMISTIC container" can be executed.
We are exploring using timed leases for OPTIMISTIC containers to ensure minimum
duration of execution. On the other hand, this mechanism allows NM's to free
up resources and thus guarantee predictable start times for CONSERVATIVE
containers.
There are additional motivations for the uses of this feature and we will
discuss them in follow-up comments.
> Extend YARN to support distributed scheduling
> ---------------------------------------------
>
> Key: YARN-2877
> URL: https://issues.apache.org/jira/browse/YARN-2877
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager, resourcemanager
> Reporter: Sriram Rao
>
> This is an umbrella JIRA that proposes to extend YARN to support distributed
> scheduling. Briefly, some of the motivations for distributed scheduling are
> the following:
> 1. Improve cluster utilization by opportunistically executing tasks otherwise
> idle resources on individual machines.
> 2. Reduce allocation latency. Tasks where the scheduling time dominates
> (i.e., task execution time is much less compared to the time required for
> obtaining a container from the RM).
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
