[
https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-5139:
-----------------------------
Attachment:
YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf
Have offline discussion with [~vinodkv], [~kasha] and [~templedf]. All gave
very valuable suggestions to the overall design.
I incorperated these suggestions to the design doc:
*Biggest update of the new design:*
It added the concept of concurrent resource allocator and resource committer:
- Resource allocator scan the read-only scheduler state, traverse the queue
structure, and make resource allocation proposal (Like allocate <3G, 1 vcore>
on host1 for app1 from queue=A). Since it won't change any scheduler state, we
can have multiple resource allocator run concurrently in different threads.
- Resource allocation proposal will be send to resource commtter which can
decide if the resource allocation proposal could be accepted or rejected, and
it will update scheduler state if any resource allocation proposal is accepted.
In the future we could implement optimistic locking logic to better merge
conflicts.
More details please refer to attached ver.2 doc.
Any comments are welcome!
> [Umbrella] Move YARN scheduler towards global scheduler
> -------------------------------------------------------
>
> Key: YARN-5139
> URL: https://issues.apache.org/jira/browse/YARN-5139
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Wangda Tan
> Assignee: Wangda Tan
> Attachments:
> YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf,
> YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf,
> wip-1.YARN-5139.patch, wip-2.YARN-5139.patch, wip-3.YARN-5139.patch
>
>
> Existing YARN scheduler is based on node heartbeat. This can lead to
> sub-optimal decisions because scheduler can only look at one node at the time
> when scheduling resources.
> Pseudo code of existing scheduling logic looks like:
> {code}
> for node in allNodes:
> Go to parentQueue
> Go to leafQueue
> for application in leafQueue.applications:
> for resource-request in application.resource-requests
> try to schedule on node
> {code}
> Considering future complex resource placement requirements, such as node
> constraints (give me "a && b || c") or anti-affinity (do not allocate HBase
> regionsevers and Storm workers on the same host), we may need to consider
> moving YARN scheduler towards global scheduling.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
