[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649253#comment-15649253 ]
Wangda Tan commented on YARN-5139: ---------------------------------- [~curino], Good point, yeah I think we should have a better structured logic to support different levels of "fairness". They may come from different places: 1) How/when to sort queues / apps, we can re-sort queues/apps for each allocated containers, or we can delay the re-sorting 2) Which I mentioned above: maximum number of pending to-be-committed resource allocations. 3) Lower level fairness such as user-limit, etc. So basically, instead of putting them into one place (such as "pluggable fairness policy"), we may need to have a couple of configurable places to update the scheduler to be more fairness or less fairness. Since I will take vacation soon, I think we could have some discussions 3-4 weeks later. > [Umbrella] Move YARN scheduler towards global scheduler > ------------------------------------------------------- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature > Reporter: Wangda Tan > Assignee: Wangda Tan > Attachments: Explanantions of Global Scheduling (YARN-5139) > Implementation.pdf, YARN-5139-Concurrent-scheduling-performance-report.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf, > YARN-5139.000.patch, wip-1.YARN-5139.patch, wip-2.YARN-5139.patch, > wip-3.YARN-5139.patch, wip-4.YARN-5139.patch, wip-5.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: > Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: > for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org