[ https://issues.apache.org/jira/browse/YARN-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16032452#comment-16032452 ]
daemon commented on YARN-6320: ------------------------------ I think I have solved the problem and in my test environment the scheduler can assign over 5000 containers per second. > FairScheduler:Identifying apps to assign in updateThread > -------------------------------------------------------- > > Key: YARN-6320 > URL: https://issues.apache.org/jira/browse/YARN-6320 > Project: Hadoop YARN > Issue Type: Bug > Reporter: Tao Jie > > In FairScheduler today, we have 1)UpdateThread that update queue/app status, > fairshare, starvation info, 2)nodeUpdate triggered by NM heartbeat that do > the scheduling. When we handle one nodeUpdate, we will top-down from the root > queue to the leafqueues and find the most needy application to allocate > container according to queue's fairshare. Also we should sort children at > each hierarchy level. > My thought is that we have a global sorted {{candidateAppList}} which keeps > apps need to assign, and move the logic that "find app that should allocate > resource to" from nodeUpdate to UpdateThread. In UpdateThread, we find > candidate apps to assign and put them into {{candidateAppList}}. In > nodeUpdate, we consume the list and allocate containers to apps. > As far as I see, we can have 3 benifits: > 1, nodeUpdate() is invoked much more frequently than update() in > UpdateThread, especially in a large cluster. As a result we can reduce much > unnecessary sorting. > 2, It will have better coordination with YARN-5829, we can indicate apps to > assign more directly rather than let nodes find the best apps to assign. > 3, It seems to be easier to introduce scheduling restricts such as nodelabel, > affinity/anti-affinity into FS, since we can pre-allocate containers > asynchronously. > [~kasha], [~templedf], [~yufeigu] like to hear your thoughts. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org