[ https://issues.apache.org/jira/browse/YARN-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14089724#comment-14089724 ]
Hudson commented on YARN-2008: ------------------------------ SUCCESS: Integrated in Hadoop-trunk-Commit #6031 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6031/]) YARN-2008. Fixed CapacityScheduler to calculate headroom based on max available capacity instead of configured max capacity. Contributed by Craig Welch (jianhe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1616580) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/DefaultResourceCalculator.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/DominantResourceCalculator.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/ResourceCalculator.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCSQueueUtils.java > CapacityScheduler may report incorrect queueMaxCap if there is hierarchy > queue structure > ----------------------------------------------------------------------------------------- > > Key: YARN-2008 > URL: https://issues.apache.org/jira/browse/YARN-2008 > Project: Hadoop YARN > Issue Type: Sub-task > Affects Versions: 2.3.0 > Reporter: Chen He > Assignee: Craig Welch > Fix For: 2.6.0 > > Attachments: YARN-2008.1.patch, YARN-2008.2.patch, YARN-2008.3.patch, > YARN-2008.4.patch, YARN-2008.5.patch, YARN-2008.6.patch, YARN-2008.7.patch, > YARN-2008.8.patch, YARN-2008.9.patch > > > If there are two queues, both allowed to use 100% of the actual resources in > the cluster. Q1 and Q2 currently use 50% of actual cluster's resources and > there is not actual space available. If we use current method to get > headroom, CapacityScheduler thinks there are still available resources for > users in Q1 but they have been used by Q2. > If the CapacityScheduelr has a hierarchy queue structure, it may report > incorrect queueMaxCap. Here is a example > || ||rootQueue|| || > | | / | > \ | > | L1ParentQueue1 | | > L1ParentQueue2 | > | (allowed to use up 80% of its parent) | | (allowed to use 20% > in minimum of its parent)| > | / | \ | | > | L2LeafQueue1 | L2LeafQueue2 | | > |(50% of its parent) | (50% of its parent in minimum) | | > When we calculate headroom of a user in L2LeafQueue2, current method will > think L2LeafQueue2 can use 40% (80%*50%) of actual rootQueue resources. > However, without checking L1ParentQueue1, we are not sure. It is possible > that L1ParentQueue2 have used 40% of rootQueue resources right now. Actually, > L2LeafQueue2 can only use 30% (60%*50%). -- This message was sent by Atlassian JIRA (v6.2#6252)