[ https://issues.apache.org/jira/browse/YARN-9351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
tianjuan updated YARN-9351: --------------------------- Description: if we configure queue capacity in absolute term, users can't use total resource of one partition even yarn.scheduler.capacity.<queue-path>.minimum-user-limit-percent is set to 100 for example there are two partition A,B, partition A has (120G memory,30 vcores), and partition B has (180G memory,60 vcores), and Queue Prod is configured with (75G memory, 25 vcores) partition A resource, like yarn.scheduler.capacity.root.Prod.accessible-node-labels.A.capacity=[memory=75Gi,vcores=25], and yarn.scheduler.capacity.root.Prod.accessible-node-labels.A.maximum-capacity=[memory=120Gi,vcores=30] yarn.scheduler.capacity.root.Prod.minimum-user-limit-percent=100, and at one point the used resource of queue Prod is (90G memory,10 vcores), at this time even though yarn.scheduler.capacity.<queue-path>.minimum-user-limit-percent is set to 100 , users in queue A can't get more resource. the reason for this is that when {color:#d04437}*computeUserLimit*{color}, partitionResource is used for comparing consumed, queueCapacity, so in the example (75G memory, 25 vcores) is the user limit. Resource currentCapacity = Resources.lessThan(resourceCalculator, partitionResource, consumed, queueCapacity) ? queueCapacity : Resources.add(consumed, required); Resource userLimitResource = Resources.max(resourceCalculator, partitionResource,Resources.divideAndCeil(resourceCalculator, resourceUsed, usersSummedByWeight),Resources.divideAndCeil(resourceCalculator,Resources.multiplyAndRoundDown(currentCapacity, getUserLimit()),100)); but when *{color:#d04437}canAssignToUser{color}* = Resources.greaterThan(resourceCalculator, clusterResource, user.getUsed(nodePartition), limit) *{color:#d04437}clusterResource{color}* {color:#333333}is used for for comparing * *used and limit, the result *is false.*{color} was: if we configure queue capacity in absolute term, users can't use total resource of one partition even yarn.scheduler.capacity.<queue-path>.minimum-user-limit-percent is set to 100 for example there are two partition A,B, partition A has (120G memory,30 vcores), and partition B has (180G memory,60 vcores), and Queue Prod is configured with (75G memory, 25 vcores) partition A resource, like yarn.scheduler.capacity.root.Prod.accessible-node-labels.A.capacity=[memory=75Gi,vcores=25], and yarn.scheduler.capacity.root.Prod.accessible-node-labels.A.maximum-capacity=[memory=120Gi,vcores=30] yarn.scheduler.capacity.root.Prod.minimum-user-limit-percent=100, and at one point the used resource of queue Prod is (90G memory,10 vcores), at this time even though yarn.scheduler.capacity.<queue-path>.minimum-user-limit-percent is set to 100 , users in queue A can't get more resource. the reason for this is that when {color:#d04437}*computeUserLimit*{color}, partitionResource is used for comparing consumed, queueCapacity, so in the example (75G memory, 25 vcores) is the user limit. Resource currentCapacity = Resources.lessThan(resourceCalculator, partitionResource, consumed, queueCapacity) ? queueCapacity : Resources.add(consumed, required); Resource userLimitResource = Resources.max(resourceCalculator, partitionResource,Resources.divideAndCeil(resourceCalculator, resourceUsed, usersSummedByWeight),Resources.divideAndCeil(resourceCalculator,Resources.multiplyAndRoundDown(currentCapacity, getUserLimit()),100)); but when canAssignToUser = Resources.greaterThan(resourceCalculator, clusterResource, user.getUsed(nodePartition), limit) *{color:#d04437}clusterResource{color}* {color:#333333}is used for for comparing ** used and limit, the result *is false.*{color} > user can't use total resources of one partition even > yarn.scheduler.capacity.<queue-path>.minimum-user-limit-percent is set to 100 > ----------------------------------------------------------------------------------------------------------------------------------- > > Key: YARN-9351 > URL: https://issues.apache.org/jira/browse/YARN-9351 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler > Affects Versions: 2.9.2 > Reporter: tianjuan > Priority: Major > > if we configure queue capacity in absolute term, users can't use total > resource of one partition even > yarn.scheduler.capacity.<queue-path>.minimum-user-limit-percent is set to 100 > for example there are two partition A,B, partition A has (120G memory,30 > vcores), and partition B has (180G memory,60 vcores), and Queue Prod is > configured with (75G memory, 25 vcores) partition A resource, like > yarn.scheduler.capacity.root.Prod.accessible-node-labels.A.capacity=[memory=75Gi,vcores=25], > and > yarn.scheduler.capacity.root.Prod.accessible-node-labels.A.maximum-capacity=[memory=120Gi,vcores=30] > yarn.scheduler.capacity.root.Prod.minimum-user-limit-percent=100, and at one > point the used resource of queue Prod is (90G memory,10 vcores), at this time > even though yarn.scheduler.capacity.<queue-path>.minimum-user-limit-percent > is set to 100 , users in queue A can't get more resource. > > the reason for this is that when {color:#d04437}*computeUserLimit*{color}, > partitionResource is used for comparing consumed, queueCapacity, so in the > example (75G memory, 25 vcores) is the user limit. > Resource currentCapacity = Resources.lessThan(resourceCalculator, > partitionResource, consumed, queueCapacity) > ? queueCapacity > : Resources.add(consumed, required); > Resource userLimitResource = Resources.max(resourceCalculator, > partitionResource,Resources.divideAndCeil(resourceCalculator, resourceUsed, > usersSummedByWeight),Resources.divideAndCeil(resourceCalculator,Resources.multiplyAndRoundDown(currentCapacity, > getUserLimit()),100)); > > but when *{color:#d04437}canAssignToUser{color}* = > Resources.greaterThan(resourceCalculator, clusterResource, > user.getUsed(nodePartition), limit) > *{color:#d04437}clusterResource{color}* {color:#333333}is used for for > comparing * *used and limit, the result *is false.*{color} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org