Peter Bacsko created YARN-10796: ----------------------------------- Summary: Capacity Scheduler: dynamic queue cannot scale out properly if it's capacity is 0% Key: YARN-10796 URL: https://issues.apache.org/jira/browse/YARN-10796 Project: Hadoop YARN Issue Type: Task Components: capacity scheduler, capacityscheduler Reporter: Peter Bacsko Assignee: Peter Bacsko
If we have a dynamic queue (AutoCreatedLeafQueue) with capacity = 0%, then it cannot properly scale even if it's max-capacity and the parent's max-capacity would allow it. Example: {noformat} Cluster Capacity: 16 GB / 16cpu (2 nodes, each with 8 GB / 8 cpu ) Container allocation size: 1G / 1 vcore Root.dynamic Effective Capacity: <memory: 8192, vCores: 8> ( 50.0%) Effective Max Capacity: <memory:16384, vCores:16> (100.0%) Template: Capacity: 40% Max Capacity: 100% User Limit Factor: 4 {noformat} leaf-queue-template.capacity = 40% leaf-queue-template.maximum-capacity = 100% leaf-queue-template.maximum-am-resource-percent = 50% leaf-queue-template.minimum-user-limit-percent =100% leaf-queue-template.user-limit-factor = 4 "root.dynamic" has a maximum capacity of 100% and a capacity of 50%. Let's assume there are running containers in these dynamic queues (MR sleep jobs): root.dynamic.user1 = 1 AM + 3 container (capacity = 40%) root.dynamic.user2 = 1 AM + 3 container (capacity = 40%) root.dynamic.user3 = 1 AM + 15 container (capacity = 0%) This scenario will result in an underutilized cluster. There will be approx 18% unused capacity. On the other hand, it's still possible to submit a new application to root.dynamic.user1 or root.dynamic.user2 and reaching a 100% utilization is possible. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org