[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14236389#comment-14236389 ]
Craig Welch commented on YARN-2637: ----------------------------------- Something to be aware of wrt the way I implemented option 2- when making the change I noticed that, in fact, the share of the maxampercent apportioned out to queues was actually based on the maximum capacity of the queue (absMaxCapacity) not the baseline value (absCapacity). I went ahead and kept this logic because it is consistent with the earlier approach and it still provides the desired control although potentially allowing more resources being provided to am's in cluster-aggregate if queues have high max values. Meaning if the total max is say 200% (sum of queue max, as opposed to the 100% baseline), then the standard .1 maxam will actually allow 20% usage of resources for application masters in the cluster. You can still manage the usage effectively, though, so I thought it best to stay as consistent with existing meaning as possible (while fixing the surprising aspect of it...), just wanted to make sure you were aware/it was documented > maximum-am-resource-percent could be violated when resource of AM is > > minimumAllocation > ---------------------------------------------------------------------------------------- > > Key: YARN-2637 > URL: https://issues.apache.org/jira/browse/YARN-2637 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Affects Versions: 2.6.0 > Reporter: Wangda Tan > Assignee: Craig Welch > Priority: Critical > Attachments: YARN-2637.0.patch, YARN-2637.1.patch, > YARN-2637.12.patch, YARN-2637.13.patch, YARN-2637.15.patch, > YARN-2637.16.patch, YARN-2637.2.patch, YARN-2637.6.patch, YARN-2637.7.patch, > YARN-2637.9.patch > > > Currently, number of AM in leaf queue will be calculated in following way: > {code} > max_am_resource = queue_max_capacity * maximum_am_resource_percent > #max_am_number = max_am_resource / minimum_allocation > #max_am_number_for_each_user = #max_am_number * userlimit * userlimit_factor > {code} > And when submit new application to RM, it will check if an app can be > activated in following way: > {code} > for (Iterator<FiCaSchedulerApp> i=pendingApplications.iterator(); > i.hasNext(); ) { > FiCaSchedulerApp application = i.next(); > > // Check queue limit > if (getNumActiveApplications() >= getMaximumActiveApplications()) { > break; > } > > // Check user limit > User user = getUser(application.getUser()); > if (user.getActiveApplications() < > getMaximumActiveApplicationsPerUser()) { > user.activateApplication(); > activeApplications.add(application); > i.remove(); > LOG.info("Application " + application.getApplicationId() + > " from user: " + application.getUser() + > " activated in queue: " + getQueueName()); > } > } > {code} > An example is, > If a queue has capacity = 1G, max_am_resource_percent = 0.2, the maximum > resource that AM can use is 200M, assuming minimum_allocation=1M, #am can be > launched is 200, and if user uses 5M for each AM (> minimum_allocation). All > apps can still be activated, and it will occupy all resource of a queue > instead of only a max_am_resource_percent of a queue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)