[
https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14275700#comment-14275700
]
Craig Welch commented on YARN-2637:
-----------------------------------
Regarding the findbugs report for LeafQueue.lastClusterResource - access to
lastClusterResource appears to be synchronized everywhere except
getAbsActualCapacity, which I don't actually see being used anywhere - I'm
going to add a findbugs exception and a comment on the method so that if it is
used in the future synchronization can be addressed
-re [~leftnoteasy] 's latest:
-re 1 - actually, user limits are based on absolute queue capacity rather than
max capacity - this is apparently intentional because, although a queue can
exceed it's absolute capacity, an individual user is not supposed to, hence my
basing the user amlimit on the absolute capacity. The approach I use fits with
the original logic in CSQueueUtils which allows a user the greater of the
userlimit share of the absolute capacity or 1/# active users (so if there are
fewer users active than would reach the userlimit they can use the full queue
absolute capacity), the only correction being that we are using the actual
value of resources by application masters instead of one based on minalloc
-re 2 - Actually, the snippet provided is not quite correct, some schedulers
provide a cpu value as well. In any case, for encapsulation reasons it's
better to use the scheduler's value in case its means of determining this
changes in the future.
-re 3 - I can't see this making the slightest difference in understandability -
since these test's paths don't populate the rmapps I would simply be
individually putting mocked ones into the map instead of the single mock +
matcher for all the apps. The way it is seems clearer to me as all of the
mocking is together instead of distributing the (mock activity, if not mock
framework...) process of putting mock rmapps into the collection throughout the
test
-re 4 - interesting, those were already there, but I also couldn't see why.
Test passes fine without them, so I removed them
-re 5 - removed
uploading updated patch in a few
> maximum-am-resource-percent could be respected for both LeafQueue/User when
> trying to activate applications.
> ------------------------------------------------------------------------------------------------------------
>
> Key: YARN-2637
> URL: https://issues.apache.org/jira/browse/YARN-2637
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 2.6.0
> Reporter: Wangda Tan
> Assignee: Craig Welch
> Priority: Critical
> Attachments: YARN-2637.0.patch, YARN-2637.1.patch,
> YARN-2637.12.patch, YARN-2637.13.patch, YARN-2637.15.patch,
> YARN-2637.16.patch, YARN-2637.17.patch, YARN-2637.18.patch,
> YARN-2637.19.patch, YARN-2637.2.patch, YARN-2637.20.patch,
> YARN-2637.21.patch, YARN-2637.22.patch, YARN-2637.23.patch,
> YARN-2637.25.patch, YARN-2637.26.patch, YARN-2637.27.patch,
> YARN-2637.28.patch, YARN-2637.29.patch, YARN-2637.30.patch,
> YARN-2637.31.patch, YARN-2637.32.patch, YARN-2637.36.patch,
> YARN-2637.38.patch, YARN-2637.39.patch, YARN-2637.6.patch, YARN-2637.7.patch,
> YARN-2637.9.patch
>
>
> Currently, number of AM in leaf queue will be calculated in following way:
> {code}
> max_am_resource = queue_max_capacity * maximum_am_resource_percent
> #max_am_number = max_am_resource / minimum_allocation
> #max_am_number_for_each_user = #max_am_number * userlimit * userlimit_factor
> {code}
> And when submit new application to RM, it will check if an app can be
> activated in following way:
> {code}
> for (Iterator<FiCaSchedulerApp> i=pendingApplications.iterator();
> i.hasNext(); ) {
> FiCaSchedulerApp application = i.next();
>
> // Check queue limit
> if (getNumActiveApplications() >= getMaximumActiveApplications()) {
> break;
> }
>
> // Check user limit
> User user = getUser(application.getUser());
> if (user.getActiveApplications() <
> getMaximumActiveApplicationsPerUser()) {
> user.activateApplication();
> activeApplications.add(application);
> i.remove();
> LOG.info("Application " + application.getApplicationId() +
> " from user: " + application.getUser() +
> " activated in queue: " + getQueueName());
> }
> }
> {code}
> An example is,
> If a queue has capacity = 1G, max_am_resource_percent = 0.2, the maximum
> resource that AM can use is 200M, assuming minimum_allocation=1M, #am can be
> launched is 200, and if user uses 5M for each AM (> minimum_allocation). All
> apps can still be activated, and it will occupy all resource of a queue
> instead of only a max_am_resource_percent of a queue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
