[
https://issues.apache.org/jira/browse/YARN-7619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289346#comment-16289346
]
Eric Payne commented on YARN-7619:
----------------------------------
[~sunilg]. Great comments. Thanks for the review.
{quote}
bq. I would say it would be best to save the recomputed user AM limit to all
users objects whenever LeafQueue#getUserAMResourceLimitPerPartition is called,
but that may cause a significant performance hit.
This seems better as per me.
{quote}
Although this could possibly be cleaner, I am reluctant to add possible
performance hits. We have some very busy queues with dozens and dozens of
concurrent users. Also, I don't want to change the LeafQueue behavior any more
than necessary.
bq. 1. I think we can avoid this storage in LeafQueue. We could add a new item
in ResourceUsage class.
An excellent point. This makes the design much cleaner. The pre-weighted user
AM limit per partition is a queue-level value that is applied to all users in
the queue, so this new value would be set in the
{{AbstractCSQueue#queueuUsage}}. Does that make sense to you?
bq. 2. could we use Resources.min instead of ternary operator?
Yes, that would make the code more efficient. Very helpful.
bq. 3. In UserInfo, we could avoid amLimitPerUserPerPartition if comment #1 is
accepted.
Agreed
bq. 4. But if user has a low user-weight and if we give latest value, how far
off it could be in worst case?
This is actually the current behavior. Yes, it can be wrong by quite a lot, but
this is just a "fall-back" in case the setting is not there for some reason.
> Max AM Resource value in CS UI is different for every user
> ----------------------------------------------------------
>
> Key: YARN-7619
> URL: https://issues.apache.org/jira/browse/YARN-7619
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, yarn
> Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2, 3.1.0
> Reporter: Eric Payne
> Assignee: Eric Payne
> Attachments: Max AM Resources is Different for Each User.png,
> YARN-7619.001.patch
>
>
> YARN-7245 addressed the problem that the {{Max AM Resource}} in the capacity
> scheduler UI used to contain the queue-level AM limit instead of the
> user-level AM limit. It fixed this by using the user-specific AM limit that
> is calculated in {{LeafQueue#activateApplications}}, stored in each user's
> {{LeafQueue#User}} object, and retrieved via
> {{UserInfo#getResourceUsageInfo}}.
> The problem is that this user-specific AM limit depends on the activity of
> other users and other applications in a queue, and it is only calculated and
> updated when a user's application is activated. So, when
> {{CapacitySchedulerPage}} retrieves the user-specific AM limit, it is a stale
> value unless an application was recently activated for a particular user.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
