[
https://issues.apache.org/jira/browse/YARN-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15924727#comment-15924727
]
Wangda Tan commented on YARN-5892:
----------------------------------
Thanks [~eepayne] for updating the patch, haven't looked into details of the
patch, a couple of high level questions/comments:
1) Can this patch properly handle user limit updates when queue refreshes? I
couldn't find related changes.
2) In the existing patch it still changes user limit, and do normalize of user
limit, calculate local user limit, etc. I'm not sure if following logic
suggested can simplify the code:
Since we have weights of users, scheduler can think number of active users (for
active user limit) and number of total users (for total user limit) will be
updated considering weights.
For example, we have 3 users, u1 (weight=2.5), u2 (weight=1), u3 (weight=3).
And u1/u2 are active users. Can we say there're 3.5 active users and 6.5 total
users? With this we may not need to normalize user limit during computation.
3) Can this patch properly handle cached per-user user-limit?
Logics of YARN-5889 assumes user limit is identical across users, so it only
needs to store pre-computed resource/version for partition/scheduling-mode.
This patch may need to change this logic since different user can have
different user-limit resource.
Another option is to cache the "unit user"'s user-limit, with that we don't
have to cache user-limit for different users. Instead, we have to multiply
weight of user by computed user limit for "unit user" (when weight != 1).
> Capacity Scheduler: Support user-specific minimum user limit percent
> --------------------------------------------------------------------
>
> Key: YARN-5892
> URL: https://issues.apache.org/jira/browse/YARN-5892
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
> Reporter: Eric Payne
> Assignee: Eric Payne
> Attachments: Active users highlighted.jpg, YARN-5892.001.patch,
> YARN-5892.002.patch, YARN-5892.003.patch
>
>
> Currently, in the capacity scheduler, the {{minimum-user-limit-percent}}
> property is per queue. A cluster admin should be able to set the minimum user
> limit percent on a per-user basis within the queue.
> This functionality is needed so that when intra-queue preemption is enabled
> (YARN-4945 / YARN-2113), some users can be deemed as more important than
> other users, and resources from VIP users won't be as likely to be preempted.
> For example, if the {{getstuffdone}} queue has a MULP of 25 percent, but user
> {{jane}} is a power user of queue {{getstuffdone}} and needs to be guaranteed
> 75 percent, the properties for {{getstuffdone}} and {{jane}} would look like
> this:
> {code}
> <property>
>
> <name>yarn.scheduler.capacity.root.getstuffdone.minimum-user-limit-percent</name>
> <value>25</value>
> </property>
> <property>
>
> <name>yarn.scheduler.capacity.root.getstuffdone.jane.minimum-user-limit-percent</name>
> <value>75</value>
> </property>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
