[
https://issues.apache.org/jira/browse/YARN-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439264#comment-15439264
]
Sunil G commented on YARN-4945:
-------------------------------
Thanks [~eepayne] for the use cases.
Overall I feel the use cases looks fine for me,. I have looked the doc as well.
Few comments from my end
1.
{noformat}
When one (or more) user(s) are below their minimun-user-limit-percent and one
(or
more) user(s) are above their minimum-user-limit-percent, resources will be
preempted after a configurable time period from the user(s) which are above
their minimumuser-limit-percent.
{noformat}
>>I think we might need to come with a limit on how much resource can be
>>preempted from over-utilizing users's apps. WE do have
>>max-preemption-per-round. But sometimes it may be more as it may be
>>configured for inter-queue. Since we are sharing this config, i think we can
>>have a config to limit the preemption for user-limit. For priority, i have
>>considered a certain limit to control this scenario. Thoughts?
2.
{noformat}
App0 in QUEUE0 is consuming 50 extra resources and is not releasing them. But,
since QUEUE0 is not preemptable, inter-queue preemption will not free the 50
resources from QUEUE0. In QUEUE1, User2 is guaranteed 50 resources, but
User1 is also, and User1 is not going over its minimum-user-limitpercent,
so User1 is not violating any policy. Therefore, the
user-limit-percentintra-queue-preemption
policy does not preempt any resources from User1.
{noformat}
>> Overall this point make sense. This is strictly intra-queue model and we
>> need not have to consider across queue demand and its shortages.
3.
{noformat}
1.3.1.3.1. TBD: I could see this going the other way. That is, we may want the
policy to preempt from App1 until both users have an equal number of
resources.
{noformat}
>> I mostly agree with you here. Once app1's containers are released, app2 can
>> get resources. CS will ensure same, I dont think preemption module need to
>> do that.
4.
bq.Should there be some (possibly configurable) limit on the percent of
containers preempted from App1?
Instead of limiting from each app, I consider only a certain % of high priority
app's demand. This can help to avoid over preemption cases. If we limit
preempting container from an app, a higher priority app may loose some
container. So for now, its better to clear lowest priority apps completely
except its AM. On same line, apps could save some of its containers (critical,
long running). For this, app itself can mark such containers. This will come as
part of first class services, and we could improve preemption after same.
5.
bq.the priority preemption policy will not preempt containers from the lower
priority app if it would cause the lower priority app to go below the user’s
minimum-user-limit-percent guarantee.
I will not consider preemption demand from a high priority if that app is
already crossing the user-limit. Such apps will be skipped while calculating
demand. I have added this support in POC patch itself.
Overall these cases make sense, and thank you very much or sharing same. Its
very good to validate and consider all problem statements at this stage itself.
:)
I have not looked your latest comment, i will help to comment on same in a
shortwhile.
> [Umbrella] Capacity Scheduler Preemption Within a queue
> -------------------------------------------------------
>
> Key: YARN-4945
> URL: https://issues.apache.org/jira/browse/YARN-4945
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Wangda Tan
> Attachments: Intra-Queue Preemption Use Cases.pdf,
> IntraQueuepreemption-CapacityScheduler (Design).pdf, YARN-2009-wip.patch
>
>
> This is umbrella ticket to track efforts of preemption within a queue to
> support features like:
> YARN-2009. YARN-2113. YARN-4781.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
