[
https://issues.apache.org/jira/browse/YARN-8379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504630#comment-16504630
]
Eric Payne commented on YARN-8379:
----------------------------------
{quote}
Why is this a preemption only concept? To avoid unnecessary thrash between the
allocation doing one thing and the preemption doing another, we should also
have a corresponding queue ordering-policy, right?
{quote}
This is only related to preemption because the capacity scheduler already
balances if resources become available. However, currently, if preemption is
enabled on all queues, preemption will stop freeing resources once all pending
queues are over their queue capacity.
The example in this JIRA's description outlines the current behavior. In that
example, if resources free up naturally from queue_b, the capacity scheduler
will assign them to queue_a. However, the preemption monitor will not preempt
them because queue_a is at its 30% queue capacity.
In 2.8 and prior releases, the preemption monitor does balance. As pointed out
above, the balancing behavior was changed as part of YARN-5864
> Add an option to allow Capacity Scheduler preemption to balance satisfied
> queues
> --------------------------------------------------------------------------------
>
> Key: YARN-8379
> URL: https://issues.apache.org/jira/browse/YARN-8379
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Wangda Tan
> Assignee: Zian Chen
> Priority: Major
> Attachments: YARN-8379.001.patch, YARN-8379.002.patch
>
>
> Existing capacity scheduler only supports preemption for an underutilized
> queue to reach its guaranteed resource. In addition to that, there’s an
> requirement to get better balance between queues when all of them reach
> guaranteed resource but with different fairness resource.
> An example is, 3 queues with capacity, queue_a = 30%, queue_b = 30%, queue_c
> = 40%. At time T. queue_a is using 30%, queue_b is using 70%. Existing
> scheduler preemption won't happen. But this is unfair to queue_a since
> queue_a has the same guaranteed resources.
> Before YARN-5864, capacity scheduler do additional preemption to balance
> queues. We changed the logic since it could preempt too many containers
> between queues when all queues are satisfied.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
