[
https://issues.apache.org/jira/browse/YARN-8379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511522#comment-16511522
]
Zian Chen commented on YARN-8379:
---------------------------------
Thank you [~eepayne] for your suggestions. I'm totally agreed for the concerns
of misleading by naming this feature using " fair" or "fairness". For your
concerns, please check my questions and comments below,
1. Which patch were you used for the cross queue preemption test? Patch 003?
Could you share your queue settings (like queue hierarchy, each queue's
guarantee, used and pending resources, queue priority) so that I can reproduce
the case and see what the issue is?
2. Will address failed UTs, find bugs, etc.
3. The reason we add {{FifoCandidatesSelector}} to candidatesSelectionPolicies
twice is that we want to make conservative preemption when we do the balance.
The balance preemption (which is FifoCandidatesSelector with
allowQueuesBalanceAfterAllQueuesSatisfied set to true) will only happen when
all queues are meet or beyond its guarantee if any of the queues is
underutilized, this balance will not be triggered. The reason we implement like
this is we don't want balance and preempt queue to satisfy its guarantee not
conflict with each other and preempt containers back and forth.
4. The reason for this code is explained in item 3.
{code:java|title=FifoCandidatesSelector#selectCandidates}
for (Set<RMContainer> containers : selectedCandidates.values()) {
if (!containers.isEmpty()) {
if (LOG.isDebugEnabled()) {
LOG.debug(...);
}
}
return selectedCandidates;
}
{code}
5. Agree for the log statement.
6. Agree to remove "Fair" or "Fairness"
7. Agree
[~leftnoteasy] , [~cheersyang] , any concerns or suggestions on the latest
patch? Thanks!
> Add an option to allow Capacity Scheduler preemption to balance satisfied
> queues
> --------------------------------------------------------------------------------
>
> Key: YARN-8379
> URL: https://issues.apache.org/jira/browse/YARN-8379
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Wangda Tan
> Assignee: Zian Chen
> Priority: Major
> Attachments: YARN-8379.001.patch, YARN-8379.002.patch,
> YARN-8379.003.patch
>
>
> Existing capacity scheduler only supports preemption for an underutilized
> queue to reach its guaranteed resource. In addition to that, there’s an
> requirement to get better balance between queues when all of them reach
> guaranteed resource but with different fairness resource.
> An example is, 3 queues with capacity, queue_a = 30%, queue_b = 30%, queue_c
> = 40%. At time T. queue_a is using 30%, queue_b is using 70%. Existing
> scheduler preemption won't happen. But this is unfair to queue_a since
> queue_a has the same guaranteed resources.
> Before YARN-5864, capacity scheduler do additional preemption to balance
> queues. We changed the logic since it could preempt too many containers
> between queues when all queues are satisfied.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
