[
https://issues.apache.org/jira/browse/YARN-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14390455#comment-14390455
]
Peng Zhang commented on YARN-3405:
----------------------------------
I've a primitive idea to fix this and YARN-3414 under current preemption
architecture.
1. When calculation preemption request, update parent's preemption request.
2. if parent's usage reached its fair share, it will not propagate preemption
request upside again. So preemption request in parent queue means preemption
needed between its children.
3. During preempting phase, walk from root to downside
a. if parent queue has preemption request, it will do preemption between its
children for the request(process like now, find the most over fair, and preempt
recursively).
b. And then(including after doing 3.a and the case not need preempt between
children), traverse its children and repeat 3.a;
This process bring in traverse of the tree. And I think this will not affect
performance severely because there are usually small amount of queues.
> FairScheduler's preemption cannot happen between sibling in some case
> ---------------------------------------------------------------------
>
> Key: YARN-3405
> URL: https://issues.apache.org/jira/browse/YARN-3405
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
> Affects Versions: 2.7.0
> Reporter: Peng Zhang
> Assignee: Peng Zhang
> Priority: Critical
>
> Queue hierarchy described as below:
> {noformat}
> root
> / \
> queue-1 queue-2
> / \
> queue-1-1 queue-1-2
> {noformat}
> Assume cluster resource is 100
> # queue-1-1 and queue-2 has app. Each get 50 usage and 50 fairshare.
> # When queue-1-2 is active, and it cause some new preemption request for
> fairshare 25.
> # When preemption from root, it has possibility to find preemption candidate
> is queue-2. If so preemptContainerPreCheck for queue-2 return false because
> it's equal to its fairshare.
> # Finally queue-1-2 will be waiting for resource release form queue-1-1
> itself.
> What I expect here is that queue-1-2 preempt from queue-1-1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
