[ https://issues.apache.org/jira/browse/YARN-4333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15132002#comment-15132002 ]
Tao Jie commented on YARN-4333: ------------------------------- In updated patch,some improvements for preemption within queue: 1, When calculating resource deficit to preempt, we have {{resToPreempt}} which is total resource to preempt among queues, and {{Map<String, Resource> withinQueueResToPreempt}} which contains amount of resource to preempt within queues. 2, In preemptResource, we should consider both {{resToPreempt}} and {{withinQueueResToPreempt}}. And we will preempt resource due to {{withinQueueResToPreempt}}(if any) before {{resToPreempt}}. 3, Once we preempt container due to preemption within queue, we will add resouce to {{Map<String, Resource> resourceDeficitWithinQueues}}, which maintains resource preempted due to within-queue preemption for each queue. When scheduling resource, we will prefer queues those have resource deficit in resourceDeficitWithinQueues. > Fair scheduler should support preemption within queue > ----------------------------------------------------- > > Key: YARN-4333 > URL: https://issues.apache.org/jira/browse/YARN-4333 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler > Affects Versions: 2.6.0 > Reporter: Tao Jie > Assignee: Tao Jie > Attachments: YARN-4333.001.patch, YARN-4333.002.patch > > > Now each app in fair scheduler is allocated its fairshare, however fairshare > resource is not ensured even if fairSharePreemption is enabled. > Consider: > 1, When the cluster is idle, we submit app1 to queueA,which takes maxResource > of queueA. > 2, Then the cluster becomes busy, but app1 does not release any resource, > queueA resource usage is over its fairshare > 3, Then we submit app2(maybe with higher priority) to queueA. Now app2 has > its own fairshare, but could not obtain any resource, since queueA is still > over its fairshare and resource will not assign to queueA anymore. Also, > preemption is not triggered in this case. > So we should allow preemption within queue, when app is starved for fairshare. -- This message was sent by Atlassian JIRA (v6.3.4#6332)