[ https://issues.apache.org/jira/browse/YARN-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951222#comment-15951222 ]
Eric Payne commented on YARN-2113: ---------------------------------- Thanks [~sunilg] for the updated patch. It looks good in gneral, but I still have one concern. With the latest patch, there is still a use case in which containers from under-served users can be preempted. Please consider the following: - Queue total resources: 100 - Minimum user limit percent: 50% || *APP* || *USER* || *PRIORITY* || *USED* || *PENDING* || | app1 | user1 | 1 | 50 | 0 | | app2 | user2 | 1 | 50 | 0 | | app3 | user3 | 5 | 0 | 10 | In words, the MULP is 50%, so both {{user1}} and {{user2}} are at their user limit. When {{user3}} comes along with an app at a higher priority, the intra-queue preemption monitor should not preempt from either {{user1}} or {{user2}}. But it does. In fact, it will preempt as much as needed until {{user3}} reaches its MULP guarantee. This could allow users to use priorities to abuse the user limit guarantees. This does not happen when the priority of {{app3}} is the same or below that of the other users' apps. I am uploading a temporary test patch that shows this. It has 3 new tests. The difference between them is only the priority of {{app3}} from {{user3}}. The one that fails is {{testNoIntraQueuePreemptionIfBelowUserLimitAndHighPriorityExtraUsers}}. The other two work as expected. I had to also modify {{ProportionalCapacityPreemptionPolicyMockFramework.java}} to take an MULP as a paramter in the {{appsConfig}} rather than allow {{mockApplications()}} to calculate it. The file {{TestNoIntraQueuePreemptionIfBelowUserLimitAndDifferentPrioritiesWithExtraUsers.txt}} should be applied on top of {{YARN-2113.0004.patch}}. I also have a few other small comments about the code that I will upload separately. > Add cross-user preemption within CapacityScheduler's leaf-queue > --------------------------------------------------------------- > > Key: YARN-2113 > URL: https://issues.apache.org/jira/browse/YARN-2113 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler > Reporter: Vinod Kumar Vavilapalli > Assignee: Sunil G > Attachments: YARN-2113.0001.patch, YARN-2113.0002.patch, > YARN-2113.0003.patch, YARN-2113.0004.patch, YARN-2113.v0.patch > > > Preemption today only works across queues and moves around resources across > queues per demand and usage. We should also have user-level preemption within > a queue, to balance capacity across users in a predictable manner. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org