[ https://issues.apache.org/jira/browse/YARN-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15576235#comment-15576235 ]
Eric Payne commented on YARN-2009: ---------------------------------- [~sunilg], I have one concern about how {{TempAppPerPartition#toBePreempted}} is calculated. Please consider the use case of where one low priority app is taking up all resources and a second high priority app is trying to get started: _NOTE; I am ignoring {{userLimitResource}} and {{idealAssignedForUser}} because in this use case there is only 1 user. Ignoring {{app.selected}} as well._ {code:title=FifoIntraQueuePreemptionPlugin simplified pseudo code} calculateIdealAssignedResourcePerApp: for each app { app.idealAssigned = min((app.totalUsed + app.pending), queueTotalUnassigned) } ... calculateToBePreemptedResourcePerApp: for each app { app.toBePreempted = app.totalUsed - app.idealAssigned - app.AMUsed } {code} For this use case, let's consider that all containers, including the AMs, will be 512 MB. {{app1}} is running, consuming all resources in the queue. The RM creates {{app2}} and sets its pending value to be 512 MB (for the AM). Intra-queue preemption does the following during each round: ||AppID||priority||totalUsed MB||AMUsed MB||pending MB||queueTotalUnassigned MB|| |2|2|0|0|512|12288| {noformat:title=FifoIntraQueuePreemptionPlugin#calculateIdealAssignedResourcePerApp} app2.idealAssigned = min((app2.totalUsed + app2.pending), queueTotalUnassigned) = min((0 + 512), 12288) = 512 queueTotalUnassigned = queueTotalUnassigned - app2.idealAssigned = 12288 - 512 = 11776 {noformat} ||AppID||priority||totalUsed MB||AMUsed MB||pending MB||queueTotalUnassigned MB|| |1|1|12288|512|3584|11776| {noformat:title=FifoIntraQueuePreemptionPlugin#calculateIdealAssignedResourcePerApp} app1.idealAssigned = min((app1.totalUsed + app1.pending), queueTotalUnassigned) = min((12288 + 3584), 11776) = 11776 queueTotalUnassigned = queueTotalUnassigned - app2.idealAssigned = 11776 - 11776 = 0 {noformat} {noformat:title=FifoIntraQueuePreemptionPlugin#calculateToBePreemptedResourcePerApp} app1.toBePreempted = app1.totalUsed - app1.idealAssigned - app1.AMUsed = 12288 - 11776 - 512 = 0 {noformat} In this case, {{app1.toBePreempted}} should be 512 instead of 0. Since it remains 0, no container ever gets preempted and {{app2}} can never get it's AM container. I'm not yet ready to suggest a solution, but I wanted to point this out while we work on it. > Priority support for preemption in ProportionalCapacityPreemptionPolicy > ----------------------------------------------------------------------- > > Key: YARN-2009 > URL: https://issues.apache.org/jira/browse/YARN-2009 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler > Reporter: Devaraj K > Assignee: Sunil G > Attachments: YARN-2009.0001.patch, YARN-2009.0002.patch, > YARN-2009.0003.patch, YARN-2009.0004.patch, YARN-2009.0005.patch, > YARN-2009.0006.patch, YARN-2009.0007.patch > > > While preempting containers based on the queue ideal assignment, we may need > to consider preempting the low priority application containers first. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org