[
https://issues.apache.org/jira/browse/YARN-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14570039#comment-14570039
]
Craig Welch commented on YARN-3510:
-----------------------------------
[~leftnoteasy] and I had some offline discussion. The patch currently here is
simply meant to keep from unbalancing whatever allocation process is active by,
generally, keeping relative usage between applications the same. It doesn't
attempt to actively re-allocate in a way which achieves the overall allocation
policy, i.e., "as if all the applications had started at once". (this is a
more complex proposition, obviously). There's a desire to have this because,
among other things, sometime down the road we may do preemption just among
users/applications in a queue and it will be necessary for the preemption to
actively work toward the allocation goals to do that, rather than just maintain
current levels. This will add some medium level complexity to the current
patch, deltas with the current approach are:
Since the effect of preemption on order for fairness doesn't occur until the
container is released, and we want to consider it right away, there will need
to be a need to retain info about "pending preemption" for comparison on the
app resources (it will be a deduction from usage for ordering purposes, as if
the preemption had already happened)
The preemptEvenly loop will need to reorder the app which was preempted after
each preemption and then restart the iteration over apps (not necessarily over
all apps, again, just until the first preemption)
> Create an extension of ProportionalCapacityPreemptionPolicy which preempts a
> number of containers from each application in a way which respects fairness
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-3510
> URL: https://issues.apache.org/jira/browse/YARN-3510
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
> Reporter: Craig Welch
> Assignee: Craig Welch
> Attachments: YARN-3510.2.patch, YARN-3510.3.patch, YARN-3510.5.patch,
> YARN-3510.6.patch
>
>
> The ProportionalCapacityPreemptionPolicy preempts as many containers from
> applications as it can during it's preemption run. For fifo this makes
> sense, as it is prempting in reverse order & therefore maintaining the
> primacy of the "oldest". For fair ordering this does not have the desired
> effect - instead, it should preempt a number of containers from each
> application which maintains a fair balance /close to a fair balance between
> them
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
