[
https://issues.apache.org/jira/browse/YARN-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044535#comment-14044535
]
Sunil G commented on YARN-2022:
-------------------------------
Hi [~mayank_bansal] and [~leftnoteasy]
I was checking few scenarios with these two options.
As Wangda mentioned, getAbsoluteMaximumCapacity is used for calculating total
#application within a queue. With current preemption mechanism, each
application will be completely preempted and hence there arises no scenario
that AMs will be occupy a big part of cluster. But with saving AM feature w.r.t
maxAMCapacity (and *getAbsoluteMaximumCapacity()*), I feel there can be a
scenario where major portion of cluster going for AMs.
A scenario with QueueA (with 20% capacity and 100% maximum), QueueB (with 80%
capacity and 100% maximum) and 0.5 as AM percentage. Assume 100% cluster is
taken over by QueueA jobs with 50% of AMs (different users). When demand is
there from QueueB, QueueA will end up with 50% of AMs (saving AMs w.r.t
maxAMCapacity). Now QueueB will start running jobs and it can also take 25% of
AMs. So there are now more chances of having more AMs in cluster compared to
earlier design.
May be I feel we can use *getAbsoluteCapacity* for now (seems more safer) and
as Wangda mentioned we can discuss in another JIRA about the other corner cases
w.r.t preemption and CapacityScheduler configurations. Please suggest your
thoughts w.r.t this scenario.
> Preempting an Application Master container can be kept as least priority when
> multiple applications are marked for preemption by
> ProportionalCapacityPreemptionPolicy
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-2022
> URL: https://issues.apache.org/jira/browse/YARN-2022
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
> Affects Versions: 2.4.0
> Reporter: Sunil G
> Assignee: Sunil G
> Attachments: YARN-2022-DesignDraft.docx, YARN-2022.2.patch,
> YARN-2022.3.patch, YARN-2022.4.patch, YARN-2022.5.patch, YARN-2022.6.patch,
> YARN-2022.7.patch, Yarn-2022.1.patch
>
>
> Cluster Size = 16GB [2NM's]
> Queue A Capacity = 50%
> Queue B Capacity = 50%
> Consider there are 3 applications running in Queue A which has taken the full
> cluster capacity.
> J1 = 2GB AM + 1GB * 4 Maps
> J2 = 2GB AM + 1GB * 4 Maps
> J3 = 2GB AM + 1GB * 2 Maps
> Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ].
> Currently in this scenario, Jobs J3 will get killed including its AM.
> It is better if AM can be given least priority among multiple applications.
> In this same scenario, map tasks from J3 and J2 can be preempted.
> Later when cluster is free, maps can be allocated to these Jobs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
