[
https://issues.apache.org/jira/browse/YARN-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276404#comment-17276404
]
Eric Payne commented on YARN-10559:
-----------------------------------
[~ananyo_rao], sorry for the delay. I'm still reviewing the changes, but I have
one concern with the requirements.
{code:title=FifoIntraQueuePreemptionPlugin#setFairShareForApps}
+ * we firstly ensure all the apps in the queue get equal resources.
{code}
I don't think this is exactly correct. When a queue in the Capacity Scheduler
has FairOrderingPolicy set, it will grow each user's share of the resources at
a fair pace. If user1 has app1 and user2 has app2 and app3, and if all 3 apps
are requesting resources, app1 will receive resources faster than app2. app2
and app3 together will receive resources at roughly the same rate as app1.
The total of resources assigned to user1 and user2 will grow at roughly the
same amount, but the apps themselves will not receive resources at the same
rate.
So, when we preempt, we want to mimic that same behavior. I'm still trying to
fully understand the code, so it may be that the code actually does what I
said, but at the very least, the statement is misleading and I want to make
sure we are on the same page regarding the requirements.
> Fair sharing intra-queue preemption support in Capacity Scheduler
> -----------------------------------------------------------------
>
> Key: YARN-10559
> URL: https://issues.apache.org/jira/browse/YARN-10559
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
> Affects Versions: 3.1.4
> Reporter: VADAGA ANANYO RAO
> Assignee: VADAGA ANANYO RAO
> Priority: Major
> Attachments: FairOP_preemption-design_doc_v1.pdf,
> FairOP_preemption-design_doc_v2.pdf, YARN-10559.0001.patch,
> YARN-10559.0002.patch, YARN-10559.0003.patch, YARN-10559.0004.patch,
> YARN-10559.0005.patch, YARN-10559.0006.patch, YARN-10559.0007.patch,
> YARN-10559.0008.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Usecase:
> Due to the way Capacity Scheduler preemption works, If a single user submits
> a large application to a queue (using 100% of resources), that job will not
> be preempted by future applications from the same user within the same queue.
> This implies that the later applications will be forced to wait for
> completion of the long running application. This prevents multiple long
> running, large, applications from running concurrently.
> Support fair sharing among apps while preempting applications from same queue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
