[ 
https://issues.apache.org/jira/browse/YARN-8379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16524608#comment-16524608
 ] 

Zian Chen edited comment on YARN-8379 at 6/27/18 5:52 AM:
----------------------------------------------------------

Hi [~leftnoteasy]. thanks for the comments.
 # This test case was intend to demonstrate selected candidates will be 
actually killed after custom timeout was reached. This part of code is the 
intention.
{code:java}
editPolicy.editSchedule();
Assert.assertEquals(4, editPolicy.getToPreemptContainers().size());

// check live containers immediately, nothing happen
Assert.assertEquals(39, schedulerApp1.getLiveContainers().size());

Thread.sleep(20*1000);
// Call editSchedule again: selected containers are killed
editPolicy.editSchedule();
waitNumberOfLiveContainersFromApp(schedulerApp1, 35);{code}

 # Totally understand your suggestion here, actually I was define a private 
member variable inside interface PreemptionCandidatesSelector so that we don't 
need to pass curCandidates for every selector instance. However I found 
TestCapacitySchedulerSurgicalPreemption#testPriorityPreemptionFromHighestPriorityQueueAndOldestContainer()
 get failed which helped me realize that curCandidates is a "per round 
reinitialize thing" rather than a "per selector" thing, we should give a clean 
curCandidates HashMap every time we call editSchedule, otherwise like this UT, 
we call editschedule multiple times but the selector remain the same instance, 
then curCandidates will contains old selected containers the second time we 
call the editSchedule, we could definitely make a method inside 
PreemptionCandidatesSelector, and call it explicitly to reset curCandidates per 
round, but this way it makes the code even harder to read. Any better 
suggestions here?
 # Your suggestion is not updating toPreempt on the fly but only update 
curCandidates, then after each round, update toPreempt right after 
selectCandidates is finished? If this is the correct understanding, I think 
from time complexity point of view, these two strategy should be same since we 
always need to go through each temp selected candidates and add it into 
toPreempt with duplicate check, but it helps with maintaining status of 
selected containers in one place(curCandidates) and avoid inconsistency. We can 
further discuss this part.

I'll address the asflicense issue with the next patch. Thanks!


was (Author: zian chen):
Hi [~leftnoteasy]. thanks for the comments.
 # This test case was intend to demonstrate selected candidates will be 
actually killed after custom timeout was reached. This part of code is the 
intention.
{code:java}
editPolicy.editSchedule();
Assert.assertEquals(4, editPolicy.getToPreemptContainers().size());

// check live containers immediately, nothing happen
Assert.assertEquals(39, schedulerApp1.getLiveContainers().size());

Thread.sleep(20*1000);
// Call editSchedule again: selected containers are killed
editPolicy.editSchedule();
waitNumberOfLiveContainersFromApp(schedulerApp1, 35);{code}

 # Totally understand your suggestion here, actually I was define a private 
member variable inside interface PreemptionCandidatesSelector so that we don't 
need to pass curCandidates for every selector instance. However I found 
TestCapacitySchedulerSurgicalPreemption#testPriorityPreemptionFromHighestPriorityQueueAndOldestContainer()
 get failed which helped me realize that curCandidates is a "per round 
reinitialize thing" rather than a "per selector" thing, we should give a clean 
curCandidates HashMap every time we call editSchedule, otherwise like this UT, 
we call editschedule multiple times but the selector remain the same instance, 
then curCandidates will contains old selected containers the second time we 
call the editSchedule, we could definitely make a method inside 
PreemptionCandidatesSelector, and call it explicitly to reset curCandidates per 
round, but this way it makes the code even harder to read. Any better 
suggestions here?
 # Your suggestion is not updating toPreempt on the fly but only update 
curCandidates, then after each round, update toPreempt right after 
selectCandidates is finished? If this is the correct understanding, I think 
from time complexity point of view, these two strategy should be same since we 
always need to go through each temp selected candidates and add it into 
toPreempt with duplicate check, but it helps with maintaining status of 
selected containers in one place(curCandidates) and avoid inconsistency. We can 
further discuss this part.

I'll address the asflicense issue with the next patch. Thanks!

 

 

 

> Add an option to allow Capacity Scheduler preemption to balance satisfied 
> queues
> --------------------------------------------------------------------------------
>
>                 Key: YARN-8379
>                 URL: https://issues.apache.org/jira/browse/YARN-8379
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Wangda Tan
>            Assignee: Zian Chen
>            Priority: Major
>         Attachments: YARN-8379.001.patch, YARN-8379.002.patch, 
> YARN-8379.003.patch, YARN-8379.004.patch, YARN-8379.005.patch, 
> ericpayne.confs.tgz
>
>
> Existing capacity scheduler only supports preemption for an underutilized 
> queue to reach its guaranteed resource. In addition to that, there’s an 
> requirement to get better balance between queues when all of them reach 
> guaranteed resource but with different fairness resource.
> An example is, 3 queues with capacity, queue_a = 30%, queue_b = 30%, queue_c 
> = 40%. At time T. queue_a is using 30%, queue_b is using 70%. Existing 
> scheduler preemption won't happen. But this is unfair to queue_a since 
> queue_a has the same guaranteed resources.
> Before YARN-5864, capacity scheduler do additional preemption to balance 
> queues. We changed the logic since it could preempt too many containers 
> between queues when all queues are satisfied.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to