[jira] [Commented] (YARN-4945) [Umbrella] Capacity Scheduler Preemption Within a queue

[Wangda Tan (JIRA)]

    [ 
https://issues.apache.org/jira/browse/YARN-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15453773#comment-15453773
 ] 

Wangda Tan commented on YARN-4945:
----------------------------------

Hi Sunil,

Thanks for update, I haven't dig into too many details of this patch yet, I 
majorly looked at interactions between preemptable-resource-calculator and 
candidate-selector.

Some comments so far:

1) What is intraQueuePreemptionCost? And what is AppPriorityComparator?

2) IntraQueuePreemptableResourceCalculator:
2.1) For the queue-level ideal allocation, is it enough to trust result from 
previous policies? PreemptableResourceCalculator saves per-queue ideal 
allocation to PCPP#queueToPartitions. I think we don't need add extra logic in 
the IntraQueuePreemptableResourceCalculator to resursively calculate it. 
Correct?

2.2 computeIntraQueuePreemptionDemand:
- {{if (tq.intraQueuePreemptionCalculationDone == true)}}, is this always false?
- Should we lock the queue inside {{for (LeafQueue leafQUeue : queues)}}?
- return value of getResourceDemandFromAppsPerQueue is not consumed by anyone.
- TempAppPerQueue is not used by anyone

Since the logic looks incompleted yet, here's some thoughts about 
implementation/overall-code-structure from my side, hope it may help you.

{code}
IntraQueuePreemptableResourceCalculator {
        1. Uses ideal resource calculated by previous policies. 
        2. Compute per-app ideal/preemptable resource according to per-queue 
policies. (Stored in TempAppPerPartition)
}

IntraQueueCandidateSelector {
        1. Invoke IntraQueueCandidateSelector to calculate ideal/preemptable of 
apps
        2. Use 
CapacitySchedulerPreemptionUtils.deductPreemptableResourcesBasedSelectedCandidates
 to deduct preemptable resource for already selected containers.

        for (leafqueue from most underserved) {
                for (apps in reserve order) {
                        if (app.preemptable > 0) {
                                // Select container logic.
                        }
                }
        }
}
{code}

> [Umbrella] Capacity Scheduler Preemption Within a queue
> -------------------------------------------------------
>
>                 Key: YARN-4945
>                 URL: https://issues.apache.org/jira/browse/YARN-4945
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Wangda Tan
>         Attachments: Intra-Queue Preemption Use Cases.pdf, 
> IntraQueuepreemption-CapacityScheduler (Design).pdf, YARN-2009-wip.2.patch, 
> YARN-2009-wip.patch
>
>
> This is umbrella ticket to track efforts of preemption within a queue to 
> support features like:
> YARN-2009. YARN-2113. YARN-4781.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org