[jira] [Commented] (YARN-3306) [Umbrella] Proposing per-queue Policy driven scheduling in YARN

[Craig Welch (JIRA)]

    [ 
https://issues.apache.org/jira/browse/YARN-3306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14542829#comment-14542829
 ] 

Craig Welch commented on YARN-3306:
-----------------------------------

bq. Integration points being narrow is probably more a reason to work on a 
branch

I don't see why - it seems to me to suggest the opposite, that we are able to 
achieve isolation of the functionality in the main codebase without risk

bq. The code looks quite isolated

Another reason we don't need to work in a branch

We're trying to approach this iteratively, building specific, narrow 
functionalities to completion and then making them available for use and 
feedback, this will be difficult if it's all isolated away in a branch.  The 
approach so far works well for that process - much better than doing all the 
work in isolation and then bringing a much larger change into the main codebase 
all at once.  

As far as I can tell, separating this out into a branch is a net negative- 
there is overhead to doing so and it runs contrary to the iterative approach 
we're trying to take, without providing any clear benefit.

> [Umbrella] Proposing per-queue Policy driven scheduling in YARN
> ---------------------------------------------------------------
>
>                 Key: YARN-3306
>                 URL: https://issues.apache.org/jira/browse/YARN-3306
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: scheduler
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Vinod Kumar Vavilapalli
>         Attachments: PerQueuePolicydrivenschedulinginYARN.pdf
>
>
> Scheduling layout in Apache Hadoop YARN today is very coarse grained. This 
> proposal aims at converting today’s rigid scheduling in YARN to a per­-queue 
> policy driven architecture.
> We propose the creation of a c​ommon policy framework​ and implement a​common 
> set of policies​ that administrators can pick and chose per queue
>  - Make scheduling policies configurable per queue
>  - Initially, we limit ourselves to a new type of scheduling policy that 
> determines the ordering of applications within the l​eaf ­queue
>  - In the near future, we will also pursue parent­ queue level policies and 
> potential algorithm reuse through a separate type of policies that control 
> resource limits per queue, user, application etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)