[
https://issues.apache.org/jira/browse/YARN-4902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15239662#comment-15239662
]
Arun Suresh commented on YARN-4902:
-----------------------------------
Does it make sense to introduce the concept of *AffinityGroups* /
*Anti-AffinityGroups* for the purpose of simplifying affinity/anti-affinity ?
Essentially, just like we use the Reservation API to request a reservation Id,
and then include the Id in subsequent requests,
we could expose an API that allows one to request an *(Anti)AffinityGroups*
(consisting of N machine / totally X resources etc.) which inturn returns a
*groupId*. (This could internally would dynamically tag / label a group of
Machines/Nodes with the id, but I guess that is an implementation detail.)
Subsequent allocation requests can include this *groupId* and based on the
policy, The scheduler will try to schedule on the same group of machines or
different machines.
Another though is, an affinity group may be an Allocation that can be
used/shared by multiple applications. Once we de-link Allocations from
containers, It may be possible to have an uber-application (or maybe a
component in the RM itself) lease Allocations only and grant the right to start
containers against these allocations to other applications. A group of
allocations across multiple machines can constitute an anti-affinity group, and
we can introduce policies that allow subsequent apps to, say, start only 1
container of each role/type on 1 allocation within a group for eg.
I feel that allowing users to organize deployments along the lines of
affinity/anti-affinity groups (failure domains might also be something similar)
is more manageable than allowing users to specify affinity and anti-affinity
constraints with respect to an already deployed application.
> [Umbrella] Generalized and unified scheduling-strategies in YARN
> ----------------------------------------------------------------
>
> Key: YARN-4902
> URL: https://issues.apache.org/jira/browse/YARN-4902
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Wangda Tan
> Attachments: Generalized and unified scheduling-strategies in YARN
> -v0.pdf
>
>
> Apache Hadoop YARN's ResourceRequest mechanism is the core part of the YARN's
> scheduling API for applications to use. The ResourceRequest mechanism is a
> powerful API for applications (specifically ApplicationMasters) to indicate
> to YARN what size of containers are needed, and where in the cluster etc.
> However a host of new feature requirements are making the API increasingly
> more and more complex and difficult to understand by users and making it very
> complicated to implement within the code-base.
> This JIRA aims to generalize and unify all such scheduling-strategies in YARN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
