[
https://issues.apache.org/jira/browse/YARN-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13762779#comment-13762779
]
Junping Du commented on YARN-1042:
----------------------------------
Attach a patch to showcase above proposal. This is only a demo patch and
haven't included any unit tests so far.
I think there are several open questions here before we move on next step:
1. this affinity/anti-affinity rule is bi-direction or not? If task
A.affinity(B) is true then B.affinity(A) is always true or not? I guess it is
not as A may prefer a list of nodes which makes the relationship non-symmetric.
Also, that is how we can differ A prefer to live with B and C from A prefer to
live B or C. Isn't it?
2. which rule's priority is higher in case affinity rule against with
anti-affinity rule? In demo patch, affinity rule plays as higher priority but I
am not sure if this is true in real case. Do we want to make it configurable?
Or we just make sure rules updated later can override previous one if conflict.
3. Currently, the affinity/anti-affinity is only considered in node level, do
we want to expand it to other level i.e. rack level in future?
4. The API now is to add a list of taskId as affinity/anti-affinity tasks. Is
that easy to consume in application prospective?
5. the affinity/anti-affinity rules is a *must* conform rule in current
implementation which may cause task starve for longer time, do we think about
more leisure rule?
Welcome to comments. Thx!
> add ability to specify affinity/anti-affinity in container requests
> -------------------------------------------------------------------
>
> Key: YARN-1042
> URL: https://issues.apache.org/jira/browse/YARN-1042
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: resourcemanager
> Affects Versions: 3.0.0
> Reporter: Steve Loughran
> Assignee: Junping Du
> Attachments: YARN-1042-demo.patch
>
>
> container requests to the AM should be able to request anti-affinity to
> ensure that things like Region Servers don't come up on the same failure
> zones.
> Similarly, you may be able to want to specify affinity to same host or rack
> without specifying which specific host/rack. Example: bringing up a small
> giraph cluster in a large YARN cluster would benefit from having the
> processes in the same rack purely for bandwidth reasons.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
