[
https://issues.apache.org/jira/browse/YARN-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352072#comment-15352072
]
Wangda Tan commented on YARN-1042:
----------------------------------
Hi [~cheersyang], thanks for the design doc, patch and ideas.
I think there're two major problems that we need to solve.
1) What the API should look like?
2) How to make it efficient enough?
For 1), I'm agree with most of the comments from [~cchen317], existing
ResourceRequest API has some flaws that we may need to think how to fix them.
We have a design doc attached to YARN-4902 with a lot of thoughts regarding to
this, you can read it if you have interests.
And for anti-affinity/affinity API itself, we may need to consider
anti-affinity/affinity between applications/priorities. Sometimes different
services can be launched by one single YARN application, different roles from
different services could possibly have different affinity/anti-affinity
requirements. And one service/job can affinity/anti-affinity to another
service/job.
For 2), Existing scheduler is blindly looping all the nodes in the cluster, we
need a global view to schedule them for better performance, I have opened
YARN-5139 for the global scheduling framework and attached POC patch to it.
Currently I'm working on a prototype patch for this, I want to make this
feature based on YARN-5139 and probably based on YARN-4902.
I'm interested in taking this forward, assigning it to myself.
Would like to hear your thoughts.
> add ability to specify affinity/anti-affinity in container requests
> -------------------------------------------------------------------
>
> Key: YARN-1042
> URL: https://issues.apache.org/jira/browse/YARN-1042
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
> Affects Versions: 3.0.0-alpha1
> Reporter: Steve Loughran
> Assignee: Wangda Tan
> Attachments: YARN-1042-demo.patch, YARN-1042-design-doc.pdf,
> YARN-1042.001.patch, YARN-1042.002.patch
>
>
> container requests to the AM should be able to request anti-affinity to
> ensure that things like Region Servers don't come up on the same failure
> zones.
> Similarly, you may be able to want to specify affinity to same host or rack
> without specifying which specific host/rack. Example: bringing up a small
> giraph cluster in a large YARN cluster would benefit from having the
> processes in the same rack purely for bandwidth reasons.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
