[
https://issues.apache.org/jira/browse/YARN-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14585494#comment-14585494
]
Weiwei Yang commented on YARN-1042:
-----------------------------------
Hi Steve
>From your comments, I think you want yarn API to support requesting containers
>with given rules per request. Guess you want it to have the API looks like
in AMRMClient.java, ContainerRequest supports following arguments
* Resource capability
* String[] nodes
* String[] racks
* Priority priority
* boolean relaxLocality
* String nodeLabelExpression
* *ContainerAllocateRule containerAllocateRule*
the last one (in bold text) is the new argument for you to specify a particular
rule (we will discuss rule details later). Problem here is, RM needs to know
which application (or role in slider's context) the rule applies for, only then
RM can assign containers by obeying that rule when dealing with the request
coming from the same application. However, if you only specify the rule per
container request, how can RM know what are the containers need to be
considered to when applies the rule ? Let me give an example to explain
{code}
ContainerRequest containerReq1 = new ContainerRequest(capability1, nodes,
racks, priority, affinityRequiredRule);
amClient.addContainerRequest(containerReq1);
AllocateResponse allocResponse = amClient.allocate(0.1f)
{code}
The AllocationRequest AM sent to RM only tells RM that these container requests
need to use affinityRequiredRule, but RM does not know which containers this
request affine with, so RM cannot place the rule during the allocation. This
is the reason why I propose to register the mapping about
{code}
application <-> allocation-rule
{code}
when client submits the application, and keep it in the RM context, so RM can
apply the rule when there is a request coming from AM.
> add ability to specify affinity/anti-affinity in container requests
> -------------------------------------------------------------------
>
> Key: YARN-1042
> URL: https://issues.apache.org/jira/browse/YARN-1042
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
> Affects Versions: 3.0.0
> Reporter: Steve Loughran
> Assignee: Arun C Murthy
> Attachments: YARN-1042-demo.patch, YARN-1042-design-doc.pdf,
> YARN-1042.001.patch
>
>
> container requests to the AM should be able to request anti-affinity to
> ensure that things like Region Servers don't come up on the same failure
> zones.
> Similarly, you may be able to want to specify affinity to same host or rack
> without specifying which specific host/rack. Example: bringing up a small
> giraph cluster in a large YARN cluster would benefit from having the
> processes in the same rack purely for bandwidth reasons.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
