[
https://issues.apache.org/jira/browse/YARN-6355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277253#comment-16277253
]
Arun Suresh edited comment on YARN-6355 at 12/4/17 11:04 PM:
-------------------------------------------------------------
[~cheersyang],
bq. Why DefaultAMSProcessor should be the last one in the chain?
So, the primary reason was that we intended this to be a "pre" processing
framework, which implies the processors should come before the
DefaultAMSProcessor (Just like a servlet filterchain). Also, technically it is
still possible to do the processing after DefaultAMProcessor -
the OppContainerAllocator can FIRST call nextProcessor.allocate() and after
that calls returns, THEN it can do its processing.
bq. With the context of opportunistic containers, isn't it making more sense to
re-order to.. DefaultAMSProcessor -> P1 -> P2 -> P3, so that guaranteed
containers always get allocated first?
It doesn't really matter for a couple of reasons:
* Given that the O and G containers do not compete for resources currently
since all O allocation is performed by the OppAllocator - The O container
allocation is done purely based on queue length and the G containers are
allocated based on exact resource capacity of the node at the time of
allocation. so it doesnt really matter if the G containers are allocated first
or not.
* The G containers are also never allocated in the same allocate call anyway,
all G container requests are always first queued and allocated asynchronously
wrt to the allocate call.
* If the DefaultAMSProcessor is first in line, we would have to write special
code to prevent the G scheduler from adding the O container request to the
queue etc. - not that this is very complicated, but objective was to reduce
code change of existing code-paths as much as possible.
was (Author: asuresh):
[~cheersyang],
bq. Why DefaultAMSProcessor should be the last one in the chain?
So, the primary reason was that we intended this to be a "pre" processing
framework, which implies the processors should come before the
DefaultAMSProcessor (Just like a servlet filterchain)
bq. With the context of opportunistic containers, isn't it making more sense to
re-order to.. DefaultAMSProcessor -> P1 -> P2 -> P3, so that guaranteed
containers always get allocated first?
It doesn't really matter for a couple of reasons:
* Given that the O and G containers do not compete for resources currently
since all O allocation is performed by the OppAllocator - The O container
allocation is done purely based on queue length and the G containers are
allocated based on exact resource capacity of the node at the time of
allocation. so it doesnt really matter if the G containers are allocated first
or not.
* The G containers are also never allocated in the same allocate call anyway,
all G container requests are always first queued and allocated asynchronously
wrt to the allocate call.
* If the DefaultAMSProcessor is first in line, we would have to write special
code to prevent the G scheduler from adding the O container request to the
queue etc. - not that this is very complicated, but objective was to reduce
code change of existing code-paths as much as possible.
> [Umbrella] Preprocessor framework for AM and Client interactions with the RM
> ----------------------------------------------------------------------------
>
> Key: YARN-6355
> URL: https://issues.apache.org/jira/browse/YARN-6355
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Arun Suresh
> Assignee: Arun Suresh
> Labels: amrmproxy, resourcemanager
> Attachments: YARN-6355-one-pager.pdf, YARN-6355.001.patch,
> YARN-6355.002.patch, YARN-6355.003.patch, YARN-6355.004.patch,
> YARN-6355.005.patch, YARN-6355.006.patch, YARN-6355.007.patch
>
>
> Currently on the NM, we have the {{AMRMProxy}} framework to intercept the AM
> <-> RM communication and enforce policies. This is used both by YARN
> federation (YARN-2915) as well as Distributed Scheduling (YARN-2877).
> This JIRA proposes to introduce a similar framework on the the RM side, so
> that pluggable policies can be enforced on ApplicationMasterService centrally
> as well.
> This would be similar in spirit to a Java Servlet Filter Chain. Where the
> order of the interceptors can declared externally.
> Once possible usecase would be:
> the {{OpportunisticContainerAllocatorAMService}} is implemented as a wrapper
> over the {{ApplicationMasterService}}. It would probably be better to
> implement it as an Interceptor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
