[ https://issues.apache.org/jira/browse/YARN-6355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277253#comment-16277253 ]
Arun Suresh edited comment on YARN-6355 at 12/4/17 11:04 PM: ------------------------------------------------------------- [~cheersyang], bq. Why DefaultAMSProcessor should be the last one in the chain? So, the primary reason was that we intended this to be a "pre" processing framework, which implies the processors should come before the DefaultAMSProcessor (Just like a servlet filterchain). Also, technically it is still possible to do the processing after DefaultAMProcessor - the OppContainerAllocator can FIRST call nextProcessor.allocate() and after that calls returns, THEN it can do its processing. bq. With the context of opportunistic containers, isn't it making more sense to re-order to.. DefaultAMSProcessor -> P1 -> P2 -> P3, so that guaranteed containers always get allocated first? It doesn't really matter for a couple of reasons: * Given that the O and G containers do not compete for resources currently since all O allocation is performed by the OppAllocator - The O container allocation is done purely based on queue length and the G containers are allocated based on exact resource capacity of the node at the time of allocation. so it doesnt really matter if the G containers are allocated first or not. * The G containers are also never allocated in the same allocate call anyway, all G container requests are always first queued and allocated asynchronously wrt to the allocate call. * If the DefaultAMSProcessor is first in line, we would have to write special code to prevent the G scheduler from adding the O container request to the queue etc. - not that this is very complicated, but objective was to reduce code change of existing code-paths as much as possible. was (Author: asuresh): [~cheersyang], bq. Why DefaultAMSProcessor should be the last one in the chain? So, the primary reason was that we intended this to be a "pre" processing framework, which implies the processors should come before the DefaultAMSProcessor (Just like a servlet filterchain) bq. With the context of opportunistic containers, isn't it making more sense to re-order to.. DefaultAMSProcessor -> P1 -> P2 -> P3, so that guaranteed containers always get allocated first? It doesn't really matter for a couple of reasons: * Given that the O and G containers do not compete for resources currently since all O allocation is performed by the OppAllocator - The O container allocation is done purely based on queue length and the G containers are allocated based on exact resource capacity of the node at the time of allocation. so it doesnt really matter if the G containers are allocated first or not. * The G containers are also never allocated in the same allocate call anyway, all G container requests are always first queued and allocated asynchronously wrt to the allocate call. * If the DefaultAMSProcessor is first in line, we would have to write special code to prevent the G scheduler from adding the O container request to the queue etc. - not that this is very complicated, but objective was to reduce code change of existing code-paths as much as possible. > [Umbrella] Preprocessor framework for AM and Client interactions with the RM > ---------------------------------------------------------------------------- > > Key: YARN-6355 > URL: https://issues.apache.org/jira/browse/YARN-6355 > Project: Hadoop YARN > Issue Type: Improvement > Reporter: Arun Suresh > Assignee: Arun Suresh > Labels: amrmproxy, resourcemanager > Attachments: YARN-6355-one-pager.pdf, YARN-6355.001.patch, > YARN-6355.002.patch, YARN-6355.003.patch, YARN-6355.004.patch, > YARN-6355.005.patch, YARN-6355.006.patch, YARN-6355.007.patch > > > Currently on the NM, we have the {{AMRMProxy}} framework to intercept the AM > <-> RM communication and enforce policies. This is used both by YARN > federation (YARN-2915) as well as Distributed Scheduling (YARN-2877). > This JIRA proposes to introduce a similar framework on the the RM side, so > that pluggable policies can be enforced on ApplicationMasterService centrally > as well. > This would be similar in spirit to a Java Servlet Filter Chain. Where the > order of the interceptors can declared externally. > Once possible usecase would be: > the {{OpportunisticContainerAllocatorAMService}} is implemented as a wrapper > over the {{ApplicationMasterService}}. It would probably be better to > implement it as an Interceptor. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org