Ming Ma created MAPREDUCE-6694: ---------------------------------- Summary: Make AM more resilient to potential lost of any completed container notification Key: MAPREDUCE-6694 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6694 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Ming Ma
YARN tries to guarantee any completed container notification is delivered to AM under any circumstance, YARN-1372 is an example to make sure for the case of RM restart. However, under some corner cases, it is still possible a completed container notifications is lost or significantly delayed. For example, if NM host becomes dead when RM fails over. AM won't preempt reducers if it thought there is at least one mapper running. {noformat} void preemptReducesIfNeeded() { ... if (assignedRequests.maps.size() > 0) { // there are assigned mappers return; } ... {noformat} Instead of completely depending on notification from RM, it can use TaskUmbilicalProtocol to help to decide if there is any mapper running. That will make AM more resilient to any bugs in YARN. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org