[ https://issues.apache.org/jira/browse/YARN-7720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16317509#comment-16317509 ]
Botong Huang commented on YARN-7720: ------------------------------------ As a fix in our clusters, we increased the AM heartbeat timeout to 15 mins and kept NM timeout to be 10 mins. This is a new requirement from Federation, that UAM timeout needs to be longer than NM timeout. I am wondering if it is okay to change the default value in general. Any other idea on this issue? > [Federation] Race condition between second app attempt and UAM heartbeat when > first attempt node is down > -------------------------------------------------------------------------------------------------------- > > Key: YARN-7720 > URL: https://issues.apache.org/jira/browse/YARN-7720 > Project: Hadoop YARN > Issue Type: Sub-task > Reporter: Botong Huang > Assignee: Botong Huang > > In Federation, multiple attempts of an application share the same UAM in each > secondary sub-cluster. When first attempt fails, we reply on the fact that > secondary RM won't kill the existing UAM before the AM heartbeat timeout > (default at 10 min). When second attempt comes up in the home sub-cluster, it > will pick up the UAM token from Yarn Registry and resume the UAM heartbeat to > secondary RMs. > The default heartbeat timeout for NM and AM are both 10 mins. The problem is > that when the first attempt node goes down or out of connection, only after > 10 mins will the home RM mark the first attempt as failed, and then schedule > the 2nd attempt in some other node. By then the UAMs in secondaries are > already timing out, and they might not survive until the second attempt comes > up. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org