[ https://issues.apache.org/jira/browse/YARN-7102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175133#comment-16175133 ]
Botong Huang edited comment on YARN-7102 at 9/21/17 5:19 PM: ------------------------------------------------------------- Thanks [~jlowe] for the review and good point. We don't want RM to resync all nodes if RM becomes slow. How about we take one step back, allowing {{request.responseId > lastResponseId}} as it is now? We simply fix the overflow problem without changing anything else. Specifically, add one check: if {{request.responseId == lastResponseId}} then skip other checks. This would be my initial proposal in YARN-6640 v1 patch: {code} if (request.getResponseId() != lastResponse.getResponseId()) { if ((request.getResponseId() + 1) == lastResponse.getResponseId()) { /* heartbeat one step old, simply return lastReponse */ return lastResponse; } else if (request.getResponseId() + 1 < lastResponse.getResponseId()) { (resync NM...) } } (process the heartbeat...) {code} There's still potential for the RM too slow causing NM resync, but only possible for the NMs whose reponseId just wrapped around. This should be fine I guess. was (Author: botong): Thanks [~jlowe] for the review and good point. We don't want RM to resync all nodes if RM becomes slow. How about we take one step back, allowing {{request.responseId > lastResponseId}} as it is now? We simply fix the overflow problem without changing anything else. Specifically, add one check: if {{request.responseId == lastResponseId}} then skip other checks. This would be my initial proposal in YARN-6640 v1 patch: {code} if (request.getResponseId() != lastResponse.getResponseId()) { if ((request.getResponseId() + 1) == lastResponse.getResponseId()) { /* heartbeat one step old, simply return lastReponse */ return lastResponse; } else if (request.getResponseId() + 1 < lastResponse.getResponseId()) { (resync NM) } } (process the heartbeat...) {code} There's still potential for the RM too slow causing NM resync, but only possible for the NMs whose reponseId just wrapped around. This should be fine I guess. > NM heartbeat stuck when responseId overflows MAX_INT > ---------------------------------------------------- > > Key: YARN-7102 > URL: https://issues.apache.org/jira/browse/YARN-7102 > Project: Hadoop YARN > Issue Type: Bug > Reporter: Botong Huang > Assignee: Botong Huang > Priority: Critical > Attachments: YARN-7102.v1.patch, YARN-7102.v2.patch, > YARN-7102.v3.patch, YARN-7102.v4.patch, YARN-7102.v5.patch, YARN-7102.v6.patch > > > ResponseId overflow problem in NM-RM heartbeat. This is same as AM-RM > heartbeat in YARN-6640, please refer to YARN-6640 for details. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org