[
https://issues.apache.org/jira/browse/YARN-7102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175133#comment-16175133
]
Botong Huang edited comment on YARN-7102 at 9/21/17 5:19 PM:
-------------------------------------------------------------
Thanks [~jlowe] for the review and good point. We don't want RM to resync all
nodes if RM becomes slow.
How about we take one step back, allowing {{request.responseId >
lastResponseId}} as it is now? We simply fix the overflow problem without
changing anything else. Specifically, add one check: if {{request.responseId ==
lastResponseId}} then skip other checks. This would be my initial proposal in
YARN-6640 v1 patch:
{code}
if (request.getResponseId() != lastResponse.getResponseId()) {
if ((request.getResponseId() + 1) == lastResponse.getResponseId()) {
/* heartbeat one step old, simply return lastReponse */
return lastResponse;
} else if (request.getResponseId() + 1 <
lastResponse.getResponseId()) {
(resync NM)
}
}
(process the heartbeat...)
{code}
There's still potential for the RM too slow causing NM resync, but only
possible for the NMs whose reponseId just wrapped around. This should be fine I
guess.
was (Author: botong):
Thanks [~jlowe] for the review and good point. We don't want RM to resync all
nodes if RM becomes slow.
How about we take one step back, allowing {{request.responseId >
lastResponseId}} as it is now? We simply fix the overflow problem without
changing anything else. Specifically, add one check: if {{request.responseId ==
lastResponseId}} then skip other checks. This would be my initial proposal in
YARN-6640 v1 patch:
{code}
if (request.getResponseId() != lastResponse.getResponseId()) {
if ((request.getResponseId() + 1) == lastResponse.getResponseId()) {
/* heartbeat one step old, simply return lastReponse */
return lastResponse;
} else if (request.getResponseId() + 1 <
lastResponse.getResponseId()) {
throw new InvalidApplicationMasterRequestException(message);
}
}
(process the heartbeat...)
{code}
There's still potential for the RM too slow causing NM resync, but only
possible for the NMs whose reponseId just wrapped around. This should be fine I
guess.
> NM heartbeat stuck when responseId overflows MAX_INT
> ----------------------------------------------------
>
> Key: YARN-7102
> URL: https://issues.apache.org/jira/browse/YARN-7102
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Botong Huang
> Assignee: Botong Huang
> Priority: Critical
> Attachments: YARN-7102.v1.patch, YARN-7102.v2.patch,
> YARN-7102.v3.patch, YARN-7102.v4.patch, YARN-7102.v5.patch, YARN-7102.v6.patch
>
>
> ResponseId overflow problem in NM-RM heartbeat. This is same as AM-RM
> heartbeat in YARN-6640, please refer to YARN-6640 for details.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
