[ https://issues.apache.org/jira/browse/YARN-7102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16176421#comment-16176421 ]
Jason Lowe commented on YARN-7102: ---------------------------------- Not a fan of that approach either. It has a corner case with the same issue, and it doesn't solve the following scenario which is probably already happening today even without this patch: # NM heartbeats with response ID 1 # ResourceTrackerService responds with response ID 2, asynchronously posting the message with response ID 2 to RMNodeImpl # RM is slow to process the asynchronous RMNodeStatusEvent containing the updated response with ID 2, so the RMNodeImpl still has the old lastResponse with response ID 1 # NM performs next heaertbeat with response ID 2 # RM mistakenly believes this is a duplicate heartbeat message from the NM and *throws away the heartbeat update* ResourceTrackerService is already calling RMNode synchronously in two places with the updated response, specifically RMNode#updateNodeHeartbeatResponseForCleanup and RMNode#updateNodeHeartbeatResponseForUpdatedContainers, so there are two opportunities for the RMNode to update its tracking of the proper last response ID before the heartbeat response is sent to the NM. Therefore we have at least two existing opportunities to get this right even when the response ID wraps and also solve the race condition above. > NM heartbeat stuck when responseId overflows MAX_INT > ---------------------------------------------------- > > Key: YARN-7102 > URL: https://issues.apache.org/jira/browse/YARN-7102 > Project: Hadoop YARN > Issue Type: Bug > Reporter: Botong Huang > Assignee: Botong Huang > Priority: Critical > Attachments: YARN-7102.v1.patch, YARN-7102.v2.patch, > YARN-7102.v3.patch, YARN-7102.v4.patch, YARN-7102.v5.patch, YARN-7102.v6.patch > > > ResponseId overflow problem in NM-RM heartbeat. This is same as AM-RM > heartbeat in YARN-6640, please refer to YARN-6640 for details. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org