Sadanand Shenoy created HDDS-17831:
--------------------------------------

             Summary:  Re-register failing SCM endpoint after prolonged 
heartbeat failures with endpoint-local command-status cleanup
                 Key: HDDS-17831
                 URL: https://issues.apache.org/jira/browse/HDDS-17831
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Sadanand Shenoy
            Assignee: Sadanand Shenoy


A datanode can get stuck in {{HEARTBEAT}} against one SCM endpoint after 
repeated heartbeat failures. In observed cases, SCM asks the DN to re-register, 
but the failing endpoint may continue retrying heartbeat and not re-enter full 
handshake until DN restart.

Add endpoint-scoped recovery for prolonged heartbeat failures:
 * after a configurable number of consecutive failures, reset only the affected 
SCM endpoint to {{GETVERSION}}
 * continue through normal {{GETVERSION -> REGISTER -> HEARTBEAT}} flow
 * clear queued command-status/delete-ACK reports for that endpoint during 
reset (to mimic DN restart behavior for stuck ACK backlog)
 * do not reset healthy SCM endpoints
 * do not reset global datanode state
 * keep passive endpoints (Recon) unchanged
 * avoid resetting on every failure via threshold gating
 * reset missed-failure counter on successful heartbeat (existing behavior)

 



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to