Sadanand Shenoy created HDDS-17831:
--------------------------------------
Summary: Re-register failing SCM endpoint after prolonged
heartbeat failures with endpoint-local command-status cleanup
Key: HDDS-17831
URL: https://issues.apache.org/jira/browse/HDDS-17831
Project: Apache Ozone
Issue Type: Bug
Reporter: Sadanand Shenoy
Assignee: Sadanand Shenoy
A datanode can get stuck in {{HEARTBEAT}} against one SCM endpoint after
repeated heartbeat failures. In observed cases, SCM asks the DN to re-register,
but the failing endpoint may continue retrying heartbeat and not re-enter full
handshake until DN restart.
Add endpoint-scoped recovery for prolonged heartbeat failures:
* after a configurable number of consecutive failures, reset only the affected
SCM endpoint to {{GETVERSION}}
* continue through normal {{GETVERSION -> REGISTER -> HEARTBEAT}} flow
* clear queued command-status/delete-ACK reports for that endpoint during
reset (to mimic DN restart behavior for stuck ACK backlog)
* do not reset healthy SCM endpoints
* do not reset global datanode state
* keep passive endpoints (Recon) unchanged
* avoid resetting on every failure via threshold gating
* reset missed-failure counter on successful heartbeat (existing behavior)
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]