Devesh Kumar Singh created HDDS-16504:
-----------------------------------------
Summary: Datanode startup can hang indefinitely when Ratis group
recovery stalls (add a timeout/watchdog around container initialization)
Key: HDDS-16504
URL: https://issues.apache.org/jira/browse/HDDS-16504
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Datanode
Reporter: Devesh Kumar Singh
[HDDS-16425|https://issues.apache.org/jira/browse/HDDS-16425] makes datanode
startup fail cleanly when container initialization throws (e.g., a corrupt Raft
log with raft.server.log.corruption.policy = EXCEPTION): the state is marked
FAILED, waiting callers fail fast, and the datanode shuts down.
However, it does not cover the case where initialization hangs without
throwing. If OzoneContainer.start() → writeChannel.start() →
XceiverServerRatis.start() → Ratis server.start() blocks while recovering a
Raft group — for example a slow or failing volume where a disk read never
returns — then:
initializeContainerServices(...) never returns, so the thread holding
initializationLock holds it indefinitely.
OzoneContainer.start() never reaches INITIALIZED or FAILED.
VersionEndpointTask never advances the endpoint past GETVERSION, so the
datanode never registers or heartbeats.
RunningDatanodeState logs an endpoint‑task timeout every heartbeat interval,
forever, with no forward progress and no clean shutdown.
Net effect: the datanode is a "zombie" — the process is up but never registers,
and (unlike the throw path) it neither recovers nor fails cleanly. Only an
external restart clears it, and on restart the same group recovery is attempted
again, reproducing the hang.
*Current behavior*
* Throw during init → handled by HDDS‑16425 (FAILED + clean shutdown).
* Indefinite block during init (I/O stall in group recovery) → infinite hold of
initializationLock, no FAILED, no shutdown.
*Proposed change*
Add a bounded timeout/watchdog around container initialization so a stalled
startup is converted into an initialization failure that reuses the HDDS‑16425
clean‑shutdown path:
* Run initializeContainerServices(...) (or at minimum the writeChannel.start()
group‑recovery step) with a configurable timeout.
* If it does not complete within the timeout, mark initializingStatus = FAILED,
record a descriptive cause (e.g., TimeoutException("Container initialization
did not complete within <N>s")), release waiting callers with that failure, and
trigger the datanode shutdown path — the same terminal outcome as a thrown
initialization error.
* Because a thread blocked in a native/disk read generally cannot be
interrupted reliably, the watchdog should not depend on unblocking the stuck
thread; marking FAILED and initiating shutdown is sufficient (the process will
terminate).
*Configuration*
New config key (e.g., hdds.datanode.container.init.timeout) with a generous
default (group recovery on large logs can legitimately take time; the goal is
to catch true stalls, not slow‑but‑healthy startups). Timeout of 0/negative
disables the watchdog (preserves current behavior).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
