[
https://issues.apache.org/jira/browse/HDDS-16504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-16504:
----------------------------------
Labels: pull-request-available (was: )
> Datanode startup can hang indefinitely when Ratis group recovery stalls (add
> a timeout/watchdog around container initialization)
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16504
> URL: https://issues.apache.org/jira/browse/HDDS-16504
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Datanode
> Reporter: Devesh Kumar Singh
> Assignee: Devesh Kumar Singh
> Priority: Major
> Labels: pull-request-available
>
> [HDDS-16425|https://issues.apache.org/jira/browse/HDDS-16425] makes datanode
> startup fail cleanly when container initialization throws (e.g., a corrupt
> Raft log with raft.server.log.corruption.policy = EXCEPTION): the state is
> marked FAILED, waiting callers fail fast, and the datanode shuts down.
> However, it does not cover the case where initialization hangs without
> throwing. If OzoneContainer.start() → writeChannel.start() →
> XceiverServerRatis.start() → Ratis server.start() blocks while recovering a
> Raft group — for example a slow or failing volume where a disk read never
> returns — then:
> initializeContainerServices(...) never returns, so the thread holding
> initializationLock holds it indefinitely.
> OzoneContainer.start() never reaches INITIALIZED or FAILED.
> VersionEndpointTask never advances the endpoint past GETVERSION, so the
> datanode never registers or heartbeats.
> RunningDatanodeState logs an endpoint‑task timeout every heartbeat interval,
> forever, with no forward progress and no clean shutdown.
> Net effect: the datanode is a "zombie" — the process is up but never
> registers, and (unlike the throw path) it neither recovers nor fails cleanly.
> Only an external restart clears it, and on restart the same group recovery is
> attempted again, reproducing the hang.
> *Current behavior*
> * Throw during init → handled by HDDS‑16425 (FAILED + clean shutdown).
> * Indefinite block during init (I/O stall in group recovery) → infinite hold
> of initializationLock, no FAILED, no shutdown.
> *Proposed change*
> Add a bounded timeout/watchdog around container initialization so a stalled
> startup is converted into an initialization failure that reuses the
> HDDS‑16425 clean‑shutdown path:
> * Run initializeContainerServices(...) (or at minimum the
> writeChannel.start() group‑recovery step) with a configurable timeout.
> * If it does not complete within the timeout, mark initializingStatus =
> FAILED, record a descriptive cause (e.g., TimeoutException("Container
> initialization did not complete within <N>s")), release waiting callers with
> that failure, and trigger the datanode shutdown path — the same terminal
> outcome as a thrown initialization error.
> * Because a thread blocked in a native/disk read generally cannot be
> interrupted reliably, the watchdog should not depend on unblocking the stuck
> thread; marking FAILED and initiating shutdown is sufficient (the process
> will terminate).
> *Configuration*
> New config key (e.g., hdds.datanode.container.init.timeout) with a generous
> default (group recovery on large logs can legitimately take time; the goal is
> to catch true stalls, not slow‑but‑healthy startups). Timeout of 0/negative
> disables the watchdog (preserves current behavior).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
