smengcl commented on code in PR #11264:
URL: https://github.com/apache/ozone/pull/11264#discussion_r4052650222
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java:
##########
@@ -572,6 +576,62 @@ public void start(String clusterId) throws IOException {
}
}
+ /**
+ * Runs {@link #initializeContainerServices(String)}, optionally bounded by a
+ * watchdog timeout ({@code hdds.datanode.container.init.timeout}).
+ * <p>
+ * When the timeout is a positive duration, initialization runs on a separate
+ * thread so a stall - for example Ratis group recovery blocked on a failing
+ * volume inside {@code writeChannel.start()} - is turned into an
+ * {@link IOException} instead of an indefinite hang. The caller ({@code
+ * start}) then records the failure ({@code FAILED}) and the datanode shuts
+ * down cleanly. A non-positive timeout disables the watchdog and runs
+ * initialization inline, preserving the previous behavior.
+ */
+ private void initializeContainerServicesWithTimeout(String clusterId) throws
IOException {
+ Duration initTimeout =
+
config.getObject(DatanodeConfiguration.class).getContainerInitTimeout();
+ if (initTimeout == null || initTimeout.isZero() ||
initTimeout.isNegative()) {
+ initializeContainerServices(clusterId);
+ return;
+ }
+
+ ExecutorService initExecutor = Executors.newSingleThreadExecutor(
+ new ThreadFactoryBuilder().setDaemon(true)
+ .setNameFormat("OzoneContainerInit").build());
+ Future<?> initFuture = initExecutor.submit(() -> {
+ initializeContainerServices(clusterId);
+ return null;
+ });
+ try {
+ initFuture.get(initTimeout.toMillis(), TimeUnit.MILLISECONDS);
+ } catch (TimeoutException e) {
+ // Best-effort interrupt; a thread blocked on disk I/O may not respond,
+ // but the datanode will shut down once the caller marks startup FAILED.
+ initFuture.cancel(true);
+ throw new IOException("OzoneContainer initialization did not complete
within " + initTimeout
+ + ". Failing datanode startup; a stalled Ratis group recovery or
volume I/O is the likely cause.", e);
Review Comment:
Could we prevent initialization from continuing after the timeout, and
coordinate cleanup with the surviving initializer?
`cancel(true)` and `shutdownNow()` only interrupt the worker. Ratis 3.2.1's
`RaftServerProxy.startImpl()` waits on `CompletableFuture.join()`, which
ignores interruption. If recovery completes later,
`initializeContainerServices()` continues starting services even though startup
is already FAILED and shutdown may have cleaned up their dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
