henrybear327 commented on code in PR #10760:
URL: https://github.com/apache/ozone/pull/10760#discussion_r4262150263
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/local/LocalOzoneCluster.java:
##########
@@ -844,6 +949,92 @@ private void startS3Gateway(OzoneConfiguration conf)
throws Exception {
s3Gateway.call();
}
+ /**
+ * Runs a blocking in-JVM service startup under {@code timeout}, so the
launcher fails fast and
+ * rolls back instead of hanging if a startup ever blocks. {@code
startupCleanup} is the only
+ * stopper for an attempt that did not finish: a startup that ignores the
interrupt keeps running
+ * after {@code stopServices()} has rolled back, so on a timeout or an
interrupt the cleanup is
+ * queued behind it on the same single thread rather than run here. The
caller assigns its field
+ * only after this returns, so a failed attempt is never reachable from the
rollback.
+ */
+ static <V> V executeWithinStartupTimeout(String serviceName, Callable<V>
startup,
+ AutoCloseable startupCleanup, Duration timeout) throws Exception {
+ ExecutorService executor = Executors.newSingleThreadExecutor(runnable -> {
+ Thread thread = new Thread(runnable, "local-startup-" + serviceName);
+ thread.setDaemon(true);
+ return thread;
+ });
+ Future<V> future = executor.submit(startup);
+ try {
+ return future.get(timeout.toMillis(), TimeUnit.MILLISECONDS);
+ } catch (TimeoutException ex) {
+ future.cancel(true);
+ executor.submit(() -> IOUtils.close(LOG, startupCleanup));
+ throw new IOException("Local " + serviceName + " did not start within "
+ timeout + ".", ex);
+ } catch (InterruptedException ex) {
+ // Queued, not run here, for the reason given on this method: the
startup may finish late.
+ future.cancel(true);
+ executor.submit(() -> IOUtils.close(LOG, startupCleanup));
+ Thread.currentThread().interrupt();
+ throw ex;
+ } catch (ExecutionException ex) {
+ IOUtils.close(LOG, startupCleanup);
+ Throwable cause = ex.getCause();
+ throw cause instanceof Exception ? (Exception) cause : ex;
+ } finally {
+ // shutdown(), not shutdownNow(): the timeout path queues the cleanup,
which shutdownNow()
+ // would discard. The worker is a daemon thread, so it cannot keep the
JVM alive.
+ executor.shutdown();
+ }
+ }
+
+ private void startRecon(OzoneConfiguration conf) throws Exception {
+ // ReconServer reads its configuration from the static provider, and the
reset is what lets a
+ // second in-JVM start win: setConfiguration() keeps the first value. Same
pair as ReconService.
+ ConfigurationProvider.resetConfiguration();
+ ConfigurationProvider.setConfiguration(new OzoneConfiguration(conf));
+ ReconServer recon = new ReconServer();
+ int exitCode = executeWithinStartupTimeout("Recon", () ->
recon.execute(NO_ARGS),
+ recon::stop, config.getStartupTimeout());
+ if (exitCode != 0) {
+ // execute() has returned, so stopping the partially started server
cannot race it.
+ IOUtils.close(LOG, recon::stop);
+ throw new IOException("Failed to start local Recon. Exit code "
+ + exitCode + ".");
+ }
+ reconServer = recon;
+ }
+
+ private void waitForHttpEndpointReadiness(String endpoint,
+ String serviceName, Duration timeout) throws Exception {
+ waitForReadiness(() -> httpEndpointBlocker(endpoint, serviceName),
+ serviceName, timeout);
+ }
+
+ /**
+ * Returns why {@code endpoint} is not serving yet, or null once it is.
{@code ReconServer#call()}
+ * returns success even when its initialization threw, so an HTTP response
is the only readiness
+ * signal; the reason is kept because a refused connection, a wrong port and
a hung listener are
+ * indistinguishable once the wait times out.
+ */
+ private static String httpEndpointBlocker(String endpoint, String
serviceName) {
+ HttpURLConnection connection = null;
+ try {
+ connection = (HttpURLConnection) new URL(endpoint).openConnection();
+ connection.setConnectTimeout((int) READINESS_POLL_INTERVAL_MILLIS);
+ connection.setReadTimeout((int) READINESS_POLL_INTERVAL_MILLIS);
+ connection.getResponseCode();
+ return null;
Review Comment:
I have also refactored to using using `call` instead of `execute()`, as
`execute` turns a startup failure into exit code -1 and prints only the first
line of the error, but `call()` throws the exception, so `start()` can roll
back and rethrow it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
