[
https://issues.apache.org/jira/browse/HDDS-16120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18103598#comment-18103598
]
Chia-Chuan Ho commented on HDDS-16120:
--------------------------------------
Hi [~smeng]
Thanks for preparing the patch. I reviewed the approach and would like to
clarify the expected retry semantics before working on a PR.
The patch retries transparently and stops the create thread after three
consecutive failures. This avoids exposing a single transient failure, but it
introduces two policy values that may need justification: * Why should the
retry limit be three?
* Why should provide() poll the failure state every 500 ms?
After the third failure, the create thread also terminates. Subsequent
provide() calls fail quickly with the original cause, which is better than
waiting for the generic 100-second timeout, but the provider can no longer
recover.
An alternative is to place either a successfully created cluster or a build
failure into the existing bounded queue. provide() would consume the result and
immediately throw an IOException containing the original cause when it receives
a failure. The create thread would then continue building clusters.
Since the queue capacity is one, it also provides natural backpressure for
persistent failures: the create thread cannot process exceptions in an
unbounded tight loop when no consumer is draining the queue. This avoids adding
a retry count, backoff duration, or polling interval.
The behavior would be: * one failed build affects at most one provide() call;
* the original IOException or TimeoutException is preserved as the cause;
* the create thread remains available for subsequent builds;
* a partially created cluster is shut down;
* later provide() calls can still succeed.
The main semantic difference is that the patch hides a single transient failure
and retries it within the same provide() call, while the queue-result approach
surfaces that failure immediately and allows the next provide() call to recover.
The Jira description currently permits either behavior. Do we specifically want
a single provide() call to survive transient failures? If so, we should agree
on a retry policy. Otherwise, I prefer the bounded-queue approach because it
limits the blast radius without introducing arbitrary retry or timing constants.
What do you think?
> A single MiniOzoneCluster build timeout cascades into whole-suite failures in
> MiniOzoneClusterProvider
> ------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16120
> URL: https://issues.apache.org/jira/browse/HDDS-16120
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Chia-Chuan Ho
> Priority: Major
> Attachments: HDDS-16120.001.patch
>
>
> {{MiniOzoneClusterProvider.createClusters()}} builds clusters on a background
> thread and hands them to consumers through a small blocking queue. When a
> build times out ({{waitForClusterToBeReady()}} throws {{TimeoutException}})
> or fails with {{IOException}}, the background thread rethrows it as an
> unchecked {{RuntimeException("Unable to build cluster")}}. That exception is
> uncaught on the create thread, so the thread dies and the queue is never
> refilled. Every subsequent {{provide()}} then blocks until its own timeout
> and fails with "Failed to obtain available cluster in time".
> The effect is that a single slow or failed cluster build turns into failures
> for every remaining test that shares the provider. This was observed on
> master where one method timed out during cluster startup and the following
> seven methods in the same class all failed with "Failed to obtain available
> cluster in time". It also makes any transient startup slowdown
> disproportionately expensive, since one timeout can push an entire
> integration split to its 90 minute job limit.
> h3. Proposed fix
> Do not let one build failure kill the create thread. Tear down the partial
> cluster and continue (retry) so a single slow build is not fatal to the
> suite, or record the failure and surface the original cause from
> {{provide()}} while keeping the thread alive. Either way removes the "one
> timeout becomes many failures" behavior.
> h3. Testing
> Unit test for {{MiniOzoneClusterProvider}} where the builder throws on one
> build: assert that later {{provide()}} calls still succeed (retry path) or
> fail fast with the original cause, and that the create thread remains alive.
> h3. Note
> This is an amplifier that bounds the blast radius of any single slow cluster
> build; it is valuable independently of whatever triggers a slow build. It is
> not itself the root trigger of the recent master integration-job timeouts,
> and that root trigger is currently unconfirmed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
