ptlrs commented on code in PR #9954:
URL: https://github.com/apache/ozone/pull/9954#discussion_r3091993264


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/StorageVolumeChecker.java:
##########
@@ -246,18 +255,44 @@ public Set<? extends StorageVolume> checkAllVolumes(
 
     // Wait until our timeout elapses, after which we give up on
     // the remaining volumes.
-    if (!latch.await(maxAllowedTimeForCheckMs, TimeUnit.MILLISECONDS)) {
-      LOG.warn("checkAllVolumes timed out after {} ms",
-          maxAllowedTimeForCheckMs);
-    }
+    boolean completedOnTime =
+        latch.await(maxAllowedTimeForCheckMs, TimeUnit.MILLISECONDS);
 
     synchronized (this) {
-      // All volumes that have not been detected as healthy should be
-      // considered failed. This is a superset of 'failedVolumes'.
-      //
-      // Make a copy under the mutex as Sets.difference() returns a view
-      // of a potentially changing set.
-      return new HashSet<>(Sets.difference(allVolumes, healthyVolumes));
+      if (!completedOnTime) {
+        LOG.warn("checkAllVolumes timed out after {} ms."
+            + " Evaluating per-volume latch-timeout tolerance.",
+            maxAllowedTimeForCheckMs);
+      }
+
+      // Volumes that explicitly reported FAILED via check() are always
+      // returned — the IO-failure sliding window in StorageVolume.check()
+      // already applied its own tolerance.
+      final Set<StorageVolume> result = new HashSet<>(failedVolumes);
+
+      // Volumes still pending (neither healthy nor explicitly failed) at
+      // latch-timeout time.  onFailure() may have already handled some of
+      // these via timeoutHandledSet; skip those to avoid double-counting.
+      final Set<StorageVolume> pendingVolumes =
+          new HashSet<>(Sets.difference(allVolumes,
+              Sets.union(healthyVolumes, failedVolumes)));

Review Comment:
   There is nothing stopping the `failedVolumes` set from being updated between 
the creation of `result` and here.
   
   Consider the following timeline:
   T1. A volume `v1` is not part of `failedVolumes` when `result` is created. 
   T2. `onFailure` adds `v1` to `failedVolumes`. 
   T3. `pendingVolumes` doesn't have `v1` as it is now part of `failedVolumes`.
   T4. `result` was already created without `v1`.
   T5. When `result` is returned, `v1` is missing and not processed for 
failure. 
   
   



##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/StorageVolume.java:
##########
@@ -72,6 +73,7 @@
 public abstract class StorageVolume implements Checkable<Boolean, 
VolumeCheckResult> {
 
   private static final Logger LOG = 
LoggerFactory.getLogger(StorageVolume.class);
+  private static final Duration TIMEOUT_FAILURE_WINDOW = 
Duration.ofMinutes(70);

Review Comment:
   Let's make the failure window duration configurable. 
`org.apache.hadoop.ozone.container.common.statemachine.DatanodeConfiguration` 
can be used for this.  



##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/StorageVolume.java:
##########
@@ -162,6 +165,9 @@ protected StorageVolume(Builder<?> b) throws IOException {
       this.isDiskCheckEnabled = dnConf.isDiskCheckEnabled();
       this.ioTestSlidingWindow = new 
SlidingWindow(dnConf.getVolumeIOFailureTolerance(),
           dnConf.getDiskCheckSlidingWindowTimeout(), b.getClock());
+      this.timeoutFailureSlidingWindow = new SlidingWindow(
+          dnConf.getVolumeIOFailureTolerance(), TIMEOUT_FAILURE_WINDOW,
+          b.getClock());

Review Comment:
   Instead of `getVolumeIOFailureTolerance`, let's use a new configuration such 
that we can disable the rocksdb opening test independently of the disk checks. 
   
   @errose28 @adoroszlai In the new [SlidingWindow 
class](https://github.com/apache/ozone/blob/9ff7eaa6c19bbd2e8aa951197127c503fe5b2159/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/SlidingWindow.java#L97),
 we don't currently use a window size of `0` or `-1` to ignore any failures. 
   
   We have three options:
   1. Use a magic number: We can update the SlidingWindow class to ignore 
failures if the window size is `-1`. We don't use `0` here because we may want 
a sliding window where we tolerate no failures. This makes it a universal 
solution but we are now using a magic number to introduce functionality. 
   2. Toggle the checks at the sliding window layer: We can add explicit 
`enableTracking()` `disableTracking()` methods to enable and  disable the 
sliding window. However, this will require an extra configuration for each 
window that may need to be toggled. 
   3. Toggle the checks with the caller's cooperation: We can use an explicit 
boolean configuration like `isTestEnabled` for each scenario where we want to 
just toggle the sliding window.
    
   Do you have any thoughts on which approach may be preferred?



##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/StorageVolumeChecker.java:
##########
@@ -376,10 +427,34 @@ public void onFailure(@Nonnull Throwable t) {
           volume, exception);
       // If the scan was interrupted, do not count it as a volume failure.
       // This should only happen if the volume checker is being shut down.
-      if (!(t instanceof InterruptedException)) {
-        markFailed();
-        cleanup();
+      if (t instanceof InterruptedException) {
+        return;
       }
+      // Detect a per-check timeout from ThrottledAsyncChecker.
+      // Guava 28+ (including 33.5.0-jre used here) fails the TimeoutFuture
+      // with TimeoutException on timeout.
+      boolean isTimeout = exception instanceof TimeoutException;
+      if (isTimeout) {
+        // timeoutHandledSet is null for checkVolume() (sole timeout handler).
+        // For checkAllVolumes(), the set is shared with the pending-volumes
+        // loop; CAS-add determines which path owns the tolerance decision.
+        boolean firstToHandle =
+            (timeoutHandledSet == null) || timeoutHandledSet.add(volume);
+        if (firstToHandle) {
+          if (!volume.recordTimeoutAndCheckFailure()) {

Review Comment:
   This method is confusing to me when it comes to its effects along with what 
is being done by the `checkAllVolumes` method. 
   
   For a single volume check we would need to perform the 
`recordTimeoutAndCheckFailure` here but for `checkAllVolumes`, we should call 
`recordTimeoutAndCheckFailure` for all the volumes in one place instead of some 
being handled in `checkAllVolumes` and some being handled in `onFailure`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to