ptlrs commented on code in PR #9954:
URL: https://github.com/apache/ozone/pull/9954#discussion_r3112860443
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/StorageVolumeChecker.java:
##########
@@ -220,6 +221,7 @@ public Set<? extends StorageVolume> checkAllVolumes(
final Set<StorageVolume> healthyVolumes = ConcurrentHashMap.newKeySet();
final Set<StorageVolume> failedVolumes = ConcurrentHashMap.newKeySet();
final Set<StorageVolume> allVolumes = new HashSet<>();
+ final Object resultLock = this;
Review Comment:
Consider synchronizing on a `private final Object lock = new Object()` for
[safety](https://wiki.sei.cmu.edu/confluence/display/java/LCK00-J.+Use+private+final+lock+objects+to+synchronize+classes+that+may+interact+with+untrusted+code).
##########
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestStorageVolumeHealthChecks.java:
##########
@@ -324,6 +336,88 @@ public void
testCorrectDirectoryChecked(StorageVolume.Builder<?> builder)
volume.check(false);
}
+ /**
+ * With the default settings (ioFailureTolerance=1), the first simulated
Review Comment:
Change `ioFailureTolerance` to `diskCheckTimeoutFailureTolerance` here and
following tests which are related to disk check timeout.
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/StorageVolumeChecker.java:
##########
@@ -376,18 +416,41 @@ public void onFailure(@Nonnull Throwable t) {
volume, exception);
// If the scan was interrupted, do not count it as a volume failure.
// This should only happen if the volume checker is being shut down.
- if (!(t instanceof InterruptedException)) {
- markFailed();
- cleanup();
+ if (t instanceof InterruptedException) {
+ return;
+ }
+ // Detect a per-check timeout from ThrottledAsyncChecker.
+ // Guava 28+ (including 33.5.0-jre used here) fails the TimeoutFuture
+ // with TimeoutException on timeout.
+ boolean isTimeout = exception instanceof TimeoutException;
+ if (isTimeout) {
+ if (timeoutHandledInCheckAllVolumes) {
+ cleanup();
+ return;
+ }
+ if (!volume.recordTimeoutAndCheckFailure()) {
+ // Within tolerance: do NOT trigger the failure callback.
+ // The volume is not marked failed; the next check cycle will
+ // re-evaluate its health. cleanup() is intentionally not called
+ // to avoid firing handleVolumeFailures() with an empty failed set.
+ return;
+ }
+ // Tolerance exceeded — fall through to markFailed()/cleanup().
}
+ markFailed();
+ cleanup();
Review Comment:
The `if` check for `recordTimeoutAndCheckFailure` swallows the `callback`
that was passed in `checkVolume`.
Perhaps the check for `recordTimeoutAndCheckFailure` should be done inside
`markFailed` which would then skip adding the volume to failedVolumes.
Then cleanup will execute the callback but not fail the volume because
failedVolumes set will be empty on the first failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]