K0K0V0K commented on code in PR #8798:
URL: https://github.com/apache/hadoop/pull/8798#discussion_r4204417522
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java:
##########
@@ -593,25 +600,97 @@ private DiskErrorInformation
validateSubsAccessibility(File dir) {
if (!subAccessibilityValidationEnabled) {
return null;
}
- try (Stream<java.nio.file.Path> walk = Files.walk(dir.toPath())) {
- List<File> subs = walk
- .map(java.nio.file.Path::toFile)
- .collect(Collectors.toList());
- for (File sub : subs) {
- if (sub.isDirectory()) {
- DiskChecker.checkDir(sub);
- } else if (!Files.isReadable(sub.toPath())) {
- return new DiskErrorInformation(DiskErrorCause.OTHER, "Can not read
" + sub);
- } else {
- LOG.debug("{} under {} is accessible", sub, dir);
- }
- }
+ LOG.debug("Start checking accessibility for: {}", dir);
+
+ try {
+ Files.walkFileTree(dir.toPath(), Collections.emptySet(),
+ subAccessibilityValidationMaxDepth, new SubAccessibilityVisitor());
} catch (IOException | UncheckedIOException | SecurityException e) {
return new DiskErrorInformation(DiskErrorCause.OTHER, e.getMessage());
}
return null;
}
+ /**
+ * Walks the content of an NM directories.
+ * It fails with an {@link IOException} if the NM user cannot access the
entry.
+ * Entries must be readable, writable and executable.
+ * Symbolic links are neither followed nor checked.
+ * <p>
+ * Entries deleted during the walk are ignored, because
+ * localization and cleanup modify these directories concurrently.
+ */
+ @VisibleForTesting
+ static class SubAccessibilityVisitor
+ extends SimpleFileVisitor<java.nio.file.Path> {
+ private final String nmUser;
+
+ SubAccessibilityVisitor() {
+ this(System.getProperty("user.name"));
+ }
+
+ @VisibleForTesting
+ SubAccessibilityVisitor(String nmUser) {
+ this.nmUser = nmUser;
+ }
+
+ @Override
+ public FileVisitResult preVisitDirectory(java.nio.file.Path p,
Review Comment:
Thanks @Hean-Chhinling for fixing this problem!
I think the first dir what will be checked here is the root dir of the NM
local files. (correct me if i am wrong)
So the first step will validate this is owned by NM or not, and if not it
will report everything OK.
I think that would be nice, if we can change this behaviour.
IF nm local dirs not owned by yarn user -> report bad health.
UPDATE: see above ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
