Hean-Chhinling opened a new pull request, #8798: URL: https://github.com/apache/hadoop/pull/8798
Contains content generated by Cursor. ### Description of PR The content accessibility validation added by [YARN-11703](https://issues.apache.org/jira/browse/YARN-11703) walks every entry under theNodeManager local and log dirs. It collects the **full list of paths** first and then checks each one. The NodeManager keeps creating, renaming and deleting files in those dirs the whole time (localization `_tmp` dirs being renamed, cache cleanup, application cleanup, log aggregation). Therefore, any entries that disappear between clean ups and accessibility checking are reported as unreadable and the whole disk is marked as unhealhty. Additionally, It also fails on no-longer-existing symlinks, and on directories owned by application users under `LinuxContainerExecutor`, because it requires write access everywhere. A falsely failed disk causes shuffle fetch failures (map output on that disk is no longer searched), duplicate localizations that leak the old copy, and unhealthy NodeManagers. See YARN-<id> for details. This PR changes `DirectoryCollection#validateSubsAccessibility` to the following: 1. Stop at a configurable depth, via the new `yarn.nodemanager.disk-health-checker.working-dir-content-accessibility-validation.max-depth` property default 3, because it is deepest directory in `filecache/<id>/<resource>` or `usercache/<user>/appcache` to start a new containers 2. Walk with `Files.walkFileTree` and check entries as they are visited; 3. Ignore entries that no longer exist (`NoSuchFileException`, or `Files.notExists` with `NOFOLLOW_LINKS` after a failed check); 4. Not follow symlinks; 5. Skip entries, and the subtrees of directories, that aren't owned by the NodeManager user; 6. No longer call `DiskChecker.checkDir()` on subdirectories, which could re-create directories that cleanup had just removed. ### How was this patch tested? Tested manually on a cluster with reproducing the issue and verified the patch solve the problem. In the patch, I have written the following unit-tests: - `testSubAccessibilityToleratesEntryDeletedDuringWalk`: hands the visitor an entry that was deleted after its attributes were read, so the race is reproduced deterministically. - `testSubAccessibilityIgnoresBrokenSymlink` - `testSubAccessibilityRespectsMaxDepth`: an unreadable file below the depth limit is ignored, and found once the limit is raised. - `testSubAccessibilitySkipsEntriesOwnedByOtherUsers` - `testSubAccessibilityUnderConcurrentChurn`: a background thread keeps creating and deleting `filecache/N_tmp` trees while the check runs 500 times. Against the code before this change it fails within the first couple of iterations with `Can not read .../filecache/N_tmp/...`; with the change all iterations pass. ### For code changes: - [ ] Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: Have the integration tests been executed and the endpoint declared according to the connector-specific documentation? *Note: Automated CI testing doesn't cover all cases so manual testing with cloud storage is still required.* - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? ### AI Tooling If an AI tool was used: - [X] The PR includes the phrase "Contains content generated by <tool>" where <tool> is the name of the AI tool used. - [ ] My use of AI contributions follows the ASF legal policy https://www.apache.org/legal/generative-tooling.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
