adoroszlai opened a new pull request, #4287: URL: https://github.com/apache/ozone/pull/4287
## What changes were proposed in this pull request? Fix intermittent failures in `ITestOzoneContractRootDir` and `ITestOzoneContractDelete`, which were caused by `KeyManagerImpl#createFakeDirIfShould` due to: 1. `OMKeysDeleteRequest` first marks keys deleted in the cache, removal is propagated to the DB in a background thread. `createFakeDirIfShould` checks the key table, but not the cache. Thus it intermittently finds the deleted entry, returning a fake dir that might no longer be needed. 2. `createFakeDirIfShould` may even return a fake dir for a normal directory that was just deleted. ``` [JUnit-testMkDirDepth1[0]] TRACE (BasicOzoneFileSystem.java:mkdirs(761)) - mkdir() path:/testmkdirdepth1 [JUnit-testMkDirDepth1[0]] TRACE (BasicOzoneClientAdapterImpl.java:createDirectory(312)) - creating dir for key:testmkdirdepth1 [OM StateMachine ApplyTransaction Thread - 0] DEBUG (OMDirectoryCreateRequest.java:logResult(323)) - Directory created. Volume:volume21291, Bucket:bucket26900, Key:testmkdirdepth1 [JUnit-testMkDirDepth1[0]] DEBUG (BasicOzoneFileSystem.java:delete(540)) - Delete path /testmkdirdepth1 - recursive true [JUnit-testMkDirDepth1[0]] DEBUG (BasicOzoneFileSystem.java:delete(567)) - delete: Path is a directory: /testmkdirdepth1 [JUnit-testMkDirDepth1[0]] TRACE (BasicOzoneFileSystem.java:innerDelete(518)) - delete() path:/testmkdirdepth1 recursive:true [JUnit-testMkDirDepth1[0]] TRACE (BasicOzoneFileSystem.java:processKey(501)) - deleting key:[testmkdirdepth1/] [OM StateMachine ApplyTransaction Thread - 0] DEBUG (OMKeysDeleteRequest.java:validateAndUpdateCache(223)) - Keys delete success. Volume:volume21291, Bucket:bucket26900, Keys:testmkdirdepth1/ [JUnit-testMkDirDepth1[0]] TRACE (BasicOzoneFileSystem.java:getFileStatus(779)) - getFileStatus() path:/testmkdirdepth1 [JUnit-testMkDirDepth1[0]] TRACE (BasicOzoneFileSystem.java:pathToKey(1046)) - path for key:/testmkdirdepth1 is:o3fs://bucket26900.volume21291/testmkdirdepth1 [IPC Server handler 2 on default port 45085] DEBUG (KeyManagerImpl.java:createFakeDirIfShould(1193)) - Returning fake dir testmkdirdepth1/ for /volume21291/bucket26900/testmkdirdepth1/ -> /volume21291/bucket26900/testmkdirdepth1/ ``` The PR proposes to fix this by: 1. requiring that the item which makes the fake dir necessary is not deleted, 2. requiring the fake dir to be strictly a prefix of the item, not equal to it. Also check the cache for any matching entries first. Also refactor contract tests to use `BeforeParam` and `AfterParam` to hook into the JUnit test lifecycle, instead of restarting the cluster from the test constructor (this turned out not to be the cause of the bug, but worth keeping). https://issues.apache.org/jira/browse/HDDS-7991 ## How was this patch tested? Executed the intermittent contract tests repeatedly: https://github.com/adoroszlai/hadoop-ozone/actions/runs/4216890609 (100/100 passed) https://github.com/adoroszlai/hadoop-ozone/actions/runs/4217247319 (100/100 passed) https://github.com/adoroszlai/hadoop-ozone/actions/runs/4217317817 (non-FSO only, 100/100 passed) It probably also fixes HDDS-7940, so also included `TestOzoneFileSystem` and `TestRootedOzoneFileSystem` in one batch: https://github.com/adoroszlai/hadoop-ozone/actions/runs/4217350948 (90/91 passed; one failure in `TestOzoneFileSystem.testListStatusWithIntermediateDir`, which doesn't include delete, hence not affected by this bug/fix) Regular CI: https://github.com/adoroszlai/hadoop-ozone/actions/runs/4217340348 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
