steveloughran commented on a change in pull request #1815: HADOOP-16801:
S3Guard-listFiles will not query s3 if all listings are…
URL: https://github.com/apache/hadoop/pull/1815#discussion_r370307065
##########
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreAuthoritativeMode.java
##########
@@ -291,6 +295,102 @@ public void testListStatusMakesEmptyDirAuth() throws
Throwable {
assertListDoesNotUpdateAuth(dir);
}
+ @Test
+ public void testListFilesRecursiveWhenAllListingsAreAuthoritative()
+ throws Exception {
+ describe("listFiles does not make further calls to the fs when"
+ + "all nested directory listings are authoritative");
+ Set<Path> files = new HashSet<>();
+
+ Path parentDir = dir;
+ Path parentFile = dirFile;
+ Path nestedDir1 = new Path(dir, "nested1");
+ Path nestedFile1 = new Path(nestedDir1, "nestedFile1");
+ Path nestedDir2 = new Path(nestedDir1, "nested2/");
+ Path nestedFile2 = new Path(nestedDir2, "nestedFile2");
+
+ files.add(parentFile);
+ files.add(nestedFile1);
+ files.add(nestedFile2);
+
+ authFS.mkdirs(parentDir);
+ authFS.mkdirs(nestedDir1);
+ authFS.mkdirs(nestedDir2);
+ touchFile(parentFile);
+ touchFile(nestedFile1);
+ touchFile(nestedFile2);
+
+ // making listStatus call to mark directories authoritative
+ authFS.listStatus(parentDir);
+ authFS.listStatus(nestedDir1);
+ authFS.listStatus(nestedDir2);
+
+ S3AStorageStatistics statistics = authFS.getStorageStatistics();
+ statistics.reset();
+
+ RemoteIterator<LocatedFileStatus> statusIterator =
+ authFS.listFiles(dir, true);
+
+ while (statusIterator.hasNext()) {
+ LocatedFileStatus locatedFileStatus = statusIterator.next();
+ assertTrue("This path does not exist in original listing: " +
+ locatedFileStatus.getPath(),
+ files.remove(locatedFileStatus.getPath()));
+ }
+ assertEquals("Some files were missing from authoritative listing:"
+ + Arrays.toString(files.toArray()), 0, files.size());
Review comment:
assertJ's assert will list the array, so is needed here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
