steveloughran commented on issue #1861: HADOOP-13230. Optionally retain directory markers URL: https://github.com/apache/hadoop/pull/1861#issuecomment-600775261 HADOOP-13230. directory markers the LIST call asks for two objects when needEmptyDirectoryFlag = true, so can distinguish dir marker exists from dir marker exists + children moved much of the prefix/object analysis into S3ListResult where I intend to add some unit tests for the result parsing. Change the enum for all innerGetFileStatus calls from ALL to FILES_AND_DIRECTORIES, as we no longer need to do *any* HEAD / request on a marker; the list finds it after all. There may be more risk of delayed consistency in listings Tests: one mocking test fails (as usual); also failures in ITestS3AFileOperationCost, ITestS3GuardOutOfBandOperations, ITestRestrictedReadAccess ``` [ERROR] Failures: [ERROR] ITestS3AFileOperationCost.testCostOfGetFileStatusOnEmptyDir:161->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_list_requests starting=1 current=2 diff=1: object_list_requests expected:<0> but was:<1> [ERROR] ITestS3AFileOperationCost.testCostOfGetFileStatusOnEmptyDir:159->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_metadata_requests starting=4 current=5 diff=1: object_metadata_requests expected:<2> but was:<1> [ERROR] ITestS3AFileOperationCost.testCostOfGetFileStatusOnMissingFile:180->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_metadata_requests starting=0 current=1 diff=1: object_metadata_requests expected:<2> but was:<1> [ERROR] ITestS3AFileOperationCost.testCostOfGetFileStatusOnMissingFile:180->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_metadata_requests starting=2 current=3 diff=1: object_metadata_requests expected:<2> but was:<1> [ERROR] ITestS3AFileOperationCost.testCostOfGetFileStatusOnMissingSubPath:192->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_metadata_requests starting=0 current=1 diff=1: object_metadata_requests expected:<2> but was:<1> [ERROR] ITestS3AFileOperationCost.testCostOfGetFileStatusOnMissingSubPath:192->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_metadata_requests starting=2 current=3 diff=1: object_metadata_requests expected:<2> but was:<1> [ERROR] ITestS3AFileOperationCost.testCostOfGetFileStatusOnNonEmptyDir:215->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_metadata_requests starting=5 current=6 diff=1: object_metadata_requests expected:<2> but was:<1> [ERROR] ITestS3AFileOperationCost.testCreateCost:511->verifyOperationCount:140->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_metadata_requests starting=2 current=3 diff=1: object_metadata_requests expected:<2> but was:<1> [ERROR] ITestS3AFileOperationCost.testDirProbes:474 [LIST output is not considered empty] Expecting: <S3AFileStatus{path=s3a://stevel-london/fork-0001/test/testEmptyDirHeadProbe; isDirectory=true; modification_time=0; access_time=0; owner=stevel; group=stevel; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=TRUE eTag=null versionId=null> to match 'is empty' predicate. [ERROR] ITestRestrictedReadAccess.testNoReadAccess:304->checkDeleteOperations:637->accessDenied:680 Expected a java.nio.file.AccessDeniedException to be thrown, but got the result: : true [ERROR] ITestRestrictedReadAccess.testNoReadAccess:298->checkBasicFileOperations:413->accessDeniedIf:697 Expected a java.nio.file.AccessDeniedException to be thrown, but got the result: : [Lorg.apache.hadoop.fs.FileStatus;@1a902257 [ERROR] Errors: [ERROR] ITestS3GuardOutOfBandOperations.testListingDelete:988->expectExceptionWhenReadingOpenFileAPI:1055 ยป Execution ``` the access ones are failing because LIST is working whereas a HEAD would fail if the caller doesn't have read access to the object. OOB ops may be from me setting up a new bucket. Cost ones: well, our costs have come down. literally
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
