[GitHub] [hadoop] steveloughran commented on issue #1861: HADOOP-13230. Optionally retain directory markers
steveloughran commented on issue #1861: HADOOP-13230. Optionally retain directory markers URL: https://github.com/apache/hadoop/pull/1861#issuecomment-615315234 failure in restricted permission test as ls empty dir now succeeds in list (no HEAD, see) ``` [INFO] Running org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationTokens[INFO] Running org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationInFileystem[ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: 31.512 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess[ERROR] testNoReadAccess[raw](org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess) Time elapsed: 26.071 s <<< FAILURE! java.lang.AssertionError: Expected a java.nio.file.AccessDeniedException to be thrown, but got the result: : [Lorg.apache.hadoop.fs.s3a.S3AFileStatus;@53d4d5a at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:499) at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384) at org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess.accessDeniedIf(ITestRestrictedReadAccess.java:697) at org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess.checkBasicFileOperations(ITestRestrictedReadAccess.java:413) at org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess.testNoReadAccess(ITestRestrictedReadAccess.java:298) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1861: HADOOP-13230. Optionally retain directory markers
steveloughran commented on issue #1861: HADOOP-13230. Optionally retain directory markers URL: https://github.com/apache/hadoop/pull/1861#issuecomment-610280135 Operation cost tests failiing; I'm trying to (a) cut down things to a minimum and (b) understand why every head/list call takes place, rather than change the assertions to match what's happening. Hadn't realised until now, for example, that rename(file, dest) has 3 HEAD calls, for example ``` [INFO] [ERROR] Failures: [ERROR] ITestS3AFileOperationCost.testFakeDirectoryDeletion:598->verifyMetrics:490->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 after rename(srcFilePath, destFilePath) [object_delete_requests starting=5 current=8 diff=3, directories_created starting=4 current=5 diff=1, directories_deleted starting=0 current=0 diff=0, fake_directories_deleted starting=31 current=46 diff=15, object_list_requests starting=20 current=23 diff=3, object_metadata_requests starting=18 current=21 diff=3] dest dir depth=8: directories_created expected:<0> but was:<1> [ERROR] ITestS3AFileOperationCost.testFakeDirectoryDeletion:621->verifyMetrics:490->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 after rename(s3a://stevel-london/test/testFakeDirectoryDeletion%5Braw-keep-markers%5D/1/2/3/4/5/6/source2.txt, s3a://stevel-london/test/testFakeDirectoryDeletion[raw-keep-markers]/1/2/3/4/5/6/source.txt) [object_delete_requests starting=1 current=2 diff=1, directories_created starting=4 current=5 diff=1, directories_deleted starting=0 current=0 diff=0, fake_directories_deleted starting=0 current=0 diff=0, object_list_requests starting=25 current=28 diff=3, object_metadata_requests starting=25 current=28 diff=3] dest dir depth=8: directories_created expected:<0> but was:<1> [INFO] ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1861: HADOOP-13230. Optionally retain directory markers
steveloughran commented on issue #1861: HADOOP-13230. Optionally retain directory markers URL: https://github.com/apache/hadoop/pull/1861#issuecomment-609773143 ``` java.lang.NullPointerException at org.apache.hadoop.fs.s3a.S3ListResult.getCommonPrefixes(S3ListResult.java:100) at org.apache.hadoop.fs.s3a.S3ListResult.hasPrefixesOrObjects(S3ListResult.java:143) at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3066) at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2935) at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2795) at org.apache.hadoop.fs.s3a.TestS3AGetFileStatus.testFakeDirectory(TestS3AGetFileStatus.java:84) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1861: HADOOP-13230. Optionally retain directory markers
steveloughran commented on issue #1861: HADOOP-13230. Optionally retain directory markers URL: https://github.com/apache/hadoop/pull/1861#issuecomment-600775261 HADOOP-13230. directory markers the LIST call asks for two objects when needEmptyDirectoryFlag = true, so can distinguish dir marker exists from dir marker exists + children moved much of the prefix/object analysis into S3ListResult where I intend to add some unit tests for the result parsing. Change the enum for all innerGetFileStatus calls from ALL to FILES_AND_DIRECTORIES, as we no longer need to do *any* HEAD / request on a marker; the list finds it after all. There may be more risk of delayed consistency in listings Tests: one mocking test fails (as usual); also failures in ITestS3AFileOperationCost, ITestS3GuardOutOfBandOperations, ITestRestrictedReadAccess ``` [ERROR] Failures: [ERROR] ITestS3AFileOperationCost.testCostOfGetFileStatusOnEmptyDir:161->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_list_requests starting=1 current=2 diff=1: object_list_requests expected:<0> but was:<1> [ERROR] ITestS3AFileOperationCost.testCostOfGetFileStatusOnEmptyDir:159->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_metadata_requests starting=4 current=5 diff=1: object_metadata_requests expected:<2> but was:<1> [ERROR] ITestS3AFileOperationCost.testCostOfGetFileStatusOnMissingFile:180->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_metadata_requests starting=0 current=1 diff=1: object_metadata_requests expected:<2> but was:<1> [ERROR] ITestS3AFileOperationCost.testCostOfGetFileStatusOnMissingFile:180->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_metadata_requests starting=2 current=3 diff=1: object_metadata_requests expected:<2> but was:<1> [ERROR] ITestS3AFileOperationCost.testCostOfGetFileStatusOnMissingSubPath:192->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_metadata_requests starting=0 current=1 diff=1: object_metadata_requests expected:<2> but was:<1> [ERROR] ITestS3AFileOperationCost.testCostOfGetFileStatusOnMissingSubPath:192->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_metadata_requests starting=2 current=3 diff=1: object_metadata_requests expected:<2> but was:<1> [ERROR] ITestS3AFileOperationCost.testCostOfGetFileStatusOnNonEmptyDir:215->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_metadata_requests starting=5 current=6 diff=1: object_metadata_requests expected:<2> but was:<1> [ERROR] ITestS3AFileOperationCost.testCreateCost:511->verifyOperationCount:140->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_metadata_requests starting=2 current=3 diff=1: object_metadata_requests expected:<2> but was:<1> [ERROR] ITestS3AFileOperationCost.testDirProbes:474 [LIST output is not considered empty] Expecting: to match 'is empty' predicate. [ERROR] ITestRestrictedReadAccess.testNoReadAccess:304->checkDeleteOperations:637->accessDenied:680 Expected a java.nio.file.AccessDeniedException to be thrown, but got the result: : true [ERROR] ITestRestrictedReadAccess.testNoReadAccess:298->checkBasicFileOperations:413->accessDeniedIf:697 Expected a java.nio.file.AccessDeniedException to be thrown, but got the result: : [Lorg.apache.hadoop.fs.FileStatus;@1a902257 [ERROR] Errors: [ERROR] ITestS3GuardOutOfBandOperations.testListingDelete:988->expectExceptionWhenReadingOpenFileAPI:1055 ยป Execution ``` the access ones are failing because LIST is working whereas a HEAD would fail if the caller doesn't have read access to the object. OOB ops may be from me setting up a new bucket. Cost ones: well, our costs have come down. literally This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
