[ https://issues.apache.org/jira/browse/HADOOP-15370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16696674#comment-16696674 ]
Gabor Bota commented on HADOOP-15370: ------------------------------------- mvn verify ran against eu-west-1. s3guard: dynamo, non-auth and auth after that. Results are not looking good. For non-authoritative: {noformat} [ERROR] Failures: [ERROR] ITestDynamoDBMetadataStore>MetadataStoreTestBase.testDeleteSubtree:321->MetadataStoreTestBase.deleteSubtreeHelper:346->MetadataStoreTestBase.assertEmptyDirectory:927->MetadataStoreTestBase.assertDirectorySize:882->Assert.assertNotNull:712->Assert.assertTrue:41->Assert.fail:88 Directory /ADirectory2 is null in cache [ERROR] ITestDynamoDBMetadataStore>MetadataStoreTestBase.testDeleteSubtreeHostPath:326->MetadataStoreTestBase.deleteSubtreeHelper:346->MetadataStoreTestBase.assertEmptyDirectory:927->MetadataStoreTestBase.assertDirectorySize:882->Assert.assertNotNull:712->Assert.assertTrue:41->Assert.fail:88 Directory s3a://cloudera-dev-gabor-ireland/ADirectory2 is null in cache [ERROR] ITestDynamoDBMetadataStore.testProvisionTable:594->Assert.assertEquals:631->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 expected:<20> but was:<10> [ERROR] ITestDynamoDBMetadataStore>MetadataStoreTestBase.testPutNew:243->MetadataStoreTestBase.assertEmptyDirs:932->MetadataStoreTestBase.assertEmptyDirectory:927->MetadataStoreTestBase.assertDirectorySize:882->Assert.assertNotNull:712->Assert.assertTrue:41->Assert.fail:88 Directory /da2 is null in cache [ERROR] Errors: [ERROR] ITestDynamoDBMetadataStore.testBatchWrite:318->doTestBatchWrite:365 NullPointer [ERROR] ITestDynamoDBMetadataStore>MetadataStoreTestBase.testPruneUnsetsAuthoritative:737 ? NullPointer [ERROR] ITestS3GuardToolDynamoDB>AbstractS3GuardToolTestBase.testDestroyNoBucket:336->AbstractS3GuardToolTestBase.run:115 ? IllegalArgument [ERROR] ITestS3AMiniYarnCluster.setup:68 ? NoClassDefFound org/bouncycastle/jce/provid... {noformat} I've seen {{ITestS3AMiniYarnCluster#testWithMiniCluster}} and {{ITestS3GuardToolDynamoDB#testDestroyNoBucket}} but not the others. These failures/errors are not related to the patch (obviously). I did a regression testing with clean trunk, and got the same results. Failures/errors with non-authoritative: {noformat} [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 19.912 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster [ERROR] testWithMiniCluster(org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster) Time elapsed: 19.697 s <<< ERROR! java.lang.NoClassDefFoundError: org/bouncycastle/jce/provider/BouncyCastleProvider at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:840) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1270) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:328) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.yarn.server.MiniYARNCluster.initResourceManager(MiniYARNCluster.java:348) at org.apache.hadoop.yarn.server.MiniYARNCluster.access$200(MiniYARNCluster.java:128) at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceInit(MiniYARNCluster.java:497) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108) at org.apache.hadoop.yarn.server.MiniYARNCluster.serviceInit(MiniYARNCluster.java:316) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster.setup(ITestS3AMiniYarnCluster.java:68) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassNotFoundException: org.bouncycastle.jce.provider.BouncyCastleProvider at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 27 more {noformat} {noformat} ERROR] Tests run: 40, Failures: 4, Errors: 2, Skipped: 0, Time elapsed: 177.303 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore [ERROR] testBatchWrite(org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore) Time elapsed: 1.262 s <<< ERROR! java.lang.NullPointerException at org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore.doTestBatchWrite(ITestDynamoDBMetadataStore.java:365) (...) [ERROR] testProvisionTable(org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore) Time elapsed: 11.394 s <<< FAILURE! java.lang.AssertionError: expected:<20> but was:<10> (...) org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore.testProvisionTable(ITestDynamoDBMetadataStore.java:594) (...) [ERROR] testPruneUnsetsAuthoritative(org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore) Time elapsed: 2.323 s <<< ERROR! java.lang.NullPointerException at org.apache.hadoop.fs.s3a.s3guard.MetadataStoreTestBase.testPruneUnsetsAuthoritative(MetadataStoreTestBase.java:737) (...) [ERROR] testDeleteSubtree(org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore) Time elapsed: 2.377 s <<< FAILURE! java.lang.AssertionError: Directory /ADirectory2 is null in cache (...) [ERROR] testPutNew(org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore) Time elapsed: 1.466 s <<< FAILURE! java.lang.AssertionError: Directory /da2 is null in cache (...) [ERROR] testDeleteSubtreeHostPath(org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore) Time elapsed: 2.378 s <<< FAILURE! java.lang.AssertionError: Directory s3a://cloudera-dev-gabor-ireland/ADirectory2 is null in cache (...) {noformat} {noformat} [ERROR] testDestroyNoBucket(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB) Time elapsed: 1.921 s <<< ERROR! java.lang.IllegalArgumentException: No DynamoDB table name configured (...) {noformat} I've created a gist with the results for authoritative and non-authoritative: https://gist.github.com/bgaborg/e0d8092659cf009fcd2985ca95c574be > S3A log message on rm s3a://bucket/ not intuitive > ------------------------------------------------- > > Key: HADOOP-15370 > URL: https://issues.apache.org/jira/browse/HADOOP-15370 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 3.1.0 > Reporter: Steve Loughran > Assignee: Gabor Bota > Priority: Trivial > Attachments: HADOOP-15370.001.patch > > > when you try to delete the root of a bucket from command line, e.g. {{hadoop > fs -rm -r -skipTrash s3a://hwdev-steve-new/}}, the output isn't that useful > {code} > 2018-04-06 16:35:23,048 [main] INFO s3a.S3AFileSystem > (S3AFileSystem.java:rejectRootDirectoryDelete(1837)) - s3a delete the > hwdev-steve-new root directory of true > rm: `s3a://hwdev-steve-new/': Input/output error > 2018-04-06 16:35:23,050 [pool-2-thread-1] DEBUG s3a.S3AFileSystem > {code} > the single log message doesn't parse, and the error message raised is lost by > the FS -rm CLI command (why?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org