[ https://issues.apache.org/jira/browse/HADOOP-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15798844#comment-15798844 ]
Mingliang Liu commented on HADOOP-13934: ---------------------------------------- {{FileSystemContractBaseTest#testRenameToDirWithSamePrefixAllowed}} fails occasionally with the following errors: {code} $ mvn -Dtest=none -Dit.test='ITestS3AFileSystemContract' -q -Dscale clean verify Running org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract Tests run: 43, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 74.348 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract testRenameToDirWithSamePrefixAllowed(org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract) Time elapsed: 0.556 sec <<< FAILURE! junit.framework.AssertionFailedError: Rename result expected:<true> but was:<false> at junit.framework.Assert.fail(Assert.java:57) at junit.framework.Assert.failNotEquals(Assert.java:329) at junit.framework.Assert.assertEquals(Assert.java:78) at junit.framework.Assert.assertEquals(Assert.java:174) at junit.framework.TestCase.assertEquals(TestCase.java:333) at org.apache.hadoop.fs.FileSystemContractBaseTest.rename(FileSystemContractBaseTest.java:525) at org.apache.hadoop.fs.FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed(FileSystemContractBaseTest.java:669) {code} It seems the working directories in {{FileSystemContractBaseTest}} is shared across different test cases. Then I applied the following patch to use unique test directory for test {{testRenameToDirWithSamePrefixAllowed}}. {code:title=hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java} @@ -663,9 +663,9 @@ public void testRenameChildDirForbidden() throws Exception { */ public void testRenameToDirWithSamePrefixAllowed() throws Throwable { if (!renameSupported()) return; - Path parentdir = path("test/parentdir"); + Path parentdir = path("testRenameToDirWithSamePrefixAllowed/parentdir"); fs.mkdirs(parentdir); - Path dest = path("test/parentdirdest"); + Path dest = path("testRenameToDirWithSamePrefixAllowed/parentdirdest"); rename(parentdir, dest, true, false, true); } {code} After that, the test {{FileSystemContractBaseTest#testRenameToDirWithSamePrefixAllowed}} is failing occasionally with different error. The reason was that the parameters created in {{S3AFileSystem#innerRename}} passing to {{DynamoDBMetadataStore#move()}} are lists that may contain duplicate paths (and thus duplicate DDB primary key). Per offline discussion with [~ste...@apache.org] and [~rajesh.balamohan], the v4 patch de-duplicates items of the same primary key in DynamoDBMetadataStore side, but it is using auxiliary hashset so memory usage is higher. {code} $ mvn -Dtest=none -Dit.test='ITestS3AFileSystemContract' -q -Dscale clean verify Running org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract Tests run: 43, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 70.52 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract testRenameToDirWithSamePrefixAllowed(org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract) Time elapsed: 1.347 sec <<< ERROR! org.apache.hadoop.fs.s3a.AWSServiceIOException: move: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Provided list of item keys contains duplicates (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: 3MED8L2MSBF0SH8S4U5OK0TGQ3VV4KQNSO5AEMVJF66Q9ASUAAJG): Provided list of item keys contains duplicates (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: 3MED8L2MSBF0SH8S4U5OK0TGQ3VV4KQNSO5AEMVJF66Q9ASUAAJG) at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:171) at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.move(DynamoDBMetadataStore.java:363) at org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:862) at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:655) at org.apache.hadoop.fs.FileSystemContractBaseTest.rename(FileSystemContractBaseTest.java:525) at org.apache.hadoop.fs.FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed(FileSystemContractBaseTest.java:669) {code} This problem is not related to the scope of this patch though. We can address the problem in {{S3AFileSystem#innerRename}} or in test separately instead. We can stick with the v3 patch I think. > S3Guard: DynamoDBMetadataStore#move() could be throwing exception due to > BatchWriteItem limits > ---------------------------------------------------------------------------------------------- > > Key: HADOOP-13934 > URL: https://issues.apache.org/jira/browse/HADOOP-13934 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: HADOOP-13345 > Reporter: Rajesh Balamohan > Assignee: Mingliang Liu > Priority: Minor > Attachments: HADOOP-13934-HADOOP-13345.000.patch, > HADOOP-13934-HADOOP-13345.001.patch, HADOOP-13934-HADOOP-13345.002.patch, > HADOOP-13934-HADOOP-13345.003.patch, HADOOP-13934-HADOOP-13345.004.patch > > > When using {{DynamoDBMetadataStore}} with a insert heavy hive app , it > started throwing exceptions in {{DynamoDBMetadataStore::move}}. But just with > the following exception, it is relatively hard to debug on the real issue in > DynamoDB side. > {noformat} > Caused by: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: 1 > validation error detected: Value > '{ddb-table-name-334=[com.amazonaws.dynamodb.v20120810.WriteRequest@ca1da583, > com.amazonaws.dynamodb.v20120810.WriteRequest@ca1fc7cd, > com.amazonaws.dynamodb.v20120810.WriteRequest@ca4244e6, > com.amazonaws.dynamodb.v20120810.WriteRequest@ca2f58a9, > com.amazonaws.dynamodb.v20120810.WriteRequest@ca3525f8, > ... > ... > at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1529) > at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1167) > at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948) > at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661) > at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635) > at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618) > at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586) > at > com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573) > at > com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445) > at > com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1722) > at > com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1698) > at > com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.batchWriteItem(AmazonDynamoDBClient.java:668) > at > com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.doBatchWriteItem(BatchWriteItemImpl.java:111) > at > com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.batchWriteItem(BatchWriteItemImpl.java:52) > at > com.amazonaws.services.dynamodbv2.document.DynamoDB.batchWriteItem(DynamoDB.java:178) > at > org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.move(DynamoDBMetadataStore.java:351) > ... 28 more > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org