[
https://issues.apache.org/jira/browse/HADOOP-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-13449:
-----------------------------------
Attachment: HADOOP-13449-HADOOP-13345.013.patch
Thanks [~fabbri].
I saw you created two follow up JIRA, which are very good. We can address them
separately. I also have the v13 patch, which can be addressed separately as
well as the changes are kinda isolated to v12 patch. Specially, the v13 patch
addresses the trivial checkstyle warning, and changes the region of DynamoDB
table to create from region of S3Client to region of Bucket. I think it makes
more sense.
For the failing integration test
{{ITestS3AFileOperationCost#testFakeDirectoryDeletion}}, I found it fails with
LocalMetadataStore as well, can you verify that? The reason it fails is because
the number of create operations counter does not match after rename.
{code:title=ITestS3AFileOperationCost#testFakeDirectoryDeletion()}
fs.rename(srcFilePath, destFilePath);
state = "after rename(srcFilePath, destFilePath)";
directoriesCreated.assertDiffEquals(state, 1); <=== fail here
{code}
When the S3AFS renames a file (in this case), it will create fake parent
directories after deleting the old file when necessary. This is guarded by
{{createFakeDirectoryIfNecessary}}, which checks if the parent directory exits.
{code:title=createFakeDirectoryIfNecessary() called by rename()}
private void createFakeDirectoryIfNecessary(Path f)
throws IOException, AmazonClientException {
String key = pathToKey(f);
if (!key.isEmpty() && !exists(f)) { <=== only if nonexistent
LOG.debug("Creating new fake directory at {}", f);
createFakeDirectory(key);
}
}
{code}
However, getFileStatus() for exists() will check the metadata store only for
this. The metadatastore surely will return true and no fake directory will be
created in S3 then.
{code}
public S3AFileStatus getFileStatus(final Path f) throws IOException {
incrementStatistic(INVOCATION_GET_FILE_STATUS);
final Path path = qualify(f);
String key = pathToKey(path);
LOG.debug("Getting path status for {} ({})", path , key);
// Check MetadataStore, if any.
PathMetadata pm = metadataStore.get(path);
if (pm != null) {
// HADOOP-13760: handle deleted files, i.e. PathMetadata#isDeleted() here
return (S3AFileStatus)pm.getFileStatus();
}
...
{code}
This seems a bug elesewhere and can also be addressed/investigated separately.
> S3Guard: Implement DynamoDBMetadataStore.
> -----------------------------------------
>
> Key: HADOOP-13449
> URL: https://issues.apache.org/jira/browse/HADOOP-13449
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Chris Nauroth
> Assignee: Mingliang Liu
> Attachments: HADOOP-13449-HADOOP-13345.000.patch,
> HADOOP-13449-HADOOP-13345.001.patch, HADOOP-13449-HADOOP-13345.002.patch,
> HADOOP-13449-HADOOP-13345.003.patch, HADOOP-13449-HADOOP-13345.004.patch,
> HADOOP-13449-HADOOP-13345.005.patch, HADOOP-13449-HADOOP-13345.006.patch,
> HADOOP-13449-HADOOP-13345.007.patch, HADOOP-13449-HADOOP-13345.008.patch,
> HADOOP-13449-HADOOP-13345.009.patch, HADOOP-13449-HADOOP-13345.010.patch,
> HADOOP-13449-HADOOP-13345.011.patch, HADOOP-13449-HADOOP-13345.012.patch,
> HADOOP-13449-HADOOP-13345.013.patch
>
>
> Provide an implementation of the metadata store backed by DynamoDB.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
