[
https://issues.apache.org/jira/browse/HADOOP-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15668739#comment-15668739
]
Aaron Fabbri commented on HADOOP-13449:
---------------------------------------
Thanks [~liuml07], sounds good. Can't wait to try this stuff out.
{quote}
My final concern is that, the MetadataStore assumes all ancestor directories
(including direct parent directory) have been pre-created by the caller/user. I
have to change the base test MetadataStoreTestBase to make all the tests pass.
We have to change LocalMetadataStore accordingly. See my above comments.
{quote}
I looked in the test code and did not see the bug you mention. Feel free to
call out a particular line of code: I may have missed it.
This is the intention: Say you start by doing a
MetadataStore.put(PathMetadata(/a/b/file)).
That file should appear when listing /a/b.
That is, MetadataStore.listChildren(/a/b) should return {[/a/b/file],
isAuthoritative=false}. This is used by S3AFileSystem#listStatus() to get list
consistency (e.g. if /a/b/file is not visible in s3 yet, we will add that
entry).
However, there is no requirement that MetadataStore.get(/a/b) should return
anything. So the parent directory does not need to be created, per se, but the
new file *does* need to appear when listing the parent.
If the MetadataStore contract tests do not reflect this, I am very happy to
change them. Also you might have missed some other test code (helper
functions) that explicitly create parent dirs. We don't want to assume
creation of ancestors at all, just file to show up in parent listing
The LocalMetadataStore ends up creating a parent dir internally, but that
should be implementation detail, and not required as above.
Does that help clarify?
> S3Guard: Implement DynamoDBMetadataStore.
> -----------------------------------------
>
> Key: HADOOP-13449
> URL: https://issues.apache.org/jira/browse/HADOOP-13449
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Chris Nauroth
> Assignee: Mingliang Liu
> Attachments: HADOOP-13449-HADOOP-13345.000.patch,
> HADOOP-13449-HADOOP-13345.001.patch, HADOOP-13449-HADOOP-13345.002.patch,
> HADOOP-13449-HADOOP-13345.003.patch, HADOOP-13449-HADOOP-13345.004.patch
>
>
> Provide an implementation of the metadata store backed by DynamoDB.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
