[
https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15902191#comment-15902191
]
Aaron Fabbri commented on HADOOP-14154:
---------------------------------------
Hi [~rajesh.balamohan]. Thanks for the patch. This is an area we plan to
improve (to allow S3A to avoid round trips to s3 when MetadataStore has full
listing).
I think you're confusing the "isAuthoritative" bit on directories with the
"allow authoritative" config parameter for s3a. I hope some existing
integration tests would fail with this patch to catch this.
The "isAuthoritative" flag is, essentially, a boolean on a DirListingMetadata
that is used by the FS client to tell the MetadataStore (MS) that "this is the
full listing for this directory". If a client does put() with a
DirListingMetadata that has isAuthoritative == true, then subsequent
listChildren() *should* return the listing with isAuthoritative == true. A
MetadataStore *may* chose not to persist that isAuthoritative bit on the
directory listing, though. This is what DDB does today: it does not store the
isAuthoritative bit on directories. Thus, S3A will always check S3 as well as
DDB MS because DDB doesn't remember that it was given a full listing.
A MetadataStore *must not* return isAuthoritative==true for listChildren()
unless the FS client previously set that bit on the directory. The MS only
knows that it has the full listing if the FS client tells it.
The "allow authoritative" config flag for S3A, when true, tells S3A that it may
skip the list operation on S3 if the MS returns a full directory listing
(isAuthoritative==true on the DirListingMetadata). So that config flag should
only be used in S3A.
The correct way to address this issue is to have DDB MS persist the
isAuthoritative bit on directories.
Does that help clarify the architecture here?
> Set isAuthoritative flag when creating DirListingMetadata in DynamoDBMetaStore
> ------------------------------------------------------------------------------
>
> Key: HADOOP-14154
> URL: https://issues.apache.org/jira/browse/HADOOP-14154
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Rajesh Balamohan
> Priority: Minor
> Attachments: HADOOP-14154-HADOOP-13345.001.patch,
> HADOOP-14154-HADOOP-13345.002.patch
>
>
> Currently {{DynamoDBMetaStore::listChildren}} does not populate
> {{isAuthoritative}} flag when creating {{DirListingMetadata}}.
> This causes additional S3 lookups even when users have enabled
> {{fs.s3a.metadatastore.authoritative}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
