[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15904022#comment-15904022 ]
Aaron Fabbri edited comment on HADOOP-14154 at 3/9/17 10:54 PM: ---------------------------------------------------------------- {quote} Would that isAuthoritative flag have to be setup by higher level applications like Pig/Hive/MR? {quote} No, it is internal to S3A. S3A can tell when it has the full listing for a directory, and simply conveys that to the MetadataStore by setting the isAuthoritative bit. e.g. in {{S3Guard#dirListingUnion(..)}}, it always sets the flag when it puts the listing into the MetadataStore, since this function always has the full listing for the directory: {code} dirMeta.setAuthoritative(true); // This is the full directory contents {code} That codepath happens at the end of listStatus(), when it has finished computing the full directory contents. was (Author: fabbri): {quote} Would that isAuthoritative flag have to be setup by higher level applications like Pig/Hive/MR? {quote} No, it is internal to S3A. S3A can tell when it has the full listing for a directory, and simply conveys that to the MetadataStore by setting the isAuthoritative bit. e.g. in {{S3Guard#dirListingUnion(..)}}, it always sets the flag when it puts the listing into the MetadataStore, since this function always has the full listing for the directory: {code} dirMeta.setAuthoritative(true); // This is the full directory contents {code} > Set isAuthoritative flag when creating DirListingMetadata in DynamoDBMetaStore > ------------------------------------------------------------------------------ > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Reporter: Rajesh Balamohan > Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch > > > Currently {{DynamoDBMetaStore::listChildren}} does not populate > {{isAuthoritative}} flag when creating {{DirListingMetadata}}. > This causes additional S3 lookups even when users have enabled > {{fs.s3a.metadatastore.authoritative}}. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org