[ https://issues.apache.org/jira/browse/HADOOP-14266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15969811#comment-15969811 ]
Mingliang Liu commented on HADOOP-14266: ---------------------------------------- [~fabbri] your comments are very precise. Thanks for the clear explanation! I suggest we update the description of this JIRA using most of the above comments when the patch is final. After reading your comment, I also have two basic ideas to optimize further along with your proposed future enhancement. # For the {{!recursive && isAuthoritative}} case, we can return metadata store cachedFilesIterator results without asking S3. This will be similar to {{listLocatedStatus()}}. # If we have returned value order guarantee from both S3 list object request and metadata store {{DescendantsIterator}}, in {{FileStatusListingIterator}} we can maintain two moving iterators and avoid pre-iterating providedStatus, in which way it uses much less memory (no providedStatus HashSet then). I will re-visit these two ideas again next week. Perhaps we can address them later. > S3Guard: S3AFileSystem::listFiles() to employ MetadataStore > ----------------------------------------------------------- > > Key: HADOOP-14266 > URL: https://issues.apache.org/jira/browse/HADOOP-14266 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: HADOOP-13345 > Reporter: Mingliang Liu > Assignee: Mingliang Liu > Attachments: HADOOP-14266-HADOOP-13345.000.patch, > HADOOP-14266-HADOOP-13345.001.patch, HADOOP-14266-HADOOP-13345.002.patch, > HADOOP-14266-HADOOP-13345.003.patch, HADOOP-14266-HADOOP-13345.003.patch, > HADOOP-14266-HADOOP-13345.004.patch, HADOOP-14266-HADOOP-13345-005.patch, > HADOOP-14266-HADOOP-13345.005.patch, HADOOP-14266-HADOOP-13345.006.patch > > > Similar to [HADOOP-13926], this is to track the effort of employing > MetadataStore in {{S3AFileSystem::listFiles()}}. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org