[ https://issues.apache.org/jira/browse/HADOOP-16801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mustafa Iman reassigned HADOOP-16801: ------------------------------------- Assignee: Mustafa Iman > S3Guard queries S3 with recursive file listings > ----------------------------------------------- > > Key: HADOOP-16801 > URL: https://issues.apache.org/jira/browse/HADOOP-16801 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 3.3.0 > Reporter: Mustafa Iman > Assignee: Mustafa Iman > Priority: Minor > Attachments: HADOOP-aws-no-prefetch.prelim.patch > > > S3AFileSystem#listFiles with recursive option, queries S3 even when directory > listing is authoritative. FileStatusListingIterator is created with given > entries from metadata store > [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Listing.java#L126] > . However, FileStatusListingIterator has an ObjectListingIterator that > prefetches from s3 regardless of authoritative listing. We observed this > behavior when using DynamDBMetadataStore. > I suppressed the unnecessary S3 calls by providing a dumb listing iterator to > listFiles call in the provided patch. Obviously this is not a solution. Just > demonstrating the source of the problem. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org