[ https://issues.apache.org/jira/browse/MAPREDUCE-6760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated MAPREDUCE-6760: -------------------------------------- Target Version/s: (was: 3.3.0) > LocatedFileStatusFetcher to use listFiles(recursive) > ---------------------------------------------------- > > Key: MAPREDUCE-6760 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6760 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 > Affects Versions: 2.8.0 > Reporter: Steve Loughran > Priority: Major > > {{LocatedFileStatusFetcher}} does parallelized path listing, but it does make > recursive calls on every subdir. > If we could switch it to use {{FileSystem.listFiles(recursive)}}, object > stores that have high-performance implementations of that operation would see > significant speedup. > HADOOP-13208 implements that for S3A; Azure, swift &c can do the same. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org