[
https://issues.apache.org/jira/browse/HADOOP-10987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14192345#comment-14192345
]
Colin Patrick McCabe commented on HADOOP-10987:
-----------------------------------------------
Thanks, Kihwal. +1.
bq. Attaching new patches with the comment fixed. Since there is no actual code
change, I will not make it go through the precommit process. This is the diff:
I'm not sure if we're allowed to skip Jenkins. I submitted the latest patch
for review and will commit when Jenkins gets back. You're welcome to commit
earlier if you think it's acceptable (I just would prefer to be on the safe
side).
> Provide an iterator-based listing API for FileSystem
> ----------------------------------------------------
>
> Key: HADOOP-10987
> URL: https://issues.apache.org/jira/browse/HADOOP-10987
> Project: Hadoop Common
> Issue Type: Improvement
> Affects Versions: 2.7.0
> Reporter: Kihwal Lee
> Assignee: Kihwal Lee
> Attachments: HADOOP-10987.patch, HADOOP-10987.v2.patch,
> HADOOP-10987.v3.patch, HADOOP-10987.v4.branch-2.patch, HADOOP-10987.v4.patch,
> HADOOP-10987.v4_with_comment_fix.branch-2.patch,
> HADOOP-10987.v4_with_comment_fix.patch
>
>
> Iterator based listing methods already exist in {{FileContext}} for both
> simple listing and listing with locations. However, {{FileSystem}} lacks the
> former. From what I understand, it wasn't added to {{FileSystem}} because it
> was believed to be phased out soon. Since {{FileSystem}} is very well alive
> today and new features are getting added frequently, I propose adding an
> iterator based {{listStatus}} method. As for the name of the new method, we
> can use the same name used in {{FileContext}} : {{listStatusIterator()}}.
> It will be particularly useful when listing giant directories. Without this,
> the client has to build up a huge data structure and hold it in memory. We've
> seen client JVMs running out of memory because of this.
> Once this change is made, we can modify FsShell, etc. in followup jiras.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
