[
https://issues.apache.org/jira/browse/HDFS-15196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064011#comment-17064011
]
Ayush Saxena commented on HDFS-15196:
-------------------------------------
For the end user, the remaining entries aren't distinguished whether it is is
from namenodes or are mount entries.
When all entries from namenode are done, aren't we appending all the mount
entries to the result? Then how it will shoot an extra call?
As now we send the mount entries which land between the first and last of the
entries from namenode, Say for namenode 5 entries are remaining, there may be
another 10 mount entries too, remaining after the {{last}} that we checked,
isn't it?
> RBF: RouterRpcServer getListing cannot list large dirs correctly
> ----------------------------------------------------------------
>
> Key: HDFS-15196
> URL: https://issues.apache.org/jira/browse/HDFS-15196
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Fengnan Li
> Assignee: Fengnan Li
> Priority: Critical
> Attachments: HDFS-15196.001.patch, HDFS-15196.002.patch,
> HDFS-15196.003.patch, HDFS-15196.003.patch, HDFS-15196.004.patch,
> HDFS-15196.005.patch, HDFS-15196.006.patch, HDFS-15196.007.patch,
> HDFS-15196.008.patch, HDFS-15196.009.patch, HDFS-15196.010.patch
>
>
> In RouterRpcServer, getListing function is handled as two parts:
> # Union all partial listings from destination ns + paths
> # Append mount points for the dir to be listed
> In the case of large dir which is bigger than DFSConfigKeys.DFS_LIST_LIMIT
> (with default value 1k), the batch listing will be used and the startAfter
> will be used to define the boundary of each batch listing. However, step 2
> here will add existing mount points, which will mess up with the boundary of
> the batch, thus making the next batch startAfter wrong.
> The fix is just to append the mount points when there is no more batch query
> necessary.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
