Fengnan Li created HDFS-15196:
---------------------------------
Summary: RouterRpcServer getListing cannot list large dirs
correctly
Key: HDFS-15196
URL: https://issues.apache.org/jira/browse/HDFS-15196
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Fengnan Li
Assignee: Fengnan Li
In RouterRpcServer, getListing function is handled as two parts:
# Union all partial listings from destination ns + paths
# Append mount points for the dir to be listed
In the case of large dir which is bigger than DFSConfigKeys.DFS_LIST_LIMIT
(with default value 1k), the batch listing will be used and the startAfter will
be used to define the boundary of each batch listing. However, step 2 here will
add existing mount points, which will mess up with the boundary of the batch,
thus making the next batch startAfter wrong.
The fix is just to append the mount points when there is no more batch query
necessary.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
