[
https://issues.apache.org/jira/browse/HDFS-5788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13874107#comment-13874107
]
Nathan Roberts commented on HDFS-5788:
--------------------------------------
A simple solution is:
Restrict the size to dfs.ls.limit (default 1000) files OR dfs.ls.limit block
locations, whichever comes first (obviously always returning only whole
entries, so we could send more than this number of locations)
Yes, it will require more RPCs. However, it would seem to lower the risk of a
DoS.
> listLocatedStatus response can be very large
> --------------------------------------------
>
> Key: HDFS-5788
> URL: https://issues.apache.org/jira/browse/HDFS-5788
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 3.0.0, 0.23.10, 2.2.0
> Reporter: Nathan Roberts
> Assignee: Nathan Roberts
>
> Currently we limit the size of listStatus requests to a default of 1000
> entries. This works fine except in the case of listLocatedStatus where the
> location information can be quite large. As an example, a directory with 7000
> entries, 4 blocks each, 3 way replication - a listLocatedStatus response is
> over 1MB. This can chew up very large amounts of memory in the NN if lots of
> clients try to do this simultaneously.
> Seems like it would be better if we also considered the amount of location
> information being returned when deciding how many files to return.
> Patch will follow shortly.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
