[
https://issues.apache.org/jira/browse/HDFS-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007802#comment-13007802
]
dhruba borthakur commented on HDFS-1751:
----------------------------------------
can somebody pl explain the use-case why we need to limit the maximum number of
files in a specific directory? Unlike other filesystems (ext3, vxfs, etc), HDFS
does not have the concept of indirect blocks, thus there is no overhead from
having all the files in one directory versus having the same number of files
spread out in different directories.
The only resource limitation I can think about is getListStatus() on the
directory could return large number of files, but since this is now handled via
an iterative getListStatus RPC, this should not be a problem.
> Intrinsic limits for HDFS files, directories
> --------------------------------------------
>
> Key: HDFS-1751
> URL: https://issues.apache.org/jira/browse/HDFS-1751
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Fix For: 0.23.0
>
> Attachments: HDFS-1751-2.patch, HDFS-1751-3.patch, HDFS-1751-4.patch,
> HDFS-1751.patch
>
>
> Enforce a configurable limit on:
> the length of a path component
> the number of names in a directory
> The intention is to prevent a too-long name or a too-full directory. This is
> not about RPC buffers, the length of command lines, etc. There may be good
> reasons for those kinds of limits, but that is not the intended scope of this
> feature. Consequently, a reasonable implementation might be to extend the
> existing quota checker so that it faults the creation of a name that violates
> the limits. This strategy of faulting new creation evades the problem of
> existing names or directories that violate the limits.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
