[
https://issues.apache.org/jira/browse/HDFS-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256914#comment-13256914
]
Colin Patrick McCabe commented on HDFS-3290:
--------------------------------------------
Hi Matt,
Thanks for your comments. I misunderstood how the DataNode functions, probably
because I didn't create enough blocks to see the multiple directories.
It seems like we do indeed create multiple directories. However, there is no
way to tell where a given block will be given its ID. So you have to search
every directory to find a given block. This clearly won't scale as the number
of directories increases.
So I guess the title of this JIRA can remain the same-- a "better" directory
structure that doesn't require an O(N / maxBlocksPerDir) search to read a block
would be a good thing...
Colin
> Use a better local directory layout for the datanode
> ----------------------------------------------------
>
> Key: HDFS-3290
> URL: https://issues.apache.org/jira/browse/HDFS-3290
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 0.23.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
>
> When the HDFS DataNode stores chunks in a local directory, it currently puts
> all of the chunk files into one big directory. As the number of files
> increases, this does not work well at all. Local filesystems are not
> optimized for the case where there are hundreds of thousands of files in the
> same directory. It also makes inspecting directories with standard UNIX
> tools difficult.
> Similar to the git version control system, HDFS should create a few different
> top level directories keyed off of a few bits in the chunk ID. Git uses 8
> bits. This substantially cuts down on the number of chunk files in the same
> directory and gives increased performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
