[
https://issues.apache.org/jira/browse/HDFS-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17412801#comment-17412801
]
Renukaprasad C commented on HDFS-14703:
---------------------------------------
Thanks [~jianghuazhu] for sharing your thoughts. Hope this will clarify your
doubts.
INodeMap#NAMESPACE_KEY_DEPTH is desighed with flexibility. Yes, by default it
is 2 which is cobmination of (ParentINodeId, INodeId). When you set it to 3,
then GrandParentId as well. We have tried upto level 3 with basic
functionality. But performance not measured. We continued to use with the
default value - 2. I am not sure of any use case to increase the values to
higher number (Atleast i havent done any testing on this part).
By default each partition capacity is 117965 (65536 * 1.8), we continue to use
the default values in our test. We also checked the scenarios when dynamic
partitions were added. No perf degrade on dynamic partitions, infact this is
expected to give higher throuput. We havent noticed very high CPU usage upto 1M
file write Ops (Resouce usage statistics we need to capture yet with base & FGL
Patch), so this shouldnt have any impact of the other operations (RPC or any
other server side processing tasks).
In case if you have missed the design please go through the latest desigh doc -
NameNode Fine-Grained Locking.pdf
[~shv] [~xinglin] Would you like to share your inputs?
> NameNode Fine-Grained Locking via Metadata Partitioning
> -------------------------------------------------------
>
> Key: HDFS-14703
> URL: https://issues.apache.org/jira/browse/HDFS-14703
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, namenode
> Reporter: Konstantin Shvachko
> Priority: Major
> Attachments: 001-partitioned-inodeMap-POC.tar.gz,
> 002-partitioned-inodeMap-POC.tar.gz, 003-partitioned-inodeMap-POC.tar.gz,
> NameNode Fine-Grained Locking.pdf, NameNode Fine-Grained Locking.pdf
>
>
> We target to enable fine-grained locking by splitting the in-memory namespace
> into multiple partitions each having a separate lock. Intended to improve
> performance of NameNode write operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
