[
https://issues.apache.org/jira/browse/HDFS-17366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17849694#comment-17849694
]
Chenyu Zheng commented on HDFS-17366:
-------------------------------------
[~xuzq_zander]
This is really great work, look forward to further progress!
And I have some confusion. After reading the documentation, I know that in the
final implementation, there is no dedicated bmLock, the bmlock is implemented
through dn lock and inode file lock.
(1) For block report, especially full block report. Do we need to acquire the
inode file lock for block every time we process a block? If so, will this make
the block report slower?
(2) For deleting the block lock, we first obtain the inode file lock, and then
obtain the dn lock. For the block report, we first go to the dn lock and then
get the inode file lock. Is this a possible deadlock? I see you proposed revert
HDFS-13671 in HDFS-17386. Is it to solve problem (2)?
> NameNode Fine-Grained Locking via Namespace Tree
> ------------------------------------------------
>
> Key: HDFS-17366
> URL: https://issues.apache.org/jira/browse/HDFS-17366
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, namenode
> Reporter: ZanderXu
> Assignee: ZanderXu
> Priority: Major
> Attachments: NameNode Fine-Grained Locking Based On Directory Tree.pdf
>
>
> As we all known, the write performance of NameNode is limited by the global
> lock. We target to enable fine-grained locking based on the Namespace tree to
> improve the performance of NameNode write operations.
> There are multiple motivations for creating this ticket:
> * We have implemented this fine-grained locking and gained nearly 7x
> performance improvements in our prod environment
> * Other companies made similar improvements based on their internal branch.
> Internal branches are quite different from the community, so few feedback and
> discussions in the community.
> * The topic of fine-grained locking has been discussed for a very long time,
> but still without any results.
>
> We implemented this fine-gained locking based on the namespace tree to
> maximize the number of concurrency for disjoint or independent operations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
