[ https://issues.apache.org/jira/browse/HDFS-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17346379#comment-17346379 ]
Konstantin Shvachko commented on HDFS-14703: -------------------------------------------- Thanks [~prasad-acit] and [~xinglin] for benchmarking. Very glad you guys could independently confirm 30-45% improvement. I think the PartitionedGSet implementation should benefit from both *_more cores_* and *_faster storage device_* for edits. For storage device NVME SSDs perform the best for journaling type workloads in our experience. Also please take into account this is only a POC patch. Theoretically, we should be able to scale performance proportionally to the number of cores and partitions in the GSet given we are not IO bound. > NameNode Fine-Grained Locking via Metadata Partitioning > ------------------------------------------------------- > > Key: HDFS-14703 > URL: https://issues.apache.org/jira/browse/HDFS-14703 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, namenode > Reporter: Konstantin Shvachko > Priority: Major > Attachments: 001-partitioned-inodeMap-POC.tar.gz, > 002-partitioned-inodeMap-POC.tar.gz, 003-partitioned-inodeMap-POC.tar.gz, > NameNode Fine-Grained Locking.pdf, NameNode Fine-Grained Locking.pdf > > > We target to enable fine-grained locking by splitting the in-memory namespace > into multiple partitions each having a separate lock. Intended to improve > performance of NameNode write operations. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org