[ https://issues.apache.org/jira/browse/HDFS-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907662#comment-16907662 ]
Konstantin Shvachko commented on HDFS-14703: -------------------------------------------- Attaching the POC patch. It consists of 4 commits. Apply using {{git am 001-partitioned-inodeMap-POC/*}} command. # 0001 patch is an investigation to verify that FSN lock is used together with dirLock. I just ran unit tests with this patch. Most of them pass the verification, but some don't. # 0002 patch disables dirLock. # 0003 introduces PartitionedGSet, LatchLock. It implements dynamic partitioning based on inodeId key (see INodeIdComparator) # 0004 introduces two-level key and implements static partitioning based on that key. With 0003 and 0004 patches I ran NNThroughputBenchmark creating 2 million directories with 200 concurrent threads and 128 subdirectories. So the POC implements new locking for one operation only - mkdir. The benchmark command: {{NNThroughputBenchmark -fs file:/// -op mkdirs -threads 200 -dirs 2000000 -dirsPerDir 128}} > NameNode Fine-Grained Locking via Metadata Partitioning > ------------------------------------------------------- > > Key: HDFS-14703 > URL: https://issues.apache.org/jira/browse/HDFS-14703 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, namenode > Reporter: Konstantin Shvachko > Priority: Major > Attachments: 001-partitioned-inodeMap-POC.tar.gz, NameNode > Fine-Grained Locking.pdf > > > We target to enable fine-grained locking by splitting the in-memory namespace > into multiple partitions each having a separate lock. Intended to improve > performance of NameNode write operations. -- This message was sent by Atlassian JIRA (v7.6.14#76016) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org