[ https://issues.apache.org/jira/browse/HDFS-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17387797#comment-17387797 ]
Xing Lin edited comment on HDFS-14703 at 7/27/21, 6:07 AM: ----------------------------------------------------------- [~daryn] Thanks for your comments. I will address your last question and leave other questions to [~shv]. :) Regarding the results, we used the standard NNThroughputBenchmark, with commands like the following. {code:java} ./bin/hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -fs file:/// -op mkdirs -threads 200 -dirs 10000000 -dirsPerDir 512{code} Here are a result from [~prasad-acit], since his QPS numbers are higher than what I got. {code:java} BASE: common/hadoop-hdfs-32021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: — mkdirs inputs — 2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: nrDirs = 1000000 2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: nrThreads = 200 2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 32 2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: — mkdirs stats — 2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: # operations: 1000000 2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: Elapsed Time: 17718 2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: Ops per sec: 56439.77875606727 2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: Average Time: 3 2021-05-17 11:17:36,973 INFO namenode.FSEditLog: Ending log segment 1, 1031254 PATCH: 2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: — mkdirs inputs — 2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: nrDirs = 1000000 2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: nrThreads = 200 2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 32 2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: — mkdirs stats — 2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: # operations: 1000000 2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: Elapsed Time: 15010 2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: Ops per sec: 66622.25183211193 2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: Average Time: 2 2021-05-17 11:11:09,331 INFO namenode.FSEditLog: Ending log segment 1, 1031254 {code} was (Author: xinglin): [~daryn] Thanks for your comments. I will address your last question and leave other questions to [~shv]. :) Regarding the results, we used the standard NNThroughputBenchmark, with commands like the following. ./bin/hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark *-fs* [*file:///*|file:///*] -op mkdirs -threads 200 -dirs 10000000 -dirsPerDir 512 Here are a result from [~prasad-acit], since his QPS numbers are higher than what I got. BASE: common/hadoop-hdfs-32021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: --- mkdirs inputs --- 2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: nrDirs = 1000000 2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: nrThreads = 200 2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 32 2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: --- mkdirs stats --- 2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: # operations: 1000000 2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: Elapsed Time: 17718 2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: Ops per sec: 56439.77875606727 2021-05-17 11:17:36,973 INFO namenode.NNThroughputBenchmark: Average Time: 3 2021-05-17 11:17:36,973 INFO namenode.FSEditLog: Ending log segment 1, 1031254 PATCH: 2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: --- mkdirs inputs --- 2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: nrDirs = 1000000 2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: nrThreads = 200 2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 32 2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: --- mkdirs stats --- 2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: # operations: 1000000 2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: Elapsed Time: 15010 2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: Ops per sec: 66622.25183211193 2021-05-17 11:11:09,321 INFO namenode.NNThroughputBenchmark: Average Time: 2 2021-05-17 11:11:09,331 INFO namenode.FSEditLog: Ending log segment 1, 1031254 > NameNode Fine-Grained Locking via Metadata Partitioning > ------------------------------------------------------- > > Key: HDFS-14703 > URL: https://issues.apache.org/jira/browse/HDFS-14703 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, namenode > Reporter: Konstantin Shvachko > Priority: Major > Attachments: 001-partitioned-inodeMap-POC.tar.gz, > 002-partitioned-inodeMap-POC.tar.gz, 003-partitioned-inodeMap-POC.tar.gz, > NameNode Fine-Grained Locking.pdf, NameNode Fine-Grained Locking.pdf > > > We target to enable fine-grained locking by splitting the in-memory namespace > into multiple partitions each having a separate lock. Intended to improve > performance of NameNode write operations. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org