[
https://issues.apache.org/jira/browse/HDFS-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17344970#comment-17344970
]
Xing Lin commented on HDFS-14703:
---------------------------------
[~prasad-acit] how many CPU cores does your server have? The OPS per sec seems
rather low, than I got from my Mac laptop (with 8 cores). fgl gives us 10%
improvement running on my Mac. We will find some proper hardware to do more
serial performance benchmarks.
*Trunk*
021-05-11 09:52:35,666 INFO namenode.NNThroughputBenchmark:
2021-05-11 09:52:35,667 INFO namenode.NNThroughputBenchmark: --- mkdirs inputs
---
2021-05-11 09:52:35,667 INFO namenode.NNThroughputBenchmark: nrDirs = 10000000
2021-05-11 09:52:35,667 INFO namenode.NNThroughputBenchmark: nrThreads = 200
2021-05-11 09:52:35,667 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 512
2021-05-11 09:52:35,667 INFO namenode.NNThroughputBenchmark: --- mkdirs stats
---
2021-05-11 09:52:35,667 INFO namenode.NNThroughputBenchmark: # operations:
10000000
2021-05-11 09:52:35,667 INFO namenode.NNThroughputBenchmark: Elapsed Time:
542905
2021-05-11 09:52:35,667 INFO namenode.NNThroughputBenchmark: Ops per sec:
18419.42881351249
2021-05-11 09:52:35,667 INFO namenode.NNThroughputBenchmark: Average Time: 10
2021-05-11 09:52:35,667 INFO namenode.FSEditLog: Ending log segment 5488830,
10019538
2021-05-11 09:52:35,670 INFO namenode.FSEditLog: Number of transactions:
4530710 Total time for transactions(ms): 14288 Number of transactions batched
in Syncs: 4452444 Number of syncs: 78267 SyncTimes(ms): 200575
*fgl*
021-05-11 10:58:40,142 INFO namenode.NNThroughputBenchmark:
2021-05-11 10:58:40,142 INFO namenode.NNThroughputBenchmark: --- mkdirs inputs
---
2021-05-11 10:58:40,143 INFO namenode.NNThroughputBenchmark: nrDirs = 10000000
2021-05-11 10:58:40,143 INFO namenode.NNThroughputBenchmark: nrThreads = 200
2021-05-11 10:58:40,143 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 512
2021-05-11 10:58:40,143 INFO namenode.NNThroughputBenchmark: --- mkdirs stats
---
2021-05-11 10:58:40,143 INFO namenode.NNThroughputBenchmark: # operations:
10000000
2021-05-11 10:58:40,143 INFO namenode.NNThroughputBenchmark: Elapsed Time:
505892
2021-05-11 10:58:40,143 INFO namenode.NNThroughputBenchmark: Ops per sec:
19767.06490713433
2021-05-11 10:58:40,143 INFO namenode.NNThroughputBenchmark: Average Time: 10
2021-05-11 10:58:40,143 INFO namenode.FSEditLog: Ending log segment 5826307,
10019538
2021-05-11 10:58:40,146 INFO namenode.FSEditLog: Number of transactions:
4193233 Total time f
or transactions(ms): 13990 Number of transactions batched in Syncs: 4130972
Number of syncs:
62262 SyncTimes(ms): 168203
> NameNode Fine-Grained Locking via Metadata Partitioning
> -------------------------------------------------------
>
> Key: HDFS-14703
> URL: https://issues.apache.org/jira/browse/HDFS-14703
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, namenode
> Reporter: Konstantin Shvachko
> Priority: Major
> Attachments: 001-partitioned-inodeMap-POC.tar.gz,
> 002-partitioned-inodeMap-POC.tar.gz, 003-partitioned-inodeMap-POC.tar.gz,
> NameNode Fine-Grained Locking.pdf, NameNode Fine-Grained Locking.pdf
>
>
> We target to enable fine-grained locking by splitting the in-memory namespace
> into multiple partitions each having a separate lock. Intended to improve
> performance of NameNode write operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
