Wei-Chiu Chuang created HDFS-14277:
--------------------------------------
Summary: [SBN read] Observer benchmark results
Key: HDFS-14277
URL: https://issues.apache.org/jira/browse/HDFS-14277
Project: Hadoop HDFS
Issue Type: Task
Components: ha, namenode
Affects Versions: 3.3.0
Environment: Hardware: 4-node cluster, each node has 4 core, Xeon
2.5Ghz, 25GB memory.
Software: CentOS 7.4, CDH 6.0 + Consistent Reads from Standby, Kerberos, SSL,
RPC encryption + Data Transfer Encryption, Cloudera Navigator.
Reporter: Wei-Chiu Chuang
Assignee: Wei-Chiu Chuang
Attachments: observer RPC queue processing time.png
Ran a few benchmarks and profiler (VisualVM) today on an Observer-enabled
cluster. Would like to share the results with the community. The cluster has 1
Observer node.
h2. NNThroughputBenchmark
Generate 1 million files and send fileStatus RPCs.
{code:java}
hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -fs
<namenode> -op fileStatus -threads 100 -files 1000000 -useExisting -keepResults
{code}
h3. Kerberos, SSL, RPC encryption, Data Transfer Encryption enabled:
||Node||fileStatus (Ops per sec)||
|Active NameNode|4865|
|Observer|3996|
h3. Kerberos, SSL:
||Node||fileStatus (Ops per sec)||
|Active NameNode|7078|
|Observer|6459|
Observation:
* due to the edit tailing overhead, Observer node consume 30% CPU utilization
even if the cluster is idle.
* While Active NN has less than 1ms RPC processing time, Observer node has >
5ms RPC processing time. I am still looking for the source of the longer
processing time. The longer RPC processing time may be the cause for the
performance degradation compared to that of Active NN. Note the cluster has
Cloudera Navigator installed which adds additional overhead to RPC processing
time.
* {{GlobalStateIdContext#isCoordinatedCall()}} pops up as one of the top
hotspots in the profiler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
