[
https://issues.apache.org/jira/browse/HDFS-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16524147#comment-16524147
]
Todd Lipcon commented on HDFS-13702:
------------------------------------
Here are some perf results based on a 180GB teravalidate on a small cluster, as
well as a 1GB teravalidate on LocalJobRunner (against a remote HDFS)
{code}
3.0.x original
------------------------------------
Avg map: 18sec
CPU time spent (ms) 2,208,950
GC time elapsed (ms) 68,153
Performance counter stats for './run-validate.sh' (5 runs):
22357.081985 task-clock (msec) # 2.688 CPUs utilized
( +- 6.78% )
21,573 context-switches # 0.965 K/sec
( +- 2.58% )
1,300 cpu-migrations # 0.058 K/sec
( +- 4.82% )
425,146 page-faults # 0.019 M/sec
( +- 4.52% )
63,809,409,850 cycles # 2.854 GHz
( +- 6.56% )
66,580,182,677 instructions # 1.04 insn per cycle
( +- 2.28% )
13,489,574,848 branches # 603.369 M/sec
( +- 4.58% )
158,670,595 branch-misses # 1.18% of all branches
( +- 0.35% )
8.317048233 seconds time elapsed
( +- 0.10% )
3.0.x patched:
--------------------------------------------------------------------------------
Avg map time: 14sec
CPU time spent (ms) 1,750,180
GC time elapsed (ms) 42,468
Performance counter stats for './run-validate.sh' (5 runs):
14466.559412 task-clock (msec) # 2.006 CPUs utilized
( +- 3.18% )
21,666 context-switches # 0.001 M/sec
( +- 0.55% )
1,180 cpu-migrations # 0.082 K/sec
( +- 1.91% )
234,159 page-faults # 0.016 M/sec
( +- 0.60% )
41,793,452,250 cycles # 2.889 GHz
( +- 2.77% )
55,219,815,925 instructions # 1.32 insn per cycle
( +- 1.67% )
9,837,238,534 branches # 679.998 M/sec
( +- 2.57% )
161,071,903 branch-misses # 1.64% of all branches
( +- 0.62% )
7.210730451 seconds time elapsed
( +- 0.25% )
{code}
> HTrace hooks taking 10-15% CPU in DFS client when disabled
> ----------------------------------------------------------
>
> Key: HDFS-13702
> URL: https://issues.apache.org/jira/browse/HDFS-13702
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: performance
> Affects Versions: 3.0.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Major
> Attachments: hdfs-13702.patch
>
>
> I am seeing DFSClient.newReaderTraceScope take ~15% CPU in a teravalidate
> workload even when HTrace is disabled. This is because it stringifies several
> integers. We should avoid all allocation and stringification when htrace is
> disabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
