[
https://issues.apache.org/jira/browse/HDFS-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122049#comment-16122049
]
Konstantin Shvachko edited comment on HDFS-9153 at 8/10/17 6:35 PM:
--------------------------------------------------------------------
Hey guys I think the metrics you introduced is absolutely deceiving, and has
nothing to do with the throughput the benchmark is intended to measure.
"Test exec time" is the running time of the job, which includes the compute
overhead: scheduling, cleanup, and retries if there were failed maps.
While we want to benchmark the average throughput of the actual data transfers
on HDFS. You should see the implementation measures time of transfers only.
The formatting changes are fine. But I think "Total Throughput" should be
removed.
The bug reported in MAPREDUCE-6931 makes it invalid, but even if fixed it is
still deceiving.
-Also, DFSIO issues should be filed on HDFS jira. Then you should expect more
prompt response.-
_Sorry last part was for the other jira. Please ignore._
was (Author: shv):
Hey guys I think the metrics you introduced is absolutely deceiving, and has
nothing to do with the throughput the benchmark is intended to measure.
"Test exec time" is the running time of the job, which includes the compute
overhead: scheduling, cleanup, and retries if there were failed maps.
While we want to benchmark the average throughput of the actual data transfers
on HDFS. You should see the implementation measures time of transfers only.
The formatting changes are fine. But I think "Total Throughput" should be
removed.
The bug reported in MAPREDUCE-6931 makes it invalid, but even if fixed it is
still deceiving.
Also, DFSIO issues should be filed on HDFS jira. Then you should expect more
prompt response.
> Pretty-format the output for DFSIO
> ----------------------------------
>
> Key: HDFS-9153
> URL: https://issues.apache.org/jira/browse/HDFS-9153
> Project: Hadoop HDFS
> Issue Type: Test
> Reporter: Kai Zheng
> Assignee: Kai Zheng
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: HDFS-9153-v1.patch
>
>
> Ref. the following DFSIO output, I was surprised the test throughput was only
> {{17}} MB/s, which doesn't make sense for a real cluster. Maybe it's used for
> other purpose? For users, it may make more sense to give the throughput 1610
> MB/s (1228800/763), calculated by *Total MBytes processed / Test exec time*.
> {noformat}
> 15/09/28 11:42:23 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
> 15/09/28 11:42:23 INFO fs.TestDFSIO: Date & time: Mon Sep 28
> 11:42:23 CST 2015
> 15/09/28 11:42:23 INFO fs.TestDFSIO: Number of files: 100
> 15/09/28 11:42:23 INFO fs.TestDFSIO: Total MBytes processed: 1228800.0
> 15/09/28 11:42:23 INFO fs.TestDFSIO: Throughput mb/sec:
> 17.457387239456878
> 15/09/28 11:42:23 INFO fs.TestDFSIO: Average IO rate mb/sec: 17.57563018798828
> 15/09/28 11:42:23 INFO fs.TestDFSIO: IO rate std deviation:
> 1.7076328985378455
> 15/09/28 11:42:23 INFO fs.TestDFSIO: Test exec time sec: 762.697
> 15/09/28 11:42:23 INFO fs.TestDFSIO:
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
