[
https://issues.apache.org/jira/browse/MAPREDUCE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15342237#comment-15342237
]
Sangjin Lee commented on MAPREDUCE-6720:
----------------------------------------
If we are going to store the map counters and reduce counters, we should make
the metric names such that one can easily query for them only. So in that
sense, making it a total prefix or suffix to the metric name might be a better
idea? In other words, how about something like
{noformat}
org.apache.hadoop.mapreduce.FileSystemCounter:FILE_READ_OPS for total.
MAP:org.apache.hadoop.mapreduce.FileSystemCounter:FILE_READ_OPS for MAP.
REDUCE:org.apache.hadoop.mapreduce.FileSystemCounter:FILE_READ_OPS for REDUCE.
{noformat}
or
{noformat}
org.apache.hadoop.mapreduce.FileSystemCounter:FILE_READ_OPS for total.
org.apache.hadoop.mapreduce.FileSystemCounter:FILE_READ_OPS:MAP for MAP.
org.apache.hadoop.mapreduce.FileSystemCounter:FILE_READ_OPS:REDUCE for REDUCE.
{noformat}
?
> Inconsistent values of counters across tasks and job reported to timeline
> service.
> ----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-6720
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6720
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster
> Affects Versions: YARN-2928
> Reporter: Varun Saxena
> Assignee: Varun Saxena
> Labels: yarn-2928-1st-milestone
> Attachments: Counters For Job.png, Counters for Tasks.png,
> MAPREDUCE-6720-YARN-2928.01.patch, MAPREDUCE-6720-YARN-2928.02.patch
>
>
> While testing found below issue. For some of the task counters, we do not
> have consistent values. This is not the case with every counter though.
> Consider the case of counter
> "org.apache.hadoop.mapreduce.FileSystemCounter:FILE_BYTES_WRITTEN".
> I found that its value for a flow I ran, was 936018 bytes. For the 3 apps
> associated with this flow run, the values were 312006 bytes each (which
> equals to value for a flow run i.e. 3 * 312006 = 936018). Drilling further
> down I found though that for one of the apps, the 4 tasks(2 mappers and 2
> reducers) had values as 155918 bytes each for the 2 reducers and 156003 bytes
> each for the 2 mappers.
> This means the value reported for the app should be (2 * 156003 + 2* 155918)
> or 623842 bytes but it is only 312006 bytes which indicates that only counter
> value of mappers is being picked up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
