[
https://issues.apache.org/jira/browse/MAPREDUCE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15342184#comment-15342184
]
Naganarasimha G R edited comment on MAPREDUCE-6720 at 6/21/16 5:03 PM:
-----------------------------------------------------------------------
bq. Please note that these metrics will be reported as part of task counters as
well.
well what we are doing at the end is publishing the aggregated map and/or
Reduce counters as application metrics so task counters are of no significance
and retrospectively if suppose we go ahead with map and reduce perspective app
metrics would it be of any use for aggregation across flows ? but definitely
useful for analyzing for a given app. Either we keep this information as part
of event or we need to append with Map/Reduce and keep it as app metric.
bq. Thinking more over it, we probably have no way of filtering out Map and
reduce tasks right now. Maybe we can specify in info that a particular task is
MAP or REDUCE.
Not clear what you meant by this.
was (Author: naganarasimha):
bq. Please note that these metrics will be reported as part of task counters as
well.
well what we are doing at the end is publishing the aggregated map and/or
Reduce counters as application metrics so task counters are of no significance
and retrospectively if suppose we go ahead with map and reduce perspective app
metrics would it be of any use for aggregation across flows ? but definitely
useful for analyzing for a given app. Either we keep this information as part
of event or we need to append with Map/Reduce and keep it as app metric.
> Inconsistent values of counters across tasks and job reported to timeline
> service.
> ----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-6720
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6720
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster
> Affects Versions: YARN-2928
> Reporter: Varun Saxena
> Assignee: Varun Saxena
> Labels: yarn-2928-1st-milestone
> Attachments: Counters For Job.png, Counters for Tasks.png,
> MAPREDUCE-6720-YARN-2928.01.patch
>
>
> While testing found below issue. For some of the task counters, we do not
> have consistent values. This is not the case with every counter though.
> Consider the case of counter
> "org.apache.hadoop.mapreduce.FileSystemCounter:FILE_BYTES_WRITTEN".
> I found that its value for a flow I ran, was 936018 bytes. For the 3 apps
> associated with this flow run, the values were 312006 bytes each (which
> equals to value for a flow run i.e. 3 * 312006 = 936018). Drilling further
> down I found though that for one of the apps, the 4 tasks(2 mappers and 2
> reducers) had values as 155918 bytes each for the 2 reducers and 156003 bytes
> each for the 2 mappers.
> This means the value reported for the app should be (2 * 156003 + 2* 155918)
> or 623842 bytes but it is only 312006 bytes which indicates that only counter
> value of mappers is being picked up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
