[
https://issues.apache.org/jira/browse/HADOOP-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649167#action_12649167
]
Chris Douglas commented on HADOOP-2774:
---------------------------------------
Sorry, I was unclear. The issue is not the volatility of the aggregator, but
that it is static. There are a number of contexts- JVM reuse, for one- where it
simply will not work. Aggregating disparate counters in a shared variable only
to push its _deltas_ to another aggregator is not an expected approach. A
static counter 1) is already implemented in the Counter interface; duplicating
its functionality to feed it is suspect and 2) is almost certainly more
difficult to get right for all our use cases than some of the alternatives.
Sharad's proposal seems very reasonable. I'd suggest one variant: adding a
Counter formal to the IFile.Reader and IFile.Writer constructors. In the map,
creating each Writer with a counter to track each record hitting disk should be
accurate. In the reduce, instead of incrementing the counter as the segment is
written to disk from the fetch and intermediate merges, updating it as it is
*read* from disk will yield the correct value at the end of the job. So for the
final merge into the reduce and the intermediate, on-disk merges, a counter
will be provided. This makes it unnecessary to transfer the record count to the
reduce, lets the IFile format remain exactly as it is, and should be fairly
easy to implement.
Thoughts?
> Add counters to show number of key/values that have been sorted and merged in
> the maps and reduces
> --------------------------------------------------------------------------------------------------
>
> Key: HADOOP-2774
> URL: https://issues.apache.org/jira/browse/HADOOP-2774
> Project: Hadoop Core
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Ravi Gummadi
> Fix For: 0.20.0
>
> Attachments: HADOOP-2774.patch, HADOOP-2774.patch
>
>
> For each *pass* of the sort and merge, I would like a count of the number of
> records. So for example, if the map output 100 records and they were sorted
> once, the counter would be 100. If it spilled twice and was merged together,
> it would be 200. Clearly in a multi-level merge, it may not be a multiple of
> the number of map output records. This would let the users easily see if they
> have values like io.sort.mb or io.sort.factor set too low.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
