[
http://issues.apache.org/jira/browse/HADOOP-492?page=comments#action_12431742 ]
Owen O'Malley commented on HADOOP-492:
--------------------------------------
In a lot of ways, this is a weaker form of HADOOP-48. The advantage of counters
is that it is clear how to aggregate them across tasks to form a count for the
entire job. We could do:
public class JobCounters implements Writable {
<methods to get/set generic counters for records/bytes, whatever>
public void add(JobCounters other);
}
and in JobConf:
public set/getJobCounterClass(...);
and in Reporter add:
public JobCounters getJobCounters();
and the JobCounter is sent up as part of the heartbeat.
> Global counters
> ---------------
>
> Key: HADOOP-492
> URL: http://issues.apache.org/jira/browse/HADOOP-492
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: arkady borkovsky
>
> It would be nice to have map / reduce job keep aggregated counts for
> arbitrary events occuring in its tasks -- the numer of records processed, the
> numer of exceptions of a specific type, the number of sentences in passive
> voice, whatever the jobs finds useful.
> This can be implemented by tasks periodically sending <name, value> pairs to
> the jobtracker (in some implementations such messages are piggy-backed on the
> heartbeats), so that the job tracker stores all the latests values from each
> task and aggregates them on a request. It should also make the aggregated
> values available at the job end. The value for a task would be flushed when
> the task fails.
> #491 and #490 may be related to this one.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
