I just submitted a Jira for this, with a patch against trunk.
Hadoop-3422 <https://issues.apache.org/jira/browse/HADOOP-3422>

If you pick up the patch, all of the metrics now have a leader in the name indicating which context/job is doing the reporting, and all of the job counters have their own rrd file and graph.


Jason Venner wrote:
I first verified that when I was using the file context, I saw the counters in the report.
Then I switched context's to ganglia.

I also instrumented the low level code. I am hoping someone understands this off the top of their head as I don't want to dig into the metrics code to understand the difference ;)

Here is some syslog data. Note the missing name for the metric in the JobInProgress case.

The JobTracker uses:
       metricsRecord.incrMetric("maps_launched", numMapTasksLaunched);
       metricsRecord.incrMetric("maps_completed", numMapTasksCompleted);
metricsRecord.incrMetric("reduces_launched", numReduceTasksLaunched); metricsRecord.incrMetric("reduces_completed", numReduceTasksCompleted);
       metricsRecord.incrMetric("jobs_submitted", numJobsSubmitted);
       metricsRecord.incrMetric("jobs_completed", numJobsCompleted);

While JobInProgress uses:
     for (String counter : group.getCounterNames()) {
       long value = group.getCounter(counter);
       jobMetrics.setTag("counter", group.getDisplayName(counter));
       jobMetrics.setMetric("value", (float) value);
       jobMetrics.update();


These are the job counters (JobInProgress)

2008-05-19 17:49:39,439 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: value, Type: float, Value: 36.0 to localhost/127.0.0.1:8649

2008-05-19 17:49:39,439 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: value, Type: float, Value: 20.0 to localhost/127.0.0.1:8649

2008-05-19 17:49:39,440 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: value, Type: float, Value: 0.0 to localhost/127.0.0.1:8649

2008-05-19 17:49:39,447 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: value, Type: float, Value: 432.0 to localhost/127.0.0.1:8649

2008-05-19 17:49:39,448 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: value, Type: float, Value: 0.0 to localhost/127.0.0.1:8649

2008-05-19 17:49:39,448 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: value, Type: float, Value: 18.0 to localhost/127.0.0.1:8649

2008-05-19 17:49:39,448 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: value, Type: float, Value: 576.0 to localhost/127.0.0.1:8649

These are the JobTracker counters
2008-05-19 17:49:39,449 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: jobs_completed, Type: int32, Value: 0 to localhost/127.0.0.1:8649

2008-05-19 17:49:39,449 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: jobs_submitted, Type: int32, Value: 1 to localhost/127.0.0.1:8649

2008-05-19 17:49:39,449 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: maps_completed, Type: int32, Value: 18 to localhost/127.0.0.1:8649

2008-05-19 17:49:39,449 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: maps_launched, Type: int32, Value: 20 to localhost/127.0.0.1:8649

2008-05-19 17:49:39,450 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: reduces_completed, Type: int32, Value: 0 to localhost/127.0.0.1:8649

2008-05-19 17:49:39,450 INFO org.apache.hadoop.metrics.ganglia.GangliaContext: Sending metric Name: reduces_launched, Type: int32, Value: 0 to localhost/127.0.0.1:8649

--
Jason Venner
Attributor - Program the Web <http://www.attributor.com/>
Attributor is hiring Hadoop Wranglers and coding wizards, contact if interested

Reply via email to