Re: how does hadoop handle the counter of the failed task and speculative task

[Arun C Murthy]

On Dec 25, 2010, at 10:08 PM, lei liu wrote:
I define the counter to count the bad records, there is below code  
in map task;
 reporter.incrCounter("bad',"
records', 1),

When the job is completed, the pritnt the result to use below code:
long total = counters.findCounter("bad","records").getCounter();


But I have two questions about the counter:
1. If the map task is retried 4 times, and the last map task is  
successful,  I think the counter of the other three map tasks should  
be not included in ultima result, is that right?
2. If there are speculative tasks,  I think the counter of  
speculative tasks should be not included in ultima result, is that  
right?


Yes to both. Counters are kept only from the 'successful' task- 
attempt, the others (failed, speculative etc.) are discarded.

Arun