incrementing counters should not be used for triggering record skipping
-----------------------------------------------------------------------
Key: HADOOP-4718
URL: https://issues.apache.org/jira/browse/HADOOP-4718
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Reporter: Owen O'Malley
The following code is really problematic:
{code}
public void incrCounter(String group, String counter, long amount) {
if (counters != null) {
counters.incrCounter(group, counter, amount);
}
if(skipping && SkipBadRecords.COUNTER_GROUP.equals(group) && (
SkipBadRecords.COUNTER_MAP_PROCESSED_RECORDS.equals(counter) ||
SkipBadRecords.COUNTER_REDUCE_PROCESSED_GROUPS.equals(counter))) {
//if application reports the processed records, move the
//currentRecStartIndex to the next.
//currentRecStartIndex is the start index which has not yet been
//finished and is still in task's stomach.
for(int i=0;i<amount;i++) {
currentRecStartIndex = currentRecIndexIterator.next();
}
...
}
{code}
In particular, if the user updates a counter with the wrong name, bad things
will presumably happen...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
