[
https://issues.apache.org/jira/browse/HADOOP-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated HADOOP-1042:
--------------------------------
Attachment: 1042.patch
This patch does the following (everything in the file ReduceTaskRunner.java):
1) Changes the datastructure of knownOutputs from List to Map. This eases
replacing MapOutputLocation objects for the failed fetches (if the JobTracker
later on gives us new locations for those mapIds)
2) Changes ListIterator to Iterator (since it is not straightforward to get a
ListIterator out of a Map and we anyway don't use the features of a
ListIterator)
3) Changes the order in which entries (mapId/MapOutputLocation objects) are
added in the knownOutputs Map - first entries corresponding to failed fetches
are added and then the new entries (got from JobTracker) are added. This will
ensure that the new entries overwrite the old (failed) entries (for the same
mapId hashkeys).
4) Removes the call to Collections.shuffle( ) and the associated Random object.
Since the randomness for fetching map outputs is not there anymore, we don't
need this.
5) queryJobTracker now returns a List <MapOutputLocation> instead of an array
of MapOutputLocation.
> Improve the handling of failed map output fetches
> -------------------------------------------------
>
> Key: HADOOP-1042
> URL: https://issues.apache.org/jira/browse/HADOOP-1042
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.11.2
> Reporter: Devaraj Das
> Assigned To: Devaraj Das
> Attachments: 1042.patch
>
>
> Currently, whenever fetch of a map output fails the corresponding
> MapOutputLocation is added to a List datastructure for later retrial. But, if
> the failure was due to a lost task, the entry that was added is not deleted.
> For such cases, unnecessary retrials will happen. This situation should be
> prevented.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
