[
https://issues.apache.org/jira/browse/MAPREDUCE-5891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127212#comment-14127212
]
Ming Ma commented on MAPREDUCE-5891:
------------------------------------
The patch looks good. I like Jason's idea to have
mapreduce.reduce.shuffle.fetch.retry.enabled use
${yarn.nodemanager.recovery.enabled} as default value. As for the other
approaches,
a) dynamic MR to YARN query, given NM recovery flag is a global cluster level
setting ( although it is possible to config it on per NM basis ), can we derive
the value of mapreduce.reduce.shuffle.fetch.retry.enabled at job submission
time from some YARN API call to RM?
b) shuffle protocol change. It seems Fetcher and ShuffleHandler check http
header via property key names. So if we add a new property to indicate if
recovery is supported and continue to keep the same http "version" property,
new version of fetcher might be able to work with old version of
shufflehandler, and vise versa.
> Improved shuffle error handling across NM restarts
> --------------------------------------------------
>
> Key: MAPREDUCE-5891
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5891
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 2.5.0
> Reporter: Jason Lowe
> Assignee: Junping Du
> Attachments: MAPREDUCE-5891-demo.patch, MAPREDUCE-5891-v2.patch,
> MAPREDUCE-5891-v3.patch, MAPREDUCE-5891-v4.patch, MAPREDUCE-5891.patch
>
>
> To minimize the number of map fetch failures reported by reducers across an
> NM restart it would be nice if reducers only reported a fetch failure after
> trying for at specified period of time to retrieve the data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
