[
https://issues.apache.org/jira/browse/MAPREDUCE-5891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14106119#comment-14106119
]
Jason Lowe commented on MAPREDUCE-5891:
---------------------------------------
Thanks for posting a prototype, Junping! It's similar to what I was thinking.
Some initial comments from a brief look at the patch:
It could be very inefficient to re-shuffle a large map output that we already
have. If we get an error and need to reconnect then we should rebuild the URL
based on the remaining maps rather than keep using the original URL and
throwing away data for maps we already processed. In essence we should never
reconnect and ask for what we already have.
It seems like we should only need one-level of retry rather than two. I think
we only need to retry at the top-level where we try to copy map outputs. If
copying map outputs fails either because of a severed connection, a connection
reset/timeout, or whatever and we have not spent too much time trying to
recover from errors for the same map output then we retry. As soon as we
successfully shuffle a map then we reset the time (because things are working
and we're making progress) and move onto the next map output. I don't think we
need to retry at this level and also at the socket connection level. The upper
level retry will cause us to automatically retry the lower-level socket
connection, unless I'm missing something.
We probably should use Time.monotonicNow or something similar that won't break
the desired delay behavior if the clock changes.
"miliseconds" should be "milliseconds"
> Improved shuffle error handling across NM restarts
> --------------------------------------------------
>
> Key: MAPREDUCE-5891
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5891
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 2.5.0
> Reporter: Jason Lowe
> Assignee: Junping Du
> Attachments: MAPREDUCE-5891-demo.patch
>
>
> To minimize the number of map fetch failures reported by reducers across an
> NM restart it would be nice if reducers only reported a fetch failure after
> trying for at specified period of time to retrieve the data.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
