[
https://issues.apache.org/jira/browse/MAPREDUCE-5652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985932#comment-13985932
]
Ming Ma commented on MAPREDUCE-5652:
------------------------------------
1. Regarding generic interface for restore/recover, I agree there is no much
benefit to generalize things for the sake of it. One scenario could be
something like ShuffleHandler, some ShuffleHandlers support recovery, some
don't. NM can ask if a specific ShuffleHandler if it supports recovery, NM will
manage the underlying store and pass the store object to ShuffleHandler and
ShuffleHandler manages the serialization and deserialization, etc. If NM
decides to change the underlying store and ShuffleHandler doesn't need to
change. But at this point, it seems unnecessary.
2. If ShuffleHandler gets DBException during recoverState as part of
serviceStart, should ShuffleHandler ignore the exception and continue like the
store doesn't exist? The argument for ignoring it is it is soft state and
ShuffleHandler can still run without it. Or maybe this can be configurable.
> NM Recovery. ShuffleHandler should handle NM restarts
> -----------------------------------------------------
>
> Key: MAPREDUCE-5652
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5652
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.2.0
> Reporter: Karthik Kambatla
> Assignee: Jason Lowe
> Labels: shuffle
> Attachments: MAPREDUCE-5652-v2.patch, MAPREDUCE-5652-v3.patch,
> MAPREDUCE-5652-v4.patch, MAPREDUCE-5652-v5.patch, MAPREDUCE-5652-v6.patch,
> MAPREDUCE-5652-v7.patch, MAPREDUCE-5652-v8.patch, MAPREDUCE-5652.patch
>
>
> ShuffleHandler should work across NM restarts and not require re-running
> map-tasks. On NM restart, the map outputs are cleaned up requiring
> re-execution of map tasks and should be avoided.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
