[
https://issues.apache.org/jira/browse/MAPREDUCE-5652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987668#comment-13987668
]
Jason Lowe commented on MAPREDUCE-5652:
---------------------------------------
bq. Just to confirm, protobuf should be backward compatible, e.g., the store
state serialized with version 2.4 should be readable by NM/MR compiled with
version 2.5.
Yes, the protobuf incompatibility between 2.4 and 2.5 is an issue with the
interfaces to the protobuf code and not an incompatibility with the data layout
of protobuf messages.
bq. On an unrelated note, based on how NM's AuxServices' serviceStart handles
error for each AuxService' serviceStart, if one AuxService throws some
exception, the rest of AuxServices' serviceStart will be skipped.
I might be reading the code incorrectly, but it looks like
AuxServices#serviceStart doesn't try to handle exceptions coming from
individual aux services at all. If one of their startups throws then it will
be converted into a RuntimeException (by AbstractService#start) which will
bubble up out of AuxServices and likely all the way up such that the NM startup
will fail.
As you pointed out before, a better way to handle aux services would be to run
them outside of the NM (maybe even within containers). It'd also be nice to
make them more dynamic, such that application submissions can provide an aux
service they require. They could be started on demand, ref-counted, and
stopped accordingly based on usage, which would be a smoother answer to rolling
upgrades and sharing multiple versions of the aux services within the cluster.
This is of course a non-trivial amount of work for another JIRA. ;-)
> NM Recovery. ShuffleHandler should handle NM restarts
> -----------------------------------------------------
>
> Key: MAPREDUCE-5652
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5652
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.2.0
> Reporter: Karthik Kambatla
> Assignee: Jason Lowe
> Labels: shuffle
> Attachments: MAPREDUCE-5652-v2.patch, MAPREDUCE-5652-v3.patch,
> MAPREDUCE-5652-v4.patch, MAPREDUCE-5652-v5.patch, MAPREDUCE-5652-v6.patch,
> MAPREDUCE-5652-v7.patch, MAPREDUCE-5652-v8.patch,
> MAPREDUCE-5652-v9-and-YARN-1987.patch, MAPREDUCE-5652.patch
>
>
> ShuffleHandler should work across NM restarts and not require re-running
> map-tasks. On NM restart, the map outputs are cleaned up requiring
> re-execution of map tasks and should be avoided.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
