Mikayla Konst created MAPREDUCE-7168:
----------------------------------------
Summary: Add option to not kill already-done map tasks when node
becomes unusable
Key: MAPREDUCE-7168
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7168
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: mrv2
Affects Versions: 2.9.2
Environment: Google Compute Engine (Dataproc), Java 8
Reporter: Mikayla Konst
When a node becomes unusable, if there are still reduce tasks running, all
completed map tasks that were run on that node are killed so that they can be
re-run on a different node. This is because the node can no longer serve
shuffle data, so the map task output cannot be fetched by the reducers.
If map tasks do not write their shuffle data locally, killing already-done map
tasks will make the job lose map progress unnecessarily. This change prevents
map progress from being lost when shuffle data is not written locally by
providing a property mapreduce.map.rerun-if-node-unusable that can be set to
false to prevent killing already-done map tasks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
