[
https://issues.apache.org/jira/browse/MAPREDUCE-4584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652832#comment-13652832
]
Chris Douglas commented on MAPREDUCE-4584:
------------------------------------------
[~ozawa]: I've been reading some of the iterations of your patch(es) as you've
updated them over the last few months. Our proposals are absolutely
complementary. Your approach (IIRC) involved reusing map tasks to aggregate map
output on the same host, right? MAPREDUCE-4502 can accomplish more than
checkpointing by aggregating across partitions.
We added some metadata to {{IFile}} to track which task attempts a segment
contains. I haven't looked at a recent version of your patch, but that's
certainly shared functionality.
> Umbrella: Preemption and restart of MapReduce tasks
> ---------------------------------------------------
>
> Key: MAPREDUCE-4584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4584
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: applicationmaster, mrv2, performance, resourcemanager,
> task
> Reporter: Sriram Rao
> Assignee: Chris Douglas
>
> This JIRA will track the implementation of improvements to the handling of
> intermediate data (e.g., map output). Specifically, it tracks changes in
> support of preempting running tasks, checkpointing completed work, and
> spawning one or more tasks to complete the original split/partition. These
> mechanisms allow one to manage skew in intermediate data, respond to resource
> abundance or scarcity (particularly with preemption), speculatively execute
> on the remaining work from checkpointed tasks, and automatically tune
> parameters for performance.
> Iterations will build on learnings from previous work, including the
> following:
> Technical reports:
> http://research.yahoo.com/files/yl-2012-002.pdf
> http://research.yahoo.com/files/yl-2012-003.pdf
> Source code:
> http://code.google.com/p/sailfish
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
