MR2: Map tasks rewrite data once even if output fits in sort buffer
-------------------------------------------------------------------
Key: MAPREDUCE-3252
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3252
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, task
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Critical
I found that, even if the output of a map task fits entirely in its sort
buffer, it was rewriting the output entirely rather than just renaming the
first spill into place. This is due to RawLocalFileSystem.rename() falling back
to a copy if renameTo() fails. The first rename attempt was failing because no
one has called mkdir for the output directory yet.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
