[ https://issues.apache.org/jira/browse/MAPREDUCE-26?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J resolved MAPREDUCE-26. ------------------------------ Resolution: Not A Problem We attempt a proper rename now, and only fall back to copy+delete otherwise. > The shuffle keeps the ReduceTask locked while doing a FileSystem.rename > leading to task timeouts > ------------------------------------------------------------------------------------------------ > > Key: MAPREDUCE-26 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-26 > Project: Hadoop Map/Reduce > Issue Type: Bug > Reporter: Owen O'Malley > Assignee: Owen O'Malley > > The shuffle in ReduceTask.ReduceCopier.MapOutputCopier.copyOutput locks the > entire ReduceTask while doing a FileSystem.rename operation. Unfortunately > the RawLocalFileSystem implements rename as a copy and delete, which can take > a long time. As a result the reduce is being killed as not reporting progress > for 10 minutes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira