[
https://issues.apache.org/jira/browse/MAPREDUCE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14101801#comment-14101801
]
Ming Ma commented on MAPREDUCE-4815:
------------------------------------
Siqi, thanks for the patch. It looks good overall.
Will the upgrade scenario work for task recovery? It seems the data stored in
the old output structure won't be honored by the new scheme after cluster
restarts with the patch; it skips the recoverTask given TempOutputPath doesn't
exist.
If that is the case, perhaps the patch can through some exception so that the
task attempt state changes to KILLED state for retry. Alternatively, the new
patch can be modified to handle the recovery of old directory structure, but
that seems over complicated.
> FileOutputCommitter.commitJob can be very slow for jobs with many output files
> ------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4815
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3, 2.0.1-alpha, 2.4.1
> Reporter: Jason Lowe
> Assignee: Siqi Li
> Attachments: MAPREDUCE-4815.v3.patch
>
>
> If a job generates many files to commit then the commitJob method call at the
> end of the job can take minutes. This is a performance regression from 1.x,
> as 1.x had the tasks commit directly to the final output directory as they
> were completing and commitJob had very little to do. The commit work was
> processed in parallel and overlapped the processing of outstanding tasks. In
> 0.23/2.x, the commit is single-threaded and waits until all tasks have
> completed before commencing.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
