[
https://issues.apache.org/jira/browse/HADOOP-4654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652361#action_12652361
]
Amareshwari Sriramadasu commented on HADOOP-4654:
-------------------------------------------------
bq. The case will be visible when one job (job1) finishes while others (job2)
are in middle where job1's speculative tasks might now fill up the commit queue.
If job1 has finished, it will be a no-op in TaskCommitThread, since the higher
lever directory will be deleted in garbage collect. It will still do a cleanup
for corner cases described at
http://issues.apache.org/jira/browse/HADOOP-2391?focusedCommentId=12566183#action_12566183,
which sounds good.
I also ran sort benchmark with and without the patch. And I didnt see any
performance difference,
> remove temporary output directory of failed tasks
> -------------------------------------------------
>
> Key: HADOOP-4654
> URL: https://issues.apache.org/jira/browse/HADOOP-4654
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.2, 0.18.1
> Reporter: Christian Kunz
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.20.0
>
> Attachments: patch-4654-0.18.txt
>
>
> When dfs is getting full (80+% of reserved space), the rate of write failures
> increases, such that more map-reduce tasks can fail. By not cleaning up the
> temporary output directory of tasks the situation worsens over the lifetime
> of a job, increasing the probability of the whole job failing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
