[
https://issues.apache.org/jira/browse/MAPREDUCE-1308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791965#action_12791965
]
Brian Karlak commented on MAPREDUCE-1308:
-----------------------------------------
Arun --
Thanks for the pointers. I'm not quite sure how mapred.task.timeout got set
incorrectly -- I went through our local SVN repo, and it seems to have been set
that way at our site since we were using 0.16.4 back in July 2008. Since it
was never an issue until now, we never noticed, I guess. ;-)
Parameter has been modified.
I'll check the syslogs and report back in the next comment.
Brian
> reduce tasks stall and are eventually killed
> --------------------------------------------
>
> Key: MAPREDUCE-1308
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1308
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Environment: 20-node cluster, 8 cores per machine, 32GB memory,
> Fedora Linux
> Reporter: Brian Karlak
>
> We recently migrated our 0.19.2 cluster from Gentoo Linux to Fedora Linux.
> Everything was running smoothly before, but now about 5%-10% of our jobs have
> at least one reduce task that stalls out and is eventually killed with the
> message:
> Task attempt_200912102211_1648_r_000009_0 failed to report status for
> 6003 seconds. Killing!
> The task is then re-launched and completes successfully, usually in a couple
> of minutes.
> This is problematic because our scheduled Hadoop jobs now take an extra
> hour-and-a-half to run (6000 seconds).
> There are no indications in the logs that anything is amiss. The task
> starts, a small amount of the copy/shuffle runs, and then nothing is else is
> heard from the task until it is killed. I will attach the relevant parts of
> the TaskTracker logs in the comments.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
