Does anybody have a clue? Thanks lot. --- On Thu, 4/9/09, Steve Gao <steve....@yahoo.com> wrote:
From: Steve Gao <steve....@yahoo.com> Subject: [Interesting] One reducer randomly hangs on getting 0 mapper output To: core-user@hadoop.apache.org Date: Thursday, April 9, 2009, 6:04 PM I have hadoop jobs with the last 1 reducer randomly hangs on getting 0 mapper output. By randomly I mean the job sometimes works correctly, sometimes their last 1 reducer keeps reading map output but always gets 0 data. It would hang up to 100 hours for getting 0 data until I kill it. After I kill and re-run it, it could run correctly. The hung reducer could happen on any machine of my cluster. I attach the tail of the problematic reducer's log here. Does anybody have a hint what happened? syslog logs 2009-04-09 21:57:46,445 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0 Need 15 map output(s) 2009-04-09 21:57:46,446 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0: Got 0 new map-outputs & 0 obsolete map-outputs from tasktracker and 0 map-outputs from previous failures 2009-04-09 21:57:46,446 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0 Got 0 known map output location(s); scheduling... 2009-04-09 21:57:46,446 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0 Scheduled 0 of 0 known outputs (0 slow hosts and 0 dup hosts) 2009-04-09 21:57:51,453 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0 Need 15 map output(s) 2009-04-09 21:57:51,460 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0: Got 0 new map-outputs & 0 obsolete map-outputs from tasktracker and 0 map-outputs from previous failures 2009-04-09 21:57:51,460 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0 Got 0 known map output location(s); scheduling... 2009-04-09 21:57:51,460 INFO org.apache.hadoop.mapred.ReduceTask: task_200902022141_50382_r_000008_0 Scheduled 0 of 0 known outputs (0 slow hosts and 0 dup hosts) ... (forever)