Hi!

I'm running streaming tasks on hadoop 0.17.0, and wondered, if anyone has an 
approach to debugging the following situation:

-) map have all finished (100% in http display),
-) some reducers are hanging, with the messages below.

Notice, that the task had 100 map tasks at allo, so 58 seems like an 
extraordinary high number of missing parts, long after map has officially 
finished. Plus it seems to be deterministic, it always stop at 3 reduce parts 
not finishing, although I haven't yet checked if they are always the same 
errors or not.

> 2008-06-30 15:25:41,953 INFO org.apache.hadoop.mapred.ReduceTask:
> task_200806300847_0002_r_000014_0 Need 58 map output(s) 2008-06-30
> 15:25:41,953 INFO org.apache.hadoop.mapred.ReduceTask:
> task_200806300847_0002_r_000014_0: Got 0 new map-outputs & 0 obsolete
> map-outputs from tasktracker and 0 map-outputs from previous failures
> 2008-06-30 15:25:41,954 INFO org.apache.hadoop.mapred.ReduceTask:
> task_200806300847_0002_r_000014_0 Got 0 known map output location(s);
> scheduling... 2008-06-30 15:25:41,954 INFO
> org.apache.hadoop.mapred.ReduceTask: task_200806300847_0002_r_000014_0
> Scheduled 0 of 0 known outputs (0 slow hosts and 0 dup hosts) 2008-06-30
> 15:25:46,770 INFO org.apache.hadoop.streaming.PipeMapRed: MRErrorThread
> done 2008-06-30 15:25:46,963 INFO org.apache.hadoop.mapred.ReduceTask:
> task_200806300847_0002_r_000014_0 Need 58 map output(s) 2008-06-30
> 15:25:46,963 INFO org.apache.hadoop.mapred.ReduceTask:
> task_200806300847_0002_r_000014_0: Got 0 new map-outputs & 0 obsolete
> map-outputs from tasktracker and 0 map-outputs from previous failures
> 2008-06-30 15:25:46,964 INFO org.apache.hadoop.mapred.ReduceTask:
> task_200806300847_0002_r_000014_0 Got 0 known map output location(s);
> scheduling... 2008-06-30 15:25:46,964 INFO
> org.apache.hadoop.mapred.ReduceTask: task_200806300847_0002_r_000014_0
> Scheduled 0 of 0 known outputs (0 slow hosts and 0 dup hosts)

TIA,

Andreas

Attachment: signature.asc
Description: This is a digitally signed message part.

Reply via email to