Hi Lance, Can I ask what version you were running when you saw this? Is it reproducible? I'm trying to look at the code path that might produce such a behavior and want to make sure I'm looking at the right version.
Thanks -Todd On Thu, Apr 30, 2009 at 9:33 AM, Lance Riedel <la...@dotspots.com> wrote: > Has anyone seen this before? Our task tracker produced a 2.7 gig log file > in a few hours. The entry is all the same (every 2 ms): > > 2009-04-30 02:34:40,207 INFO org.apache.hadoop.mapred.TaskTracker: > Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with > reponseId '5341 > 2009-04-30 02:34:40,398 INFO org.apache.hadoop.mapred.TaskTracker: > Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with > reponseId '5341 > 2009-04-30 02:34:40,403 INFO org.apache.hadoop.mapred.TaskTracker: > Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with > reponseId '5341 > 2009-04-30 02:34:40,411 INFO org.apache.hadoop.mapred.TaskTracker: > Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with > reponseId '5341 > 2009-04-30 02:34:40,414 INFO org.apache.hadoop.mapred.TaskTracker: > Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with > reponseId '5341 > 2009-04-30 02:34:40,417 INFO org.apache.hadoop.mapred.TaskTracker: > Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with > reponseId '5341 > 2009-04-30 02:34:40,420 INFO org.apache.hadoop.mapred.TaskTracker: > Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with > reponseId '5341 > ... (And on and on and on...) > > > These are the few lines before it started: > > 2009-04-30 02:34:29,780 INFO > org.apache.hadoop.mapred.TaskTracker.clienttrace: src: > xxx.xxx.xxx.xxx:50060, dest: 10.253.178.95:40268, bytes: 3341324, op: > MAPRED_SHUFFLE, cliID: attempt_200904291917_0352_m_000006_0 > 2009-04-30 02:34:31,522 INFO org.apache.hadoop.mapred.TaskTracker: Sent out > 418891 bytes for reduce: 12 from map: attempt_200904291917_0352_m_000007_0 > given 418891/418887 from 4301462 with (22, 171) > 2009-04-30 02:34:31,522 INFO > org.apache.hadoop.mapred.TaskTracker.clienttrace: src: > xxx.xxx.xxx.xxx:50060, dest: xxx.xxx.xxx.xxx:40268, bytes: 418891, op: > MAPRED_SHUFFLE, cliID: attempt_200904291917_0352_m_000007_0 > 2009-04-30 02:34:35,382 INFO org.apache.hadoop.mapred.TaskTracker: > attempt_200904291917_0352_r_000003_0 0.3030303% reduce > copy (10 of 11 at > 0.32 MB/s) > > 2009-04-30 02:34:38,385 INFO org.apache.hadoop.mapred.TaskTracker: > attempt_200904291917_0352_r_000003_0 0.3030303% reduce > copy (10 of 11 at > 0.32 MB/s) > > 2009-04-30 02:34:40,207 INFO org.apache.hadoop.mapred.TaskTracker: > Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with > reponseId '5341 > 2009-04-30 02:34:40,398 INFO org.apache.hadoop.mapred.TaskTracker: > Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with > reponseId '5341 > 2009-04-30 02:34:40,403 INFO org.apache.hadoop.mapred.TaskTracker: > Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with > reponseId '5341 > 2009-04-30 02:34:40,411 INFO org.apache.hadoop.mapred.TaskTracker: > Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with > reponseId '5341 > 2009-04-30 02:34:40,414 INFO org.apache.hadoop.mapred.TaskTracker: > Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with > reponseId '5341 > 2009-04-30 02:34:40,417 INFO org.apache.hadoop.mapred.TaskTracker: > Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with > reponseId '5341 > 2009-04-30 02:34:40,420 INFO org.apache.hadoop.mapred.TaskTracker: > Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with > reponseId '5341 > > --And on for 2+ gigs >
