Re: Infinite Loop Resending status from task tracker

Thu, 30 Apr 2009 10:46:07 -0700

I have not been able to reproduce. We are using version 19.1 with the following patches: 
4780-2v19.patch (Jira  4780)
closeAll3.patch (Jira 3998)

Thanks,
Lance

On Apr 30, 2009, at 10:40 AM, Todd Lipcon wrote:


Hi Lance,

Can I ask what version you were running when you saw this? Is it
reproducible? I'm trying to look at the code path that might produce such a 
behavior and want to make sure I'm looking at the right version.

Thanks
-Todd
On Thu, Apr 30, 2009 at 9:33 AM, Lance Riedel <la...@dotspots.com> wrote:
Has anyone seen this before? Our task tracker produced a 2.7 gig log file 
in a few hours. The entry is all the same (every 2 ms):

2009-04-30 02:34:40,207 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with
reponseId '5341
2009-04-30 02:34:40,398 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with
reponseId '5341
2009-04-30 02:34:40,403 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with
reponseId '5341
2009-04-30 02:34:40,411 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with
reponseId '5341
2009-04-30 02:34:40,414 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with
reponseId '5341
2009-04-30 02:34:40,417 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with
reponseId '5341
2009-04-30 02:34:40,420 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with
reponseId '5341
... (And on and on and on...)


These are the few lines before it started:

2009-04-30 02:34:29,780 INFO
org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
xxx.xxx.xxx.xxx:50060, dest: 10.253.178.95:40268, bytes: 3341324, op:
MAPRED_SHUFFLE, cliID: attempt_200904291917_0352_m_000006_0
2009-04-30 02:34:31,522 INFO org.apache.hadoop.mapred.TaskTracker: Sent out 418891 bytes for reduce: 12 from map: attempt_200904291917_0352_m_000007_0 
given 418891/418887 from 4301462 with (22, 171)
2009-04-30 02:34:31,522 INFO
org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
xxx.xxx.xxx.xxx:50060, dest: xxx.xxx.xxx.xxx:40268, bytes: 418891, op: 
MAPRED_SHUFFLE, cliID: attempt_200904291917_0352_m_000007_0
2009-04-30 02:34:35,382 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_200904291917_0352_r_000003_0 0.3030303% reduce > copy (10 of 11 at 
0.32 MB/s) >
2009-04-30 02:34:38,385 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_200904291917_0352_r_000003_0 0.3030303% reduce > copy (10 of 11 at 
0.32 MB/s) >
2009-04-30 02:34:40,207 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with
reponseId '5341
2009-04-30 02:34:40,398 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with
reponseId '5341
2009-04-30 02:34:40,403 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with
reponseId '5341
2009-04-30 02:34:40,411 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with
reponseId '5341
2009-04-30 02:34:40,414 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with
reponseId '5341
2009-04-30 02:34:40,417 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with
reponseId '5341
2009-04-30 02:34:40,420 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with
reponseId '5341

--And on for    2+ gigs

Reply via email to