Hi, We have a 200 node hadoop cluster (0.20.0) and have tweaked namenode and datanode handler to 40 and 10. The xcievers also had to be changed to 8192. But during the mapred jobs, we are seeing lot of task attempt failures saying "connection reset by peer". Following exception are there in the namenode logs. The tcp connection failures on the namenode also seems to be high. What could be wrong? (I have >6G heap for namenode). Do we need to increase the handlers further ?
Another side effect of this issue is that the SequeceFileOutput of the job seems to be corrupted. The next job is not able read some of these sequencefiles created by the previous job(eventhough the first job eventually succeeds after lot of connection related failures). namenode exceptions: java.io.IOException: Connection reset by peer 2009-10-01 00:44:32,329 INFO org.apache.hadoop.ipc.Server: IPC Server handler 24 on 9000 caught: java.nio.channels.ClosedChannelException 2009-10-01 00:44:32,330 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9000 caught: java.nio.channels.ClosedChannelException 2009-10-01 00:44:32,331 INFO org.apache.hadoop.ipc.Server: IPC Server handler 29 on 9000 caught: java.nio.channels.ClosedChannelException 2009-10-01 00:44:32,343 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9000 caught: java.nio.channels.ClosedChannelException 2009-10-01 00:44:34,575 INFO org.apache.hadoop.ipc.Server: IPC Server handler 21 on 9000 caught: java.nio.channels.ClosedChannelException 2009-10-01 00:44:34,601 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9000: readAndProcess threw exception java.io.IOException: C onnection reset by peer. Count of bytes read: 0 java.io.IOException: Connection reset by peer 2009-10-01 00:44:34,943 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9000: readAndProcess threw exception java.io.IOException: C onnection reset by peer. Count of bytes read: 0 java.io.IOException: Connection reset by peer 2009-10-01 00:44:35,641 INFO org.apache.hadoop.ipc.Server: IPC Server handler 16 on 9000 caught: java.nio.channels.ClosedChannelException 2009-10-01 00:44:40,380 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9000: readAndProcess threw exception java.io.IOException: C onnection reset by peer. Count of bytes read: 0 Exception reading sequence file by the next job: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63) at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101) at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1930) at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2062) at org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:76) at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:191) at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:175) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48) Thanks, Murali Krishna