I am sorry, but I received an error when I sent the message to the list and all responses were sent to my junk mail. So I tried to send it again, and just then noticed your emails.
Sorry!! -----Original Message----- From: Harsh J [mailto:ha...@cloudera.com] Sent: segunda-feira, 23 de julho de 2012 11:07 To: common-user@hadoop.apache.org Subject: Re: Datanode error Pablo, Perhaps you've forgotten about it but you'd ask the same question last week and you did have some responses on it. Please see your earlier thread at http://search-hadoop.com/m/0BOOh17ugmD On Mon, Jul 23, 2012 at 7:27 PM, Pablo Musa <pa...@psafe.com> wrote: > Hey guys, > I have a cluster with 11 nodes (1 NN and 10 DNs) which is running and working. > However my datanodes keep having the same errors, over and over. > > I googled the problems and tried different flags (ex: > -XX:MaxDirectMemorySize=2G) and different configs (xceivers=8192) but could > not solve it. > > Does anyone know what is the problem and how can I solve it? (the > stacktrace is at the end) > > I am running: > Java 1.7 > Hadoop 0.20.2 > Hbase 0.90.6 > Zoo 3.3.5 > > % top -> shows low load average (6% most of the time up to 60%), > already considering the number of cpus % vmstat -> shows no swap at > all % sar -> shows 75% idle cpu in the worst case > > Hope you guys can help me. > Thanks in advance, > Pablo > > 2012-07-20 00:03:44,455 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: > /DN01:50010, dest: > /DN01:43516, bytes: 396288, op: HDFS_READ, cliID: > DFSClient_hb_rs_DN01,60020,1342734302945_1342734303427, offset: 54956544, > srvID: DS-798921853-DN01-50010-1328651609047, blockid: > blk_914960691839012728_14061688, duration: > 480061254006 > 2012-07-20 00:03:44,455 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: > DatanodeRegistration(DN01:50010, > storageID=DS-798921853-DN01-50010-1328651609047, infoPort=50075, > ipcPort=50020):Got exception while serving blk_914960691839012728_14061688 to > /DN01: > java.net.SocketTimeoutException: 480000 millis timeout while waiting for > channel to be ready for write. ch : java.nio.channels.SocketChannel[connected > local=/DN01:50010 remote=/DN01:43516] > at > org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246) > at > org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159) > at > org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:397) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:493) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:279) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.jav > a:175) > > 2012-07-20 00:03:44,455 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: > DatanodeRegistration(DN01:50010, > storageID=DS-798921853-DN01-50010-1328651609047, infoPort=50075, > ipcPort=50020):DataXceiver > java.net.SocketTimeoutException: 480000 millis timeout while waiting for > channel to be ready for write. ch : java.nio.channels.SocketChannel[connected > local=/DN01:50010 remote=/DN01:43516] > at > org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246) > at > org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159) > at > org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:397) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:493) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:279) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.jav > a:175) > > 2012-07-20 00:12:11,949 INFO > org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification > succeeded for blk_4602445008578088178_5707787 > 2012-07-20 00:12:11,962 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock > blk_-8916344806514717841_14081066 received exception > java.net.SocketTimeoutException: 63000 millis timeout while waiting > for channel to be ready for read. ch : > java.nio.channels.SocketChannel[connected local=/DN01:36634 > remote=/DN03:50010] > 2012-07-20 00:12:11,962 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: > DatanodeRegistration(DN01:50010, > storageID=DS-798921853-DN01-50010-1328651609047, infoPort=50075, > ipcPort=50020):DataXceiver > java.net.SocketTimeoutException: 63000 millis timeout while waiting for > channel to be ready for read. ch : java.nio.channels.SocketChannel[connected > local=/DN01:36634 remote=/DN03:50010] > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:116) > at java.io.FilterInputStream.read(FilterInputStream.java:83) > at java.io.DataInputStream.readShort(DataInputStream.java:312) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:447) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.jav > a:183) > > > 2012-07-20 00:12:20,670 INFO > org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification > succeeded for blk_7238561256016868237_3555939 > 2012-07-20 00:12:22,541 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block > blk_-7028120671250332363_14081073 src: /DN03:50331 dest: /DN01:50010 > 2012-07-20 00:12:22,544 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in > receiveBlock for block blk_-7028120671250332363_14081073 > java.io.EOFException: while trying to read 65557 bytes > 2012-07-20 00:12:22,544 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > PacketResponder 0 for block blk_-7028120671250332363_14081073 Interrupted. > 2012-07-20 00:12:22,544 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for > block blk_-7028120671250332363_14081073 terminating > 2012-07-20 00:12:22,544 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock > blk_-7028120671250332363_14081073 received exception > java.io.EOFException: while trying to read 65557 bytes > 2012-07-20 00:12:22,544 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: > DatanodeRegistration(DN01:50010, > storageID=DS-798921853-DN01-50010-1328651609047, infoPort=50075, > ipcPort=50020):DataXceiver > java.io.EOFException: while trying to read 65557 bytes > at > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:290) > at > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:334) > at > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:398) > at > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:577) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:494) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.jav > a:183) > > > 2012-07-20 00:12:34,266 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block > blk_-1834839455324747507_14081046 src: /DN05:59897 dest: /DN01:50010 > 2012-07-20 00:12:34,267 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in > receiveBlock for block blk_-1834839455324747507_14081046 > java.io.EOFException: while trying to read 65557 bytes > 2012-07-20 00:12:34,268 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > PacketResponder 0 for block blk_-1834839455324747507_14081046 Interrupted. > 2012-07-20 00:12:34,268 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for > block blk_-1834839455324747507_14081046 terminating > 2012-07-20 00:12:34,268 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock > blk_-1834839455324747507_14081046 received exception > java.io.EOFException: while trying to read 65557 bytes > 2012-07-20 00:12:34,268 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: > DatanodeRegistration(DN01:50010, > storageID=DS-798921853-DN01-50010-1328651609047, infoPort=50075, > ipcPort=50020):DataXceiver > java.io.EOFException: while trying to read 65557 bytes > at > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:290) > at > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:334) > at > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:398) > at > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:577) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:494) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.jav > a:183) > 2012-07-20 00:12:34,269 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block > blk_3941134611454287401_14080990 src: /DN03:50345 dest: /DN01:50010 > 2012-07-20 00:12:34,270 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in > receiveBlock for block blk_3941134611454287401_14080990 > java.io.EOFException: while trying to read 65557 bytes > 2012-07-20 00:12:34,270 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > PacketResponder 0 for block blk_3941134611454287401_14080990 Interrupted. > 2012-07-20 00:12:34,271 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for > block blk_3941134611454287401_14080990 terminating > 2012-07-20 00:12:34,271 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock > blk_3941134611454287401_14080990 received exception > java.io.EOFException: while trying to read 65557 bytes > 2012-07-20 00:12:34,271 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: > DatanodeRegistration(DN01:50010, > storageID=DS-798921853-DN01-50010-1328651609047, infoPort=50075, > ipcPort=50020):DataXceiver > java.io.EOFException: while trying to read 65557 bytes > at > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:290) > at > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:334) > at > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:398) > at > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:577) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:494) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.jav > a:183) > -- Harsh J
