At one point my admin staff added 5 machines to my cluster but accidently left port 50010 among onthers firewalled. This resulted in chaos for a while until the firewall was found.
On Fri, Nov 20, 2009 at 1:01 PM, Bill Brune <bbr...@decarta.com> wrote: > > Hi I'm trying to get a small cluster up with hdfs (hadoop 0.20.1) > > It is working somewhat. I can create files and directories and retrieve > them. However I am seeing a random failure rate (about 25%) where I get > this error when storing a file, and the resultant file is stored as a 0 len > file (as reported by hadoop fs -ls). > I set this all up with straight IP addresses because I have no access to > the DNS servers here. However the datanodes do have hostnames that do NOT > resolve in DNS. (not sure if that matters) > > > Put failed: 09/11/19 15:40:09 INFO hdfs.DFSClient: Exception in > createBlockOutputStream java.io.IOException: Bad connect ack with > firstBadLink 10.241.4.101:50010 > > 09/11/19 15:40:09 INFO hdfs.DFSClient: Abandoning block > blk_8325005803148307980_1082 > > -etc,etc > > I've verified that the java process is running and listening on port 50010 > (as reported by netstat -plnt) and passphraseless ssh is working fine. > (also, the namenode web page intermittently fails to connect when asked to > browse the filesystem, (works most of the time) > > The logs on that datanode show a java.net.NoRouteToHostException for a few > blocks, then it seems to start receiving blocks fine. > > The log .... > > > 2009-11-19 06:12:30,149 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block > blk_6892009382281192058_1068 > src: /10.241.4.101:41570 dest: /10.241.4.101:50010 > 2009-11-19 06:12:30,150 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock > blk_6892009382281192058_1068 rece > ived exception java.net.NoRouteToHostException: No route to host > 2009-11-19 06:12:30,151 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration( > 10.241.4.101:50010, st > orageID=DS-886003042-127.0.0.1-50010-1258634370537, infoPort=50075, > ipcPort=50020):DataXceiver > java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) > at > org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:282) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103) > at java.lang.Thread.run(Unknown Source) > 2009-11-19 06:12:36,155 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block > blk_-7007024210780549144_106 > 8 src: /10.241.4.101:41574 dest: /10.241.4.101:50010 > 2009-11-19 06:12:36,157 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock > blk_-7007024210780549144_1068 rec > eived exception java.net.NoRouteToHostException: No route to host > 2.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) > at > org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:282) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103) > at java.lang.Thread.run(Unknown Source) > 2009-11-19 06:12:30,149 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block > blk_6892009382281192058_1068 > src: /10.241.4.101:41570 dest: /10.241.4.101:50010 > 2009-11-19 06:12:30,150 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock > blk_6892009382281192058_1068 rece > ived exception java.net.NoRouteToHostException: No route to host > 2009-11-19 06:12:30,151 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration( > 10.241.4.101:50010, storageID=DS-886003042-127.0.0.1-50010-1258634370537, > infoPort=50075, ipcPort=50020):DataXceiver > java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) > at > org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:282) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103) > at java.lang.Thread.run(Unknown Source) > 2009-11-19 06:12:36,155 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block > blk_-7007024210780549144_1068 src: /10.241.4.101:41574 dest: / > 10.241.4.101:50010 > 2009-11-19 06:12:36,157 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock > blk_-7007024210780549144_1068 received exception > java.net.NoRouteToHostException: No route to host > 2009-11-19 06:12:36,157 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration( > 10.241.4.101:50010, storageID=DS-886003042-127.0.0.1-50010-1258634370537, > infoPort=50075, ipcPort=50020):DataXceiver > java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) > at > org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:282) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103) > at java.lang.Thread.run(Unknown Source) > 2009-11-19 06:15:31,776 INFO > org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification > succeeded for blk_6602943802956432390_1059 > 2009-11-19 06:37:06,201 INFO > org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification > succeeded for blk_-3274535555692844186_1063 > 2009-11-19 06:45:15,117 INFO > org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification > succeeded for blk_753873727864345115_1067 > 2009-11-19 06:51:56,888 INFO > org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification > succeeded for blk_-810392349062598446_1062 > 2009-11-19 07:03:17,441 INFO > org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification > succeeded for blk_-4850126846096803878_1066 > 2009-11-19 07:09:26,335 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 12 blocks > got processed in 3 > msecs > > > > Any ideas would be greatly appreciated. > > Thanks -Bill > -- Pro Hadoop, a book to guide you from beginner to hadoop mastery, http://www.amazon.com/dp/1430219424?tag=jewlerymall www.prohadoopbook.com a community for Hadoop Professionals
