NoRouteToHostException during balancing will cause Master abort ---------------------------------------------------------------
Key: HBASE-3617 URL: https://issues.apache.org/jira/browse/HBASE-3617 Project: HBase Issue Type: Bug Reporter: stack Priority: Critical Fix For: 0.90.2 Via Tatsuya up on the list: {code} 2011-03-10 07:48:39,192 FATAL org.apache.hadoop.hbase.master.HMaster: Remote unexpected exception java.net.NoRouteToHostException: No route to host at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java: 206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408) at org.apache.hadoop.hbase.ipc.HBaseClient $Connection.setupIOstreams(HBaseClient.java:328) at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java: 883) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750) at org.apache.hadoop.hbase.ipc.HBaseRPC $Invoker.invoke(HBaseRPC.java:257) at $Proxy6.closeRegion(Unknown Source) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java: 589) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java: 1093) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java: 1040) at org.apache.hadoop.hbase.master.AssignmentManager.balance(AssignmentManager.java: 1831) at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java: 692) at org.apache.hadoop.hbase.master.HMaster$1.chore(HMaster.java: 583) at org.apache.hadoop.hbase.Chore.run(Chore.java:66) 2011-03-10 07:48:39,192 INFO org.apache.hadoop.hbase.master.HMaster: Aborting 2011-03-10 07:48:39,192 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=SpecialObject_Speed_Test,, 1299710751983.f0e5544339870a510c338b3029979d3e., src=ap13.secur2,60020,1299710609447, dest=ap12.secur2,60020,1299710609148 2011-03-10 07:48:39,192 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of region SpecialObject_Speed_Test,, 1299710751983.f0e5544339870a510c338b3029979d3e. (offlining) 2011-03-10 07:48:39,852 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping service threads 2011-03-10 07:48:39,852 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60000 2011-03-10 07:48:39,852 FATAL org.apache.hadoop.hbase.master.HMaster: Remote unexpected exception java.io.InterruptedIOException: Interruped while waiting for IO on channel java.nio.channels.SocketChannel[connection-pending remote=/ 10.X.X.18:60020]. 19340 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout $SelectorPool.select(SocketIOWithTimeout.java:349) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java: 203) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408) at org.apache.hadoop.hbase.ipc.HBaseClient $Connection.setupIOstreams(HBaseClient.java:328) at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java: 883) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750) at org.apache.hadoop.hbase.ipc.HBaseRPC $Invoker.invoke(HBaseRPC.java:257) at $Proxy6.closeRegion(Unknown Source) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java: 589) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java: 1093) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java: 1040) at org.apache.hadoop.hbase.master.AssignmentManager.balance(AssignmentManager.java: 1831) at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java: 692) at org.apache.hadoop.hbase.master.HMaster$1.chore(HMaster.java: 583) at org.apache.hadoop.hbase.Chore.run(Chore.java:66) 2011-03-10 07:48:39,852 INFO org.apache.hadoop.hbase.master.HMaster: Aborting {code} -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira