Hi, I've deployed hbase on 5 nodes cluster of amazon ec2 successfully and was working fine. The next day when I logon I changed the configuration file(regionservers list,slaves, masters,dfs rootdir etc) in hadoop and hbase /conf directory for the new IP address which is dynamic for ec2 then the hbase master immediately stopped after startup and generated the error message in log files, it tries to connect to the old region server address and failed. Again I removed both hadoop and hbase for fresh installation but it again generates the error in connection with an old ip address on port 60020. I would be grateful if someone could getme out of it. Thanks.
log of hbase-root-master .......... 2011-02-18 05:13:32,760 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Handling OPENED event for 70236052; deleting unassigned node 2011-02-18 05:13:32,760 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x2e3842768e0003 Deleting existing unassigned node for 70236052 that is in expected state RS_ZK_REGION_OPENED 2011-02-18 05:13:32,831 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x2e3842768e0003 Successfully deleted unassigned node for region 70236052 in expected state RS_ZK_REGION_OPENED 2011-02-18 05:13:32,831 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Opened region -ROOT-,,0.70236052 on ip-10-203-13-80.ec2.internal,60020,1298024003961 2011-02-18 05:13:52,794 INFO org.apache.hadoop.ipc.HbaseRPC: Problem connecting to server: ip-10-114-29-91.ec2.internal/10.114.29.91:60020 2011-02-18 05:14:13,811 INFO org.apache.hadoop.ipc.HbaseRPC: Problem connecting to server: ip-10-114-29-91.ec2.internal/10.114.29.91:60020 2011-02-18 05:14:34,817 INFO org.apache.hadoop.ipc.HbaseRPC: Problem connecting to server: ip-10-114-29-91.ec2.internal/10.114.29.91:60020 2011-02-18 05:14:34,817 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown. java.net.SocketTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=ip-10-114-29-91.ec2.internal/10.114.29.91:60020] at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:213) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:311) at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) at $Proxy6.getProtocolVersion(Unknown Source) at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419) at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393) at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444) at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:954) at org.apache.hadoop.hbase.catalog.CatalogTracker.getCachedConnection(CatalogTracker.java:384) at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:283) at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyMetaRegionLocation(CatalogTracker.java:478) at org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:435) at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:382) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:277) 2011-02-18 05:14:34,841 INFO org.apache.hadoop.hbase.master.HMaster: Aborting 2011-02-18 05:14:34,841 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping service threads 2011-02-18 05:14:34,841 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60000 2011-02-18 05:14:34,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60000: exiting 2011-02-18 05:14:34,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60000: exiting ..................................