Do you have a ZNode in ZK for the server you changed the IP? If the server is down, you might want to try to manually delete this ZNode and restart?
2013/11/8 Suresh V S <[email protected]> > Dear all, > > I had to change the IP address of a hbase region server in the cluster. > After effecting the IP change, all services in cluster came up fine, but > for the region server on this node. > > Kindly advise how to solve this problem. > > The log on the region server with the error portion is as below: > > 2013-11-08 13:41:13,780 DEBUG > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large > Compaction Thread to finish... > 2013-11-08 13:41:13,780 DEBUG > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Small > Compaction Thread to finish... > 2013-11-08 13:41:13,795 WARN > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node > /hbase/rs/harry,60020,1383898273175 already deleted, and this is not a retry > 2013-11-08 13:41:13,795 WARN > org.apache.hadoop.hbase.regionserver.HRegionServer: Failed deleting my > ephemeral node > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /hbase/rs/harry,60020,1383898273175 > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) > at > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:127) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:974) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:963) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1037) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:815) > at java.lang.Thread.run(Thread.java:662) > 2013-11-08 13:41:13,812 INFO org.apache.zookeeper.ZooKeeper: Session: > 0x14231e9cc68000c closed > 2013-11-08 13:41:13,812 INFO org.apache.zookeeper.ClientCnxn: EventThread > shut down > 2013-11-08 13:41:13,812 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server null; > zookeeper connection closed. > 2013-11-08 13:41:13,813 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020 > exiting > 2013-11-08 13:41:13,813 INFO > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook starting; > hbase.shutdown.hook=true; > fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@225f1ae9 > <mailto:fsShutdownHook > =org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@225f1ae9> > > Suresh. > > > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Disclaimer~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Information contained and transmitted by this e-mail is confidential and > proprietary to iGATE and its affiliates and is intended for use only by the > recipient. If you are not the intended recipient, you are hereby notified > that any dissemination, distribution, copying or use of this e-mail is > strictly prohibited and you are requested to delete this e-mail immediately > and notify the originator or [email protected] <mailto: > [email protected]>. iGATE does not enter into any agreement with any > party by e-mail. Any views expressed by an individual do not necessarily > reflect the view of iGATE. iGATE is not responsible for the consequences of > any actions taken on the basis of information provided, through this email. > The contents of an attachment to this e-mail may contain software viruses, > which could damage your own computer system. While iGATE has taken every > reasonable precaution to minimise this risk, we cannot accept liability for > any damage which you sustain as a result of software viruses. You should > carry out your own virus checks before opening an attachment. To know more > about iGATE please visit www.igate.com <http://www.igate.com>. > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >
