Cleaning up Zookeeper store should resolve this. Its quick but not 100% safe. The right way is to launch Zk CLI and manually check and verify all HBase related nodes By default, its under /HBASE_TMP_DIR/zookeeper or if you have specified it explicitly - check hbase.zookeeper.property.dataDir in hbase-site.xml
Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: [email protected] ________________________________________ From: Suresh V S [[email protected]] Sent: Friday, November 08, 2013 12:31 AM To: [email protected] Subject: Region Server not starting Dear all, I had to change the IP address of a hbase region server in the cluster. After effecting the IP change, all services in cluster came up fine, but for the region server on this node. Kindly advise how to solve this problem. The log on the region server with the error portion is as below: 2013-11-08 13:41:13,780 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish... 2013-11-08 13:41:13,780 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Small Compaction Thread to finish... 2013-11-08 13:41:13,795 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/rs/harry,60020,1383898273175 already deleted, and this is not a retry 2013-11-08 13:41:13,795 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Failed deleting my ephemeral node org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/rs/harry,60020,1383898273175 at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:127) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:974) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:963) at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1037) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:815) at java.lang.Thread.run(Thread.java:662) 2013-11-08 13:41:13,812 INFO org.apache.zookeeper.ZooKeeper: Session: 0x14231e9cc68000c closed 2013-11-08 13:41:13,812 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2013-11-08 13:41:13,812 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server null; zookeeper connection closed. 2013-11-08 13:41:13,813 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020 exiting 2013-11-08 13:41:13,813 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@225f1ae9<mailto:fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@225f1ae9> Suresh. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Disclaimer~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Information contained and transmitted by this e-mail is confidential and proprietary to iGATE and its affiliates and is intended for use only by the recipient. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, copying or use of this e-mail is strictly prohibited and you are requested to delete this e-mail immediately and notify the originator or [email protected] <mailto:[email protected]>. iGATE does not enter into any agreement with any party by e-mail. Any views expressed by an individual do not necessarily reflect the view of iGATE. iGATE is not responsible for the consequences of any actions taken on the basis of information provided, through this email. The contents of an attachment to this e-mail may contain software viruses, which could damage your own computer system. While iGATE has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checks before opening an attachment. To know more about iGATE please visit www.igate.com <http://www.igate.com>. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or [email protected] and delete or destroy any copy of this message and its attachments.
