Reg the clockoutofSync exception, just check if your cluster has same time set. This problem comes when you have time differences.
Best Regards Ram -----Original Message----- From: Stuti Awasthi [mailto:stutiawas...@hcl.com] Sent: Tuesday, September 20, 2011 1:28 PM To: user@hbase.apache.org Subject: Queries on Zookeeper failure and RegionServer restartup Hi all, I have 2 node cluster. I run Regionserver, Zookeeper on both nodes and Master on 1 and Backup Master on other. Here what I did : I stopped Zookeeper on 1 node and after that I was unable to access Hbase. ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information. Queries : 1. If one of the zookeeper is going down , cluster is inaccessible then why we are running multiple zookeeper nodes? 2. Is there some way that if one of zookeeper nodes are working, cluster can be accessible? Some other test : If I stop RegionServer and Master on 1 node, then bakupMaster becomes Master and I can access the Hbase cluster but when I try to restart Region server on the same node on which I have shut down it gives me following error . How to fix this ? 2011-09-20 12:06:03,647 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server serverName=master,60020,1316500563205, load=(requests=0, regions=0, usedHeap=22, maxHeap=993): Unhandled exception: org.apache.hadoop.hbase.ClockOutOfSyncException: Server master,60020,1316500563205 has been rejected; Reported time is too far out of sync with master. Time difference of 352381ms > max allowed of 30000ms org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server master,60020,1316500563205 has been rejected; Reported time is too far out of sync with master. Time difference of 352381ms > max allowed of 30000ms at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces sorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc torAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.j ava:96) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException. java:80) at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServ er.java:1515) at org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionS erver.java:1479) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:57 1) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server master,60020,1316500563205 has been rejected; Reported time is too far out of sync with master. Time difference of 352381ms > max allowed of 30000ms at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.ja va:181) at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManag er.java:129) at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:615) Your inputs are required Thanks Stuti ________________________________ ::DISCLAIMER:: ---------------------------------------------------------------------------- ------------------------------------------- The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or HCL or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any mail and attachments please check them for viruses and defect. ---------------------------------------------------------------------------- -------------------------------------------
