Hi
Recently we tried adding a new regionserver to the cluster. The region
server had the following logs for all the regions it tried to open.
2015-08-14 01:23:52,175 ERROR [RS_OPEN_REGION-localhost:16020-0]
handler.OpenRegionHandler: Failed open of
region=testnamespace:testtable,test_14390754745,1439165529250.7aaa895f2ec3ca602c50fcc69120c01e.,
starting to roll back the global memstore size.
java.lang.IllegalStateException: Could not instantiate a region instance.
at
org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:4566)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4873)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4845)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4801)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4752)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:356)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:126)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:4563)
... 10 more
Caused by: java.lang.IllegalArgumentException: Need table descriptor
at
org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:601)
at
org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:578)
... 15 more
In addition to this, all other region server began to stop serving request.
One thing to note is during this time a major compaction was running for a
table.
Also the hbase:meta table was in the new rs.
The clients had the exception
org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location
and jstack of clients were waiting at locateRegionInMeta.
On decommissioning the new node, the existing regionserver started serving
request. What would have caused this problem? We are using hbase-1.0.0.
Thanks
Regards
Mukund Murrali
