Joseph created HBASE-16276:
------------------------------
Summary: Inconsistency in Meta on cluster initialization and
inability to assign regions on cluster restarts
Key: HBASE-16276
URL: https://issues.apache.org/jira/browse/HBASE-16276
Project: HBase
Issue Type: Bug
Reporter: Joseph
Priority: Critical
I have recently been experimenting with a cluster running on the latest Master
commit: 519f87f
I initialize a cluster of 11 RegionServers and 3 HMaster's and then create a
giant pre-split table using HBase Pe (~500 regions). I then kill the entire
cluster while the HBase Pe test is still writing to the cluster. When I restart
the cluster, a single RS will somehow always fail to open up any regions. There
will also be a large number of regions stuck in transition with either
Pending_Close or Failed_Close on a variety of RS's in the cluster.
Running Hbase hbck when I first initialized the cluster produces:
***********************************************************************
HBaseFsck command line options:
Version: 2.0.0-fb10-SNAPSHOT
Number of live region servers: 12
Number of dead region servers: 0
Master: hbasectrl377.ash2.facebook.com,16000,1469225695984
Number of backup masters: 2
Average load: 0.16666666666666666
Number of requests: 0
Number of regions: 2
Number of regions in transition: 0
ERROR: hbase:meta, replicaId 0 is found on more than one region.
ERROR: hbase:meta table is not consistent. Run HBCK with proper fix options to
fix hbase:meta inconsistency. Exiting...
Summary:
2 inconsistencies detected.
Status: INCONSISTENT
***********************************************************************
Finally in the restarted cluster. The Meta region is assigned a region, which
looks very different from normal region names.
"hbase:meta,,1"
as opposed to
"hbase:meta,,1.1588230740"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
