Joseph created HBASE-16276: ------------------------------ Summary: Inconsistency in Meta on cluster initialization and inability to assign regions on cluster restarts Key: HBASE-16276 URL: https://issues.apache.org/jira/browse/HBASE-16276 Project: HBase Issue Type: Bug Reporter: Joseph Priority: Critical
I have recently been experimenting with a cluster running on the latest Master commit: 519f87f I initialize a cluster of 11 RegionServers and 3 HMaster's and then create a giant pre-split table using HBase Pe (~500 regions). I then kill the entire cluster while the HBase Pe test is still writing to the cluster. When I restart the cluster, a single RS will somehow always fail to open up any regions. There will also be a large number of regions stuck in transition with either Pending_Close or Failed_Close on a variety of RS's in the cluster. Running Hbase hbck when I first initialized the cluster produces: *********************************************************************** HBaseFsck command line options: Version: 2.0.0-fb10-SNAPSHOT Number of live region servers: 12 Number of dead region servers: 0 Master: hbasectrl377.ash2.facebook.com,16000,1469225695984 Number of backup masters: 2 Average load: 0.16666666666666666 Number of requests: 0 Number of regions: 2 Number of regions in transition: 0 ERROR: hbase:meta, replicaId 0 is found on more than one region. ERROR: hbase:meta table is not consistent. Run HBCK with proper fix options to fix hbase:meta inconsistency. Exiting... Summary: 2 inconsistencies detected. Status: INCONSISTENT *********************************************************************** Finally in the restarted cluster. The Meta region is assigned a region, which looks very different from normal region names. "hbase:meta,,1" as opposed to "hbase:meta,,1.1588230740" -- This message was sent by Atlassian JIRA (v6.3.4#6332)