Hi, We're running a hbase cluster including 37 regionservers. Today, we found losts of WrongRegionException when putting object into it.
hbase hbck -details reports that ==== Chain of regions in table STable is broken; edges does not contain ztxrGmCwn-6BE32s3cX1TNeHU_I= ERROR: Found inconsistency in table STable ==== echo "scan '.META.'"| hbase shell &> meta.txt grep -A1 "STARTKEY => 'EStore_everbox_z" meta.txt reports that ==== Ck=,1308802977279.71ffb1 1ffb10b8b95fd47b3eff468d00ab4e9.', STARTKEY => 'ztn0ukLW 0b8b95fd47b3eff468d00ab4 d1NSU3fuXKkkWq5ZVCk=', ENDKEY => 'ztqdVD8fCMP-dDbXUAydan e9. kboD4=', ENCODED => 71ffb10b8b95fd47b3eff468d00ab4e9, TABLE => {{NAME = -- D4=,1305619724446.c45191 45191821053d03537596f4a2e759718.', STARTKEY => ztqdVD8f 821053d03537596f4a2e7597 CMP-dDbXUAydankboD4=', ENDKEY => ' ztxrGmCwn-6BE32s3cX1TN 18. eHU_I=', ENCODED => c45191821053d03537596f4a2e759718, TABLE => {{NAME = -- pA=,1309455605341.c5c5f5 5c5f578722ea3f8d1b099313bec8298.', STARTKEY => 'zu3zVaLc 78722ea3f8d1b099313bec82 GDnnpjKCbnboXgAFspA=', ENDKEY => 'zu7qkr5fH6MMJ3GxbCv_0d 98. 6g8yI=', ENCODED => c5c5f578722ea3f8d1b099313bec8298, TABLE => {{NAME = ==== It looks like the meta indeed has a hole.(We tried scan '.META.' several times, to confirm it's not a transient status.) We've tried hbase hbck -fix, does not help. We found a thread 'wrong region exception' about two months ago. Stack suggested a 'little surgery' like ==== *So, make sure you actually have a hole. Dump out your meta table: echo "scan '.META.'"| ./bin/hbase shell &> /tmp/meta.txt Then look ensure that there is a hole between the above regions (compare start and end keys... the end key of one region needs to match the start key of the next). If indeed a hole, you need to do a little surgery inserting a new missing region (hbck should fix this but it doesn't have the smarts just yet). Basically, you create a new region with start and end keys to fill the hole then you insert it into .META. and then assign it. There are some scripts in our bin directory that do various parts of this. I'm pretty sure its beyond any but a few figuring this mess out so if you do the above foot work and provide a few more details, I'll hack up something for you (and hopefully something generalized to be use by others later, and later to be integrated into hbck).* ==== Can anyone give a detailed example, step by step instruction would be greatly appreciated. My understand is we should 1.Since we already has the lost region, we now have start and end keys. 2.generate the row represents the missing region. But how can I generate the encoded name? It looks like I need column=info:server,column=info:serverstartcode and column=info:regioninfo for the missing region. And column=info:regioninfo includes so many information. How to generate them one by one? As for the name of row, it consists of tablename, startkey, encode, and one more long number, how to get this number? 3.use assing command in the hbase shell We also tried check_meta.rb --fix, it reports ==== 11/07/06 00:09:08 WARN check_meta: hole after REGION => {NAME => 'STable,ztqdVD8fCMP-dDbXUAydankboD4=,1305619724446.c45191821053d03537596f4a2e759718.', STARTKEY => 'ztqdVD8fCMP-dDbXUAydankboD4=', ENDKEY => 'ztxrGmCwn-6BE32s3cX1TNeHU_I=', ENCODED => c45191821053d03537596f4a2e759718, TABLE => {{NAME => 'STable', FAMILIES => [{NAME => 'file', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'filelength', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'userbucket', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'userpass', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}} 11/07/06 00:28:40 WARN check_meta: Missing .regioninfo: hdfs:// hd0013.c.gj.com:9000/hbase/STable/3e6faca40a7ccad7ed8c0b5848c0f945/.regioninfo ==== The problem is still there. BTW, what about the blue warning? Is this a serious issue? The situation is quite hard to us, it looks like even we can fill the hole in the meta, we would lost all the data in the hole region, right? Thanks and regards, Mao Xu-Feng