We also check the master log, nothing interesting found. On Wed, Jul 6, 2011 at 12:58 AM, Xu-Feng Mao <m9s...@gmail.com> wrote:
> Hi, > > We're running a hbase cluster including 37 regionservers. Today, we found > losts of WrongRegionException when putting object into it. > > hbase hbck -details > reports that > ==== > Chain of regions in table STable is broken; edges does not contain > ztxrGmCwn-6BE32s3cX1TNeHU_I= > ERROR: Found inconsistency in table STable > ==== > > echo "scan '.META.'"| hbase shell &> meta.txt > grep -A1 "STARTKEY => 'EStore_everbox_z" meta.txt > reports that > ==== > Ck=,1308802977279.71ffb1 1ffb10b8b95fd47b3eff468d00ab4e9.', > STARTKEY => 'ztn0ukLW > 0b8b95fd47b3eff468d00ab4 d1NSU3fuXKkkWq5ZVCk=', ENDKEY => > 'ztqdVD8fCMP-dDbXUAydan > e9. kboD4=', ENCODED => > 71ffb10b8b95fd47b3eff468d00ab4e9, TABLE => {{NAME = > -- > D4=,1305619724446.c45191 45191821053d03537596f4a2e759718.', > STARTKEY => ztqdVD8f > 821053d03537596f4a2e7597 CMP-dDbXUAydankboD4=', ENDKEY => ' > ztxrGmCwn-6BE32s3cX1TN > 18. eHU_I=', ENCODED => > c45191821053d03537596f4a2e759718, TABLE => {{NAME = > -- > pA=,1309455605341.c5c5f5 5c5f578722ea3f8d1b099313bec8298.', > STARTKEY => 'zu3zVaLc > 78722ea3f8d1b099313bec82 GDnnpjKCbnboXgAFspA=', ENDKEY => > 'zu7qkr5fH6MMJ3GxbCv_0d > 98. 6g8yI=', ENCODED => > c5c5f578722ea3f8d1b099313bec8298, TABLE => {{NAME = > ==== > > It looks like the meta indeed has a hole.(We tried scan '.META.' several > times, to confirm it's not a transient status.) > We've tried hbase hbck -fix, does not help. > > We found a thread 'wrong region exception' about two months ago. Stack > suggested a 'little surgery' like > ==== > > *So, make sure you actually have a hole. Dump out your meta table: > > echo "scan '.META.'"| ./bin/hbase shell &> /tmp/meta.txt > > Then look ensure that there is a hole between the above regions > (compare start and end keys... the end key of one region needs to > match the start key of the next). > > If indeed a hole, you need to do a little surgery inserting a new > missing region (hbck should fix this but it doesn't have the smarts > just yet). > > Basically, you create a new region with start and end keys to fill the > hole then you insert it into .META. and then assign it. There are > some scripts in our bin directory that do various parts of this. I'm > pretty sure its beyond any but a few figuring this mess out so if you > do the above foot work and provide a few more details, I'll hack up > something for you (and hopefully something generalized to be use by > others later, and later to be integrated into hbck).* > > ==== > > Can anyone give a detailed example, step by step instruction would be > greatly appreciated. > My understand is we should > 1.Since we already has the lost region, we now have start and end keys. > 2.generate the row represents the missing region. But how can I generate > the encoded name? > It looks like I need > column=info:server,column=info:serverstartcode and column=info:regioninfo > for the missing region. > And column=info:regioninfo includes so many information. How to generate > them one by one? > As for the name of row, it consists of tablename, startkey, encode, and one > more long number, > how to get this number? > 3.use assing command in the hbase shell > > We also tried check_meta.rb --fix, it reports > ==== > 11/07/06 00:09:08 WARN check_meta: hole after REGION => {NAME => > 'STable,ztqdVD8fCMP-dDbXUAydankboD4=,1305619724446.c45191821053d03537596f4a2e759718.', > STARTKEY => 'ztqdVD8fCMP-dDbXUAydankboD4=', ENDKEY => > 'ztxrGmCwn-6BE32s3cX1TNeHU_I=', ENCODED => c45191821053d03537596f4a2e759718, > TABLE => {{NAME => 'STable', FAMILIES => [{NAME => 'file', BLOOMFILTER => > 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '3', > TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', > BLOCKCACHE => 'true'}, {NAME => 'filelength', BLOOMFILTER => 'NONE', > REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '3', TTL => ' > 2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => > 'true'}, {NAME => 'userbucket', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => > '0', COMPRESSION => 'NONE', VERSIONS => '3', TTL => '2147483647', > BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => > 'userpass', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => > 'NONE', VERSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536', > IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}} > 11/07/06 00:28:40 WARN check_meta: Missing .regioninfo: hdfs:// > hd0013.c.gj.com:9000/hbase/STable/3e6faca40a7ccad7ed8c0b5848c0f945/.regioninfo > ==== > > The problem is still there. BTW, what about the blue warning? Is this a > serious issue? > The situation is quite hard to us, it looks like even we can fill the hole > in the meta, we would lost all the data in the hole region, right? > > Thanks and regards, > > Mao Xu-Feng >