Thanks. Once again, a newbie in this regard but, Do I build hbck2-tools on similar OS & java version? I have been informed we have 'maven' installed on one host on a cluster, which is centos and (probably) a different java, and the build itself is needed for Ubuntu16.
-----Original Message----- From: Wellington Chevreuil <[email protected]> Sent: Wednesday, December 16, 2020 10:08 AM To: Hbase-User <[email protected]> Subject: Re: Removal of table rows. EXTERNAL > > Hbase:meta has info for one table region which is NOT on a master. Is > that correct? I would have expected all meta info to be stored on a master. > meta table is an "hbase system table" that has info about which regions are assigned to which region servers in your cluster. As any user table, it also has a region, and this region must be hosted on a region server in the cluster. Master is just responsible for the coordination of regions and some other housekeeping actions. As I state below, a simple method to remove everything connected to > alfa:rfilenameext would seem the easiest of tasks as the > namespace:name consistently appears on every relevant row. Looks like hbck2 *extraRegionsinMeta *is what you need here, and it should work with hbase 2.1. It's not available on hbck2 1.0 release, but you can build a new hbck2 jar out of current master branch, and that would give you an hbck2 with extraRegionsinMeta. You can do it by cloning the repo below, then do a *mvn install* from the main module: https://github.com/apache/hbase-operator-tools.git Em qua., 16 de dez. de 2020 às 08:02, Marc Hoppins <[email protected]> escreveu: > Hbase is 2.1.0 (via Cloudera CDH 6.3.2) > > The UI page for the master shows NO report links/tabs. > > Hbase:meta has info for one table region which is NOT on a master. Is > that correct? I would have expected all meta info to be stored on a master. > > Name Region Server > Read Requests Write Requests > Num.Storefiles MemSize Locality > hbase:meta,,1.1588230740 ba-wtmp05.asgardalfa.hq.com:16030 > 219,168,603 35,419 1 MB 4 0 B > 1.0 > > -----Original Message----- > From: Wellington Chevreuil <[email protected]> > Sent: Tuesday, December 15, 2020 7:10 PM > To: Hbase-User <[email protected]> > Subject: Re: Removal of table rows. > > EXTERNAL > > > > > I am an HBASE newbie so I apologise if I am being repetitious. > > > > Apologies also if this is not the right group. Am not sure if this > > may be more suited to 'dev' list. > > > Welcome, this is the right channel for this kind of questions. > > > > The solution offered by hbase-operator-tools - extraRegionsinMeta - > > offered hope. Once again, however, another problem has surfaced: > > this tools command for extra regions is incompatible with the hbase > > version we are running. > > > This command does not rely on any Master/RegionServer interface, so > it should not have any incompatibility issues. It only uses public > client API to cleanup meta table, so maybe you just need to build > latest hbck2 master branch version? It would work for any hbase 2, in > theory, it could even work with hbase 1, but that was never tested. > Luckly, you got Stack's attention, so yeah, if you confirm on his > previous questions we might be able to help with further directions. > > Em ter., 15 de dez. de 2020 às 17:23, Stack <[email protected]> escreveu: > > > On Tue, Dec 15, 2020 at 7:46 AM Marc Hoppins <[email protected]> > wrote: > > > > > Hi all, > > > > > > I am an HBASE newbie so I apologise if I am being repetitious. > > > > > > Apologies also if this is not the right group. Am not sure if this > > > may be more suited to 'dev' list. However, > > > > > > A problem question and a technical/wishlist question. > > > > > > Problem: > > > > > > I have inherited a problem with an HBASE table. The original issue > > > may have erupted due to a network outage. A table has 48 region > > > in > > transition > > > operations, stuck that way for several weeks. A previous attempt > > > to fix things with hbck failed. An attempt to DISABLE then DROP > > > the table also failed. The four or five attempts to work the table > > > ALSO now had stuck procedures. Subsequent DFS and ZOO operations > > > left the situation where there was no data and no real table: just > > > a 6K file in an empty > > structure. > > > > > > > > Which version of hbase, do you know Marc? (Look at the base of the > > master UI. It'll tell you. Sounds like it is an hbase-2.x). > > > > What is the name of the 6k file? (I am trying to understand what the > > file you are referencing is). > > > > The table directory was removed from hdfs and zookeeper? > > > Yes. The data had already been removed from the previous tech attempts > to fix the issue. > > Thus, the HDFS 6K file structure contained no actual information, just > empty files and directories. Zkshell had been used to remove znodes. > > > > > > > When I got to the problem my knowledge of HBASE was nil. It is > > > little better than that now but anyway... > > > > > > Fortunately for me this is a testing/dev cluster. The 'owner' was > > > content that the table can be removed - and appeared to have > > > already been > > done...of > > > a kind. > > > > > > Reading and reading of others' similar issues lead me to the point > > > I also was going to clean the HDFS data and ZK data for this > > > table. I shut down HBASE, cleaned HDFS and ZK node data, deleted > > > the masterprocwals and restarted HBASE. > > > > > > When all came up I was happy to see that the affected table > > > appeared nowhere and that the procedures had all disappeared. > > > > > > However, when I hopped to hbase master, even though no table of > > > that name existed, 48 regions were still in transition. Further > > > research steered > > me > > > toward hbase:meta and sure enough, the references to the RITs > > > lived > > happily > > > among other data for other tables. > > > > > > > > Do you have a 'Procedures and Locks' tab on your master home page? > > If so, what does it report? Is there an "HBCK Report" tab? If so, > > what does it show? > > > > When I got to the problem procedures & locks showed one item for each > problem on that table. After removal of masterprocwals they all > disappeared. The only entry now is one normal-looking WAL log file > entry whose timestamp changes according to operation time (currently > 8.25am today). > > > > > > The solution offered by hbase-operator-tools - extraRegionsinMeta > > > - offered hope. Once again, however, another problem has surfaced: > > > this tools command for extra regions is incompatible with the > > > hbase version we are running. > > > > > > So... > > > > > > How can I remove the references to namespace:kaput_table from > hbase:meta? > > > > > > > > > Sounds like an hbase-2.1.x or a 2.0.x. > > > > Will wait on your answers to the above... It might be a crass delete > > of each row from the hbase:meta table then restart (even then, if > > procedures in the procedure store, you may have to clear it again as > > you did above before the restart to purge the procedures as you > > don't have tooling to do it from cmdline... do you have an 'hbck2 > > bypass > --override'?). > > > A crass delete of each row would work for me if I could Identify how > to remove the rows. Sample data:- > > alfa:rfilenameext column=table:state, timestamp=1604493139455, > value=\x08\x00 > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167. > column=info:regioninfo, timestamp=1604388258225, value={ENCODED => > 35925292c25898671e5a894ce387e167, NAME => > 'alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.', > STARTKEY => '', ENDKEY => '0'} > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167. > column=info:seqnumDuringOpen, timestamp=1601269814633, > value=\x00\x00\x00\x00\x00\x00\x008 > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167. > column=info:server, timestamp=1601269814633, value= > ba-wtmp04.asgardalfa.hq.com:16020 > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167. > column=info:serverstartcode, timestamp=1601269814633, > value=1601061167123 > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167. > column=info:sn, timestamp=1604388258050, > value=ba-wtmp04.asgardalfa.hq.com > ,16020,1601061167123 > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167. > column=info:state, timestamp=1604388258225, value=CLOSED > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2. > column=info:regioninfo, timestamp=1600969938610, value={ENCODED => > 787d1455b84f2d846ce6089392f01fd2, NAME => > 'alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.', > STARTKEY => '0', ENDKEY => '1'} > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2. > column=info:seqnumDuringOpen, timestamp=1600780187722, > value=\x00\x00\x00\x00\x00\x02^\xBB > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2. > column=info:server, timestamp=1600780187722, value= > ba-wtmp08.asgardalfa.hq.com:16020 > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2. > column=info:serverstartcode, timestamp=1600780187722, > value=1600780162556 > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2. > column=info:sn, timestamp=1600969938610, > value=ba-wtmp07.asgardalfa.hq.com > ,16020,1600936054386 > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2. > column=info:state, timestamp=1600969938610, value=OPENING > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e. > column=info:regioninfo, timestamp=1601060563980, value={ENCODED => > aa9d89b40a9def31a080fdd1776acb4e, NAME => > 'alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.', > STARTKEY => '1', ENDKEY => '2'} > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e. > column=info:seqnumDuringOpen, timestamp=1600780186976, > value=\x00\x00\x00\x00\x00\x02^\xA9 > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e. > column=info:server, timestamp=1600780186976, value= > dr1-wtmp02.asgardalfa.hq.com:16020 > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e. > column=info:serverstartcode, timestamp=1600780186976, > value=1600780163021 > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e. > column=info:sn, timestamp=1601060563980, > value=ba-wtmp05.asgardalfa.hq.com > ,16020,1601049145467 > > > As I state below, a simple method to remove everything connected to > alfa:rfilenameext would seem the easiest of tasks as the > namespace:name consistently appears on every relevant row. > > > > > > > > Technical: > > > > > > Is there to be any implementation of such a fix within HBASe > > > itself where table manipulation can be performed by Eg., > > > > > > delete hbase:meta namespace > > > delete hbase:meta namespace:table > > > > > > or even > > > > > > scan hbase:meta filter = namespace:table | deleterow > > > > > > or some such? > > > > > > > > Scan doesn't return result (unfortunately) so you can assign to a > > shell variable; it dumps scan output2 on stdout/stderr. > > > > I'm sure there is a better way but something like below (don't laugh! > > and I've not really tried it so be careful): > > > > # Get rows for the BAD_TABLENAME from hbase:meta table $ echo 'scan > > "hbase:meta", {ROWPREFIXFILTER => "BAD_TABLENAME"}'| hbase shell > > > /tmp/out.txt # Need to get the row only. Above got rows and columns. > > We don't want to filter on column # because we do not know which > > columns are still in hbase:meta. Need to filter on table # again to > > cut the shell preamble and footer out. Unique the rows. > > $ cat /tmp/out.txt |awk '{print $1}'|grep BAD_TABLENAME | sort -u > > > /tmp/unique_rows.txt # Now you have rows to delete. For each in > > shell do... 'deleteall "ROW_X"'... Could # edit /tmp/unique_rows.txt > > and per for add .... deleteall " prefix and a " > > suffix.... to make > > # a file with lines of deleteall "ROW1", etc. then pass it to the shell: > > hbase shell /tmp/delete_rows.txt. > > > > But before messing w/ the above, lets get the answers to above questions. > > Thanks, > > S > > > > > > > > > > > > > > > > > > > Thanks > > > > > > M > > > > > >
