Hbase is 2.1.0 (via Cloudera CDH 6.3.2)
The UI page for the master shows NO report links/tabs.
Hbase:meta has info for one table region which is NOT on a master. Is that
correct? I would have expected all meta info to be stored on a master.
Name Region Server
Read Requests Write Requests Num.Storefiles
MemSize Locality
hbase:meta,,1.1588230740 ba-wtmp05.asgardalfa.hq.com:16030
219,168,603 35,419 1 MB 4 0 B
1.0
-----Original Message-----
From: Wellington Chevreuil <[email protected]>
Sent: Tuesday, December 15, 2020 7:10 PM
To: Hbase-User <[email protected]>
Subject: Re: Removal of table rows.
EXTERNAL
>
> I am an HBASE newbie so I apologise if I am being repetitious.
>
> Apologies also if this is not the right group. Am not sure if this may
> be more suited to 'dev' list.
>
Welcome, this is the right channel for this kind of questions.
> The solution offered by hbase-operator-tools - extraRegionsinMeta -
> offered hope. Once again, however, another problem has surfaced: this
> tools command for extra regions is incompatible with the hbase version
> we are running.
>
This command does not rely on any Master/RegionServer interface, so it should
not have any incompatibility issues. It only uses public client API to cleanup
meta table, so maybe you just need to build latest hbck2 master branch version?
It would work for any hbase 2, in theory, it could even work with hbase 1, but
that was never tested. Luckly, you got Stack's attention, so yeah, if you
confirm on his previous questions we might be able to help with further
directions.
Em ter., 15 de dez. de 2020 às 17:23, Stack <[email protected]> escreveu:
> On Tue, Dec 15, 2020 at 7:46 AM Marc Hoppins <[email protected]> wrote:
>
> > Hi all,
> >
> > I am an HBASE newbie so I apologise if I am being repetitious.
> >
> > Apologies also if this is not the right group. Am not sure if this
> > may be more suited to 'dev' list. However,
> >
> > A problem question and a technical/wishlist question.
> >
> > Problem:
> >
> > I have inherited a problem with an HBASE table. The original issue
> > may have erupted due to a network outage. A table has 48 region in
> transition
> > operations, stuck that way for several weeks. A previous attempt to
> > fix things with hbck failed. An attempt to DISABLE then DROP the
> > table also failed. The four or five attempts to work the table ALSO
> > now had stuck procedures. Subsequent DFS and ZOO operations left
> > the situation where there was no data and no real table: just a 6K
> > file in an empty
> structure.
> >
> >
> Which version of hbase, do you know Marc? (Look at the base of the
> master UI. It'll tell you. Sounds like it is an hbase-2.x).
>
> What is the name of the 6k file? (I am trying to understand what the
> file you are referencing is).
>
> The table directory was removed from hdfs and zookeeper?
>
Yes. The data had already been removed from the previous tech attempts to fix
the issue.
Thus, the HDFS 6K file structure contained no actual information, just empty
files and directories. Zkshell had been used to remove znodes.
>
> > When I got to the problem my knowledge of HBASE was nil. It is
> > little better than that now but anyway...
> >
> > Fortunately for me this is a testing/dev cluster. The 'owner' was
> > content that the table can be removed - and appeared to have already
> > been
> done...of
> > a kind.
> >
> > Reading and reading of others' similar issues lead me to the point I
> > also was going to clean the HDFS data and ZK data for this table. I
> > shut down HBASE, cleaned HDFS and ZK node data, deleted the
> > masterprocwals and restarted HBASE.
> >
> > When all came up I was happy to see that the affected table appeared
> > nowhere and that the procedures had all disappeared.
> >
> > However, when I hopped to hbase master, even though no table of that
> > name existed, 48 regions were still in transition. Further research
> > steered
> me
> > toward hbase:meta and sure enough, the references to the RITs lived
> happily
> > among other data for other tables.
> >
> >
> Do you have a 'Procedures and Locks' tab on your master home page? If
> so, what does it report? Is there an "HBCK Report" tab? If so, what
> does it show?
>
When I got to the problem procedures & locks showed one item for each problem
on that table. After removal of masterprocwals they all disappeared. The only
entry now is one normal-looking WAL log file entry whose timestamp changes
according to operation time (currently 8.25am today).
>
> > The solution offered by hbase-operator-tools - extraRegionsinMeta -
> > offered hope. Once again, however, another problem has surfaced:
> > this tools command for extra regions is incompatible with the hbase
> > version we are running.
> >
> > So...
> >
> > How can I remove the references to namespace:kaput_table from hbase:meta?
> >
> >
> > Sounds like an hbase-2.1.x or a 2.0.x.
>
> Will wait on your answers to the above... It might be a crass delete
> of each row from the hbase:meta table then restart (even then, if
> procedures in the procedure store, you may have to clear it again as
> you did above before the restart to purge the procedures as you don't
> have tooling to do it from cmdline... do you have an 'hbck2 bypass
> --override'?).
>
A crass delete of each row would work for me if I could Identify how to remove
the rows. Sample data:-
alfa:rfilenameext column=table:state, timestamp=1604493139455, value=\x08\x00
alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
column=info:regioninfo, timestamp=1604388258225, value={ENCODED =>
35925292c25898671e5a894ce387e167, NAME =>
'alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.', STARTKEY
=> '', ENDKEY => '0'}
alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
column=info:seqnumDuringOpen, timestamp=1601269814633,
value=\x00\x00\x00\x00\x00\x00\x008
alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
column=info:server, timestamp=1601269814633,
value=ba-wtmp04.asgardalfa.hq.com:16020
alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
column=info:serverstartcode, timestamp=1601269814633, value=1601061167123
alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
column=info:sn, timestamp=1604388258050,
value=ba-wtmp04.asgardalfa.hq.com,16020,1601061167123
alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
column=info:state, timestamp=1604388258225, value=CLOSED
alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
column=info:regioninfo, timestamp=1600969938610, value={ENCODED =>
787d1455b84f2d846ce6089392f01fd2, NAME =>
'alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.', STARTKEY
=> '0', ENDKEY => '1'}
alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
column=info:seqnumDuringOpen, timestamp=1600780187722,
value=\x00\x00\x00\x00\x00\x02^\xBB
alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
column=info:server, timestamp=1600780187722,
value=ba-wtmp08.asgardalfa.hq.com:16020
alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
column=info:serverstartcode, timestamp=1600780187722, value=1600780162556
alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
column=info:sn, timestamp=1600969938610,
value=ba-wtmp07.asgardalfa.hq.com,16020,1600936054386
alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
column=info:state, timestamp=1600969938610, value=OPENING
alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
column=info:regioninfo, timestamp=1601060563980, value={ENCODED =>
aa9d89b40a9def31a080fdd1776acb4e, NAME =>
'alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.', STARTKEY
=> '1', ENDKEY => '2'}
alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
column=info:seqnumDuringOpen, timestamp=1600780186976,
value=\x00\x00\x00\x00\x00\x02^\xA9
alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
column=info:server, timestamp=1600780186976,
value=dr1-wtmp02.asgardalfa.hq.com:16020
alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
column=info:serverstartcode, timestamp=1600780186976, value=1600780163021
alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
column=info:sn, timestamp=1601060563980,
value=ba-wtmp05.asgardalfa.hq.com,16020,1601049145467
As I state below, a simple method to remove everything connected to
alfa:rfilenameext would seem the easiest of tasks as the namespace:name
consistently appears on every relevant row.
>
>
> > Technical:
> >
> > Is there to be any implementation of such a fix within HBASe itself
> > where table manipulation can be performed by Eg.,
> >
> > delete hbase:meta namespace
> > delete hbase:meta namespace:table
> >
> > or even
> >
> > scan hbase:meta filter = namespace:table | deleterow
> >
> > or some such?
> >
> >
> Scan doesn't return result (unfortunately) so you can assign to a
> shell variable; it dumps scan output2 on stdout/stderr.
>
> I'm sure there is a better way but something like below (don't laugh!
> and I've not really tried it so be careful):
>
> # Get rows for the BAD_TABLENAME from hbase:meta table $ echo 'scan
> "hbase:meta", {ROWPREFIXFILTER => "BAD_TABLENAME"}'| hbase shell >
> /tmp/out.txt # Need to get the row only. Above got rows and columns.
> We don't want to filter on column # because we do not know which
> columns are still in hbase:meta. Need to filter on table # again to
> cut the shell preamble and footer out. Unique the rows.
> $ cat /tmp/out.txt |awk '{print $1}'|grep BAD_TABLENAME | sort -u >
> /tmp/unique_rows.txt # Now you have rows to delete. For each in shell
> do... 'deleteall "ROW_X"'... Could # edit /tmp/unique_rows.txt and per
> for add .... deleteall " prefix and a "
> suffix.... to make
> # a file with lines of deleteall "ROW1", etc. then pass it to the shell:
> hbase shell /tmp/delete_rows.txt.
>
> But before messing w/ the above, lets get the answers to above questions.
> Thanks,
> S
>
>
>
>
>
>
>
>
> > Thanks
> >
> > M
> >
>
