On Tue, Dec 15, 2020 at 7:46 AM Marc Hoppins <marc.hopp...@eset.sk> wrote:
> Hi all,
>
> I am an HBASE newbie so I apologise if I am being repetitious.
>
> Apologies also if this is not the right group. Am not sure if this may be
> more suited to 'dev' list. However,
>
> A problem question and a technical/wishlist question.
>
> Problem:
>
> I have inherited a problem with an HBASE table. The original issue may
> have erupted due to a network outage. A table has 48 region in transition
> operations, stuck that way for several weeks. A previous attempt to fix
> things with hbck failed. An attempt to DISABLE then DROP the table also
> failed. The four or five attempts to work the table ALSO now had stuck
> procedures. Subsequent DFS and ZOO operations left the situation where
> there was no data and no real table: just a 6K file in an empty structure.
>
>
Which version of hbase, do you know Marc? (Look at the base of the master
UI. It'll tell you. Sounds like it is an hbase-2.x).
What is the name of the 6k file? (I am trying to understand what the file
you are referencing is).
The table directory was removed from hdfs and zookeeper?
> When I got to the problem my knowledge of HBASE was nil. It is little
> better than that now but anyway...
>
> Fortunately for me this is a testing/dev cluster. The 'owner' was content
> that the table can be removed - and appeared to have already been done...of
> a kind.
>
> Reading and reading of others' similar issues lead me to the point I also
> was going to clean the HDFS data and ZK data for this table. I shut down
> HBASE, cleaned HDFS and ZK node data, deleted the masterprocwals and
> restarted HBASE.
>
> When all came up I was happy to see that the affected table appeared
> nowhere and that the procedures had all disappeared.
>
> However, when I hopped to hbase master, even though no table of that name
> existed, 48 regions were still in transition. Further research steered me
> toward hbase:meta and sure enough, the references to the RITs lived happily
> among other data for other tables.
>
>
Do you have a 'Procedures and Locks' tab on your master home page? If so,
what does it report? Is there an "HBCK Report" tab? If so, what does it
show?
> The solution offered by hbase-operator-tools - extraRegionsinMeta -
> offered hope. Once again, however, another problem has surfaced: this
> tools command for extra regions is incompatible with the hbase version we
> are running.
>
> So...
>
> How can I remove the references to namespace:kaput_table from hbase:meta?
>
>
> Sounds like an hbase-2.1.x or a 2.0.x.
Will wait on your answers to the above... It might be a crass delete of
each row from the hbase:meta table then restart (even then, if procedures
in the procedure store, you may have to clear it again as you did above
before the restart to purge the procedures as you don't have tooling to do
it from cmdline... do you have an 'hbck2 bypass --override'?).
> Technical:
>
> Is there to be any implementation of such a fix within HBASe itself where
> table manipulation can be performed by Eg.,
>
> delete hbase:meta namespace
> delete hbase:meta namespace:table
>
> or even
>
> scan hbase:meta filter = namespace:table | deleterow
>
> or some such?
>
>
Scan doesn't return result (unfortunately) so you can assign to a shell
variable; it dumps scan output2 on stdout/stderr.
I'm sure there is a better way but something like below (don't laugh! and
I've not really tried it so be careful):
# Get rows for the BAD_TABLENAME from hbase:meta table
$ echo 'scan "hbase:meta", {ROWPREFIXFILTER => "BAD_TABLENAME"}'| hbase
shell > /tmp/out.txt
# Need to get the row only. Above got rows and columns. We don't want to
filter on column
# because we do not know which columns are still in hbase:meta. Need to
filter on table
# again to cut the shell preamble and footer out. Unique the rows.
$ cat /tmp/out.txt |awk '{print $1}'|grep BAD_TABLENAME | sort -u >
/tmp/unique_rows.txt
# Now you have rows to delete. For each in shell do... 'deleteall
"ROW_X"'... Could
# edit /tmp/unique_rows.txt and per for add .... deleteall " prefix and a "
suffix.... to make
# a file with lines of deleteall "ROW1", etc. then pass it to the shell:
hbase shell /tmp/delete_rows.txt.
But before messing w/ the above, lets get the answers to above questions.
Thanks,
S
> Thanks
>
> M
>
