On Tue, Aug 14, 2012 at 3:42 PM, David Koch <ogd...@googlemail.com> wrote:
> Hello,
>
> It's a fully-distributed environment (CDH3). Hbase hbck sometimes reports
> inconsistencies like:
>
> ERROR: Region { meta =>
> _myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27.,
> hdfs => null, deployed =>
> stag-back-9.ezakus.net:60020;_myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27.
> } found in META, but not in HDFS, and deployed on
> <server_name_replaced>:60020
>
> However, in the end it always tells me the table is ok.
During normal operation, transitory conditions can look like problems to hbck.
> Only for _myTable
> does the master log contain the warning messages mentioned in my original
> post, one every 5 minutes:
>
The CatalogJanitor runs every 5 minutes. Its the thing that does
checks and janitorial tasks on the .META. table.
> 2012-08-14 23:53:53,167 WARN org.apache.hadoop.hbase.master.CatalogJanitor:
> REGIONINFO_QUALIFIER is empty in keyvalues={_myTable
> ,,1344874729874.61499712e41143df595f2215d1ab7b91./info:server/1344876299527/Put/vlen=29,
> _myTable
> ,,1344874729874.61499712e41143df595f2215d1ab7b91./info:serverstartcode/1344876299527/Put/vlen=8}
>
> The region server log where _myTable is hosted contains the following:
>
> 2012-08-14 11:36:31,801 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to
> open region: _myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27.
> 2012-08-14 11:36:31,832 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Onlined _myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27.; next
> sequenceid=1
> 2012-08-14 11:36:31,860 INFO org.apache.hadoop.hbase.catalog.MetaEditor:
> Updated row _myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27. in
> region .META.,,1 with server=<server_name_replaced>:60020,
> startcode=1344877239935
>
> I remember disabling the table from base shell with the plan to drop it
> later - I may then have launched the application which (re-)creates
> _myTable if it does not exist without ever having issued the drop before.
> Not sure if that could be part of the issue.
>
Yes. Could be. You could delete the rows in .META. that mention your
__myTable (Be careful you only delete this row). That should fix the
master log message.
> The data can be re-generated quickly. If it gets rid of the warning I can
> just disable and drop the table. Would I need to carry out any other
> maintenance tasks like editing .META before re-creating a "clean" version
> of this table?
>
Just do the above delete. That should take care of it.
Find the row name in .META. that mentions your __myTable (is it
"_myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27."?)
For example:
hbase> deleteall ".META.",
"_myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27."
St.Ack
