HBase 0.96.1.1-cdh5.0.0 is the hbase version
I enabled DEBUG and I do not see anything new mentioned above. I think
it must still have references since this return:
return cleanMergeRegion(region, mergeRegions.getFirst(),
mergeRegions.getSecond());
on CatalogJanitor must be returning false in order for me to get the
"merge qualifier" message I placed in my first email. It must still
have references, I just don't know if it should since the merge was
already successful. How can I force a "de-reference"?
I can see if running the merge repeatedly in a timed loop makes slow
progress or if the reference is kept.
On Sat, Jun 28, 2014 at 10:48 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> Which release of hbase are you using ?
>
> Did you turn on DEBUG logging ?
> Do you see the following log (from CatalogJanitor) in master log ?
>
> if (mergeRegions.getFirst() == null || mergeRegions.getSecond() ==
> null)
> {
>
> LOG.error("Merged region " + region.getRegionNameAsString()
>
> + " has only one merge qualifier in META.");
>
> return false;
>
> }
>
> One possibility is that region 190d50047a820d9b3ef588429c9065ea still has
> reference(s)
>
> From CatalogJanitor#cleanMergeQualifier():
>
> if (regionFs == null || !regionFs.hasReferences(htd)) {
>
>
> Cheers
>
>
> On Sat, Jun 28, 2014 at 7:12 AM, Geovanie Marquez <
> geovanie.marq...@gmail.com> wrote:
>
> > I have a table that blew up in the number of regions, I fixed the
> > configuration and now need to bring the regions back down to our desired
> > number of 16. I wrote a program to do exactly that and it worked when
> > tested on a single node cluster, but now in production I get perpetually
> > stuck on a merge with the log statement in the HMaster log below.
> >
> > Typically this is the sequence of events:
> > Table has the following -
> > RegionA
> > RegionB
> > RegionC
> >
> > RegionMergedA = merge(RegionA, RegionB)
> >
> > RegionMergeB = merge(RegionMergedA, RegionC)
> > Then the logs just get stuck repeating the following, where RegionMergeB
> =
> > 190d50047a820d9b3ef588429c9065ea
> >
> > The perpetual statement on the master node. I waited 40 minutes to see if
> > it cleared to no avail.
> > 9:50:58.059 AM INFO
> > org.apache.hadoop.hbase.master.handler.DispatchMergingRegionHandler
> > Skip merging regions
> >
> registry.medical.records,,1403961571625.190d50047a820d9b3ef588429c9065ea.,
> >
> >
> registry.medical.records,\x0D\xFC\xE0\xD3\xFA\xF5FJ\x9D\xC6F\xD3\xE19*\xEAWLAB115\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x16%\xB6,1400656218257.11a7c474dd162ae4d7fc4a5573dda858.,
> > because region 190d50047a820d9b3ef588429c9065ea has merge qualifier
> >
> >
> > The logs on the regionserver show a successful merge.
> > Regions merged, hbase:meta updated, and report to master.
> >
> >
> region_a=registry.medical.records,,1403959070153.fa5121800ef7841875a153b3fef48d95.,
> >
> >
> region_b=registry.medical.records,\x0C\xEE\xA7\x0CbNE\xA3\xAA\xC1g\x05\x86\xFA\xF3\xFEWLAB169\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x94_(,1400656255740.e750ecca47b5dd64060d7bc40e0c0a57.,merged
> >
> >
> region=registry.medical.records,,1403961571625.190d50047a820d9b3ef588429c9065ea..
> > Region merge took 0sec
> >
> > How do I clear the merge qualifier on the region that has been merged?
> > I tried:
> > *admin.runCatalogScan();
> > *table.clearRegionCache();
> >
> > This also happens when I try to merge on the shell:
> > merge_region '190d50047a820d9b3ef588429c9065ea',
> > '11a7c474dd162ae4d7fc4a5573dda858'
> >
>
