Sorry for the confusion. I should have used the same place to grab the numbers from.
In my original email, I grabbed the reqs/sec metric from the Hbase UI directly. It was showing 1000 Reqs/sec. Right now, when I look at that, the number is 891 Reqs/sec, but it fluctuates. The 55K Reqs/sec in my last email, I got from Cloudera Manager. I am not sure why it is reporting a different number. It was 55K Reqs/sec, but now it jumped back up to 60K Reqs/sec. On Sun, Jan 25, 2015 at 9:37 PM, Ted Yu <yuzhih...@gmail.com> wrote: > From your first email: > > bq. 1000reqs/sec vs 4-5 reqs/sec elsewhere > > Now the request count becomes 55K requests/sec ? > Just want to clarify whether the request count increased or decreased when > compaction is not in progress. > > Cheers > > On Sun, Jan 25, 2015 at 6:23 PM, Pun Intended <punintende...@gmail.com> > wrote: > > > Thanks! > > I ran hbase hbck and all seems healthy. > > > > 0 inconsistencies detected. > > Status: OK > > > > Also the meta region shows to be 100% compacted, but I still see 55K > > requests/sec to that RegionServer: > > > > Region > > Name Num. Compacting KVs Num. > > Compacted KVs Compaction Progress > > > > hbase:meta,,1.1588230740 > > 195349 195349 100.00% > > > > best > > > > > > On Sun, Jan 25, 2015 at 8:28 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > > > After the adjustment, hbase:meta would still be accessed for region > > > locations. But the request count should come down noticeably. > > > > > > bq. Is there a way to inspect whether I have corruptions in the META > > table? > > > > > > Please take a look at: > > > http://hbase.apache.org/book.html#hbck.in.depth > > > > > > Cheers > > > > > > On Sun, Jan 25, 2015 at 5:20 PM, Pun Intended <punintende...@gmail.com > > > > > wrote: > > > > > > > Will do! Thanks you for your suggestion. I will keep an eye on the > read > > > > rate to that RegionServer, but in case it doesn't drop, do you think > > > there > > > > should be something else I need to look at? > > > > > > > > * Is there a way to inspect whether I have corruptions in the META > > table? > > > > * Any other possible misconfiguration that comes to mind that I > should > > > keep > > > > an eye for? > > > > > > > > best > > > > > > > > On Sun, Jan 25, 2015 at 8:05 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > > > > > > > Suggest you lower the frequency of compacting hbase:meta > > > > > > > > > > Move the second command to another script. > > > > > > > > > > Cheers > > > > > > > > > > > > > > > > > > > > > On Jan 25, 2015, at 4:46 PM, Pun Intended < > punintende...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > Hi Ted, > > > > > > > > > > > > I have a small script: major_compact_hbase.hbsh > > > > > > > > > > > > major_compact 'users' > > > > > > major_compact 'hbase:meta' > > > > > > exit > > > > > > > > > > > > > > > > > > and a cron calls it once a day with: > > > > > > /usr/bin/hbase shell major_compact_hbase.hbsh > > > > > > > > > > > > Thanks again! > > > > > > > > > > > > > > > > > >> On Sun, Jan 25, 2015 at 7:38 PM, Ted Yu <yuzhih...@gmail.com> > > > wrote: > > > > > >> > > > > > >> How do you trigger major compaction manually ? > > > > > >> > > > > > >> Cheers > > > > > >> > > > > > >> > > > > > >> > > > > > >>>> On Jan 25, 2015, at 3:40 PM, Pun Intended < > > > punintende...@gmail.com> > > > > > >>> wrote: > > > > > >>> > > > > > >>> Thank you so much for your help! > > > > > >>> You are right, the slow region server hosts hbase:meta!!! I am > > > using > > > > > >> Hbase > > > > > >>> 0.96.1.1 > > > > > >>> > > > > > >>> I have major compactions disabled. I run major compaction > > manually > > > > > once a > > > > > >>> day and when it dpes that besides requesting major compaction > on > > my > > > > > >> tables, > > > > > >>> it also requests a major_compact 'hbase:meta'. Is that what > could > > > be > > > > > >>> causing the issue? I did a scan on 'hbase:meta' and I see that > it > > > > > >> contains > > > > > >>> about the same number of regions as it should - I have two > tables > > > > with > > > > > >>> ~3000 regions each. > > > > > >>> > > > > > >>> Do you think I have some misconfiguration? Do I need to do some > > > > cleanup > > > > > >> on > > > > > >>> the meta region? Am I hitting some Hbase bug? Or am I being > dumb > > > > > >> requesting > > > > > >>> major compaction for the meta region daily? > > > > > >>> > > > > > >>> I believe on the client side I am doing things right - using > > > scanner > > > > > >>> caching, etc. > > > > > >>> > > > > > >>> Thank you so much for your help!! > > > > > >> > > > > > > > > > > > > > > >
