On Sun, Nov 27, 2016 at 6:53 PM, Timothy Brown <t...@siftscience.com> wrote:
> Hi Everyone, > > I apologize for starting an additional thread about this but I wasn't > subscribed to the users mailing list when I sent the original and can't > figure out how to respond to the original :( > > Original Message: > > We are seeing about 80% CPU utilization on the Region Server that solely > serves the meta table while other region servers typically have under 50% > CPU utilization. Is this expected? > > What is the difference when you compare servers? More requests? More i/o? Thread dump the metadata server and let us see a link in here? (What you attached below is cut-off... just as it is getting to the good part). > Here's some more info about our cluster: > HBase version 1.2 > Which 1.2? > Number of regions: 72 > Number of tables: 97 > On whole cluster? (Can't have more tables than regions...) > Approx. requests per second to meta region server: 3k > Can you see who is hitting he meta region most? (Enable rpc-level TRACE logging on the server hosting meta for a minute or so and see where the requests are coming in from). What is your cache hit rate? Can you get it higher? Is there much writing going on against meta? Or is cluster stable regards region movement/creation? > Approx. requests per second to entire HBase cluster: 90k > > Additional info: > > > From Storefile Metrics: > Stores Num: 1 > Storefiles: 1 > Storefile Size: 30m > Uncompressed Storefile Size: 30m > Index Size: 459k > > This from meta table? That is very small. > > I/O for the region server with only meta on it: > 48M bytes in > Whats all the writing about? > 5.9B bytes out > > This is disk or network? If network, is that 5.9 bytes? Thanks Tim, S > I used the debug dump on the region server's UI but it was too large > for paste bin so here's a portion of it: http://pastebin.com/nkYhEceE > > > Thanks for the help, > > Tim >
