Responses inlined. On Mon, Nov 28, 2016 at 12:45 PM, Stack <st...@duboce.net> wrote:
> On Sun, Nov 27, 2016 at 6:53 PM, Timothy Brown <t...@siftscience.com> > wrote: > > > Hi Everyone, > > > > I apologize for starting an additional thread about this but I wasn't > > subscribed to the users mailing list when I sent the original and can't > > figure out how to respond to the original :( > > > > Original Message: > > > > We are seeing about 80% CPU utilization on the Region Server that solely > > serves the meta table while other region servers typically have under 50% > > CPU utilization. Is this expected? > > > > What is the difference when you compare servers? More requests? More i/o? > Thread dump the metadata server and let us see a link in here? (What you > attached below is cut-off... just as it is getting to the good part). > > > There are more requests to the server containing meta. The network in bytes are greater for the meta regionserver than the others but the network out bytes are less. Here's a dropbox link to the output https://dl.dropboxusercontent.com/u/ 54494127/thread_dump.txt. I apologize for the cliffhanger. > > > Here's some more info about our cluster: > > HBase version 1.2 > > > > Which 1.2? > > 1.2.0 which is bundled with CDH 5.8.0 > > > > Number of regions: 72 > > Number of tables: 97 > > > > On whole cluster? (Can't have more tables than regions...) > > > An error on my part, I meant to put 72 region servers. > > > Approx. requests per second to meta region server: 3k > > > > Can you see who is hitting he meta region most? (Enable rpc-level TRACE > logging on the server hosting meta for a minute or so and see where the > requests are coming in from). > > What is your cache hit rate? Can you get it higher? > > Cache hit rate is above 99%. We see very little disk reads. > Is there much writing going on against meta? Or is cluster stable regards > region movement/creation? > > Writing is very infrequent. The cluster is stable with regards to region movement and creation. > > > > Approx. requests per second to entire HBase cluster: 90k > > > > Additional info: > > > > > > From Storefile Metrics: > > Stores Num: 1 > > Storefiles: 1 > > Storefile Size: 30m > > Uncompressed Storefile Size: 30m > > Index Size: 459k > > > > > This from meta table? That is very small. > > Yes this is from the meta table. > > > > > I/O for the region server with only meta on it: > > 48M bytes in > > > > > Whats all the writing about? > > I'm not sure. According to the AWS dashboard there are no disk writes at that time. > > > > 5.9B bytes out > > > > > This is disk or network? If network, is that 5.9 bytes? > > This is network and thats 5.9 billion byes. (I'm using the AWS dashboard for this) > Thanks Tim, > S > > > > > I used the debug dump on the region server's UI but it was too large > > for paste bin so here's a portion of it: http://pastebin.com/nkYhEceE > > > > > > Thanks for the help, > > > > Tim > > >
