Hi Stack: Yes, we do hit full GC in the 8G heap. I see ... so, you are saying that it would just be linearly proportional ... so, I should expect a 3x pause increase with 24G _if_ the full GC hits. I agree with you ... with much more head room (and mslab etc enabled by default) we shouldn't normally hit a full GC.
I can surely try this out on a test cluster ... thanks for the pointer on the 3x increase if it hits full gc. That quantifies it for me much better. Thanks --Suraj On Mon, Jul 8, 2013 at 11:56 AM, Stack <st...@duboce.net> wrote: > On Mon, Jul 8, 2013 at 11:09 AM, Suraj Varma <svarma...@gmail.com> wrote: > > > Hello: > > We have an HBase cluster with region servers running on 8GB heap size > with > > a 0.6 block cache (it is a read heavy cluster, with bursty write traffic > > via MR jobs). (version: hbase-0.94.6.1) > > > > During HBaseCon, while speaking to a few attendees, I heard some folks > were > > running region servers as high as 24GB and some others in the 16GB range. > > > > So - question: Are there any special GC recommendations (tuning > parameters, > > flags, etc) that folks who run at these large heaps can recommend while > > moving up from an 8GB heap? i.e. for 16GB and for 24GB RS heaps ... ? > > > > I'm especially concerned about long pauses causing zk session timeouts > and > > consequent RS shutdowns. Our boxes do have a lot of RAM and we are > > exploring how we can use more of it for the cluster while maintaining > > overall stability. > > > > Also - if there are clusters running multiple region servers per host, > I'd > > be very interested to know what RS heap sizes those are being run at ... > > and whether this was chosen as an alternative to running a single RS with > > large heap. > > > > (I know I'll have to test the GC stuff out on my cluster and for my > > workloads anyway ... but just trying to get a feel of what sort of tuning > > options had to be used to have a stable HBase cluster with 16 or 24GB RS > > heaps). > > > > > You hit full GC in this 8G heap Suraj? Can you try running one server at > 24G to see how it does (with GC logging enabled so you can watch it over > time)? On one hand, more heap may make it so you avoid full GC -- if you > are hitting them now at 8G -- because application has more head room. On > other hand, yes, if a full GC hits, it will be gone for proportionally > longer than for your 8G heap. > > St.Ack >
