Jayesh: Is it possible for you to share the JVM parameters ? Thanks
On Fri, Apr 22, 2016 at 7:48 AM, Thakrar, Jayesh < jthak...@conversantmedia.com> wrote: > Karthik, > > Yes, tuning can help - but the biggest help is to give "sufficient" memory > to the regionserver. > And "sufficient" is relative - e.g. we have a 75GB heap (our increases > were like this - 8, 12, 16, 24, 32, 45, 60 and 75 GB) > Note that we had started off with 45 GB RAM and are now running on 148 GB > RAM servers. > > Now we are very stable at 75GB and with appropriate JVM tuning our GCs are > well contained (sub-100 ms). > > Also, although I like G1, it did not fare well for us at 75 GB - with the > best tuning we could get upto 20-25 second pauses (HBase 1.0). > CMS seems to work best for us. > > Jayesh > > -----Original Message----- > From: karthi keyan [mailto:karthi93.san...@gmail.com] > Sent: Friday, April 22, 2016 7:04 AM > To: user@hbase.apache.org; dingdongc...@baidu.com > Subject: Re: Region server crashes after GC pause time > > Hi Ding, > > I have increased the Heap to 2G , still getting out of memory exception . > Actually i had write the data to HBase at 40K writes/sec . > Is there any parameter to tune up , as my knowledge "- > XX:CMSInitiatingOccupancyFraction=N " i tuned like this in HBase. > Is there any other parameter required to resolve this issue??? > > Best, > Karthik > > On Thu, Apr 14, 2016 at 12:21 PM, Ding,Dongchao <dingdongc...@baidu.com> > wrote: > > > Dump the jvm heap，analysis the the heap and find which query(s) cost > > so many memory？ > > In my ever bad case，the RS crashed for Long GC pauses because of a big > > query on Batch Get operation. > > > > > > In addition，I think you can increase the men of JVM, 512m is so small > > for RS. > > > > > > > > 在 16/4/14 14:00， "karthi keyan" <karthi93.san...@gmail.com> 写入: > > > > >Hi , > > > > > >i got this issue in HBase while at peak time handling more requests . > > >can any one pls guide me to resolve the Long GC pauses in hbase . > > > > > >JDK-7 , JVM heap 512m > > > > > >HBase 0.98.13 > > > > > > > > > INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM > > >or host machine (eg GC): pause of approximately 1466ms GC pool > > >'ConcurrentMarkSweep' had collection(s): count=1 time=1967ms INFO > > >[JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host > > >machine (eg GC): pause of approximately 2304ms GC pool > > >'ConcurrentMarkSweep' had collection(s): count=1 time=2775ms INFO > > >[JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host > > >machine (eg GC): pause of approximately 2287ms GC pool > > >'ConcurrentMarkSweep' had collection(s): count=1 time=2775ms > > > > > > INFO [RS:0;0:0:0:0:0:0:0:0:44037-SendThread(<host>:2181)] > > >zookeeper.ClientCnxn: Client session timed out, have not heard from > > >server in 6819ms for sessionid 0x1540ab48b280004, closing socket > > >connection and attempting reconnect INFO > > >[SplitLogWorker-<host>,44037,1460468489645-SendThread(<host>:2181)] > > >zookeeper.ClientCnxn: Client session timed out, have not heard from > > >server in 6819ms for sessionid 0x1540ab48b280005, closing socket > > >connection and attempting reconnect > > > > > >Once after this HBase Region Server moved to Dead state. > > > > > >Best, > > >Karthik > > > > > > > > > This email and any files included with it may contain privileged, > proprietary and/or confidential information that is for the sole use > of the intended recipient(s). Any disclosure, copying, distribution, > posting, or use of the information contained in or attached to this > email is prohibited unless permitted by the sender. If you have > received this email in error, please immediately notify the sender > via return email, telephone, or fax and destroy this original transmission > and its included files without reading or saving it in any manner. > Thank you. >
