Karthik, Yes, tuning can help - but the biggest help is to give "sufficient" memory to the regionserver. And "sufficient" is relative - e.g. we have a 75GB heap (our increases were like this - 8, 12, 16, 24, 32, 45, 60 and 75 GB) Note that we had started off with 45 GB RAM and are now running on 148 GB RAM servers.
Now we are very stable at 75GB and with appropriate JVM tuning our GCs are well contained (sub-100 ms). Also, although I like G1, it did not fare well for us at 75 GB - with the best tuning we could get upto 20-25 second pauses (HBase 1.0). CMS seems to work best for us. Jayesh -----Original Message----- From: karthi keyan [mailto:karthi93.san...@gmail.com] Sent: Friday, April 22, 2016 7:04 AM To: user@hbase.apache.org; dingdongc...@baidu.com Subject: Re: Region server crashes after GC pause time Hi Ding, I have increased the Heap to 2G , still getting out of memory exception . Actually i had write the data to HBase at 40K writes/sec . Is there any parameter to tune up , as my knowledge "- XX:CMSInitiatingOccupancyFraction=N " i tuned like this in HBase. Is there any other parameter required to resolve this issue??? Best, Karthik On Thu, Apr 14, 2016 at 12:21 PM, Ding,Dongchao <dingdongc...@baidu.com> wrote: > Dump the jvm heap，analysis the the heap and find which query(s) cost > so many memory？ > In my ever bad case，the RS crashed for Long GC pauses because of a big > query on Batch Get operation. > > > In addition，I think you can increase the men of JVM, 512m is so small > for RS. > > > > 在 16/4/14 14:00， "karthi keyan" <karthi93.san...@gmail.com> 写入: > > >Hi , > > > >i got this issue in HBase while at peak time handling more requests . > >can any one pls guide me to resolve the Long GC pauses in hbase . > > > >JDK-7 , JVM heap 512m > > > >HBase 0.98.13 > > > > > > INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM > >or host machine (eg GC): pause of approximately 1466ms GC pool > >'ConcurrentMarkSweep' had collection(s): count=1 time=1967ms INFO > >[JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host > >machine (eg GC): pause of approximately 2304ms GC pool > >'ConcurrentMarkSweep' had collection(s): count=1 time=2775ms INFO > >[JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host > >machine (eg GC): pause of approximately 2287ms GC pool > >'ConcurrentMarkSweep' had collection(s): count=1 time=2775ms > > > > INFO [RS:0;0:0:0:0:0:0:0:0:44037-SendThread(<host>:2181)] > >zookeeper.ClientCnxn: Client session timed out, have not heard from > >server in 6819ms for sessionid 0x1540ab48b280004, closing socket > >connection and attempting reconnect INFO > >[SplitLogWorker-<host>,44037,1460468489645-SendThread(<host>:2181)] > >zookeeper.ClientCnxn: Client session timed out, have not heard from > >server in 6819ms for sessionid 0x1540ab48b280005, closing socket > >connection and attempting reconnect > > > >Once after this HBase Region Server moved to Dead state. > > > >Best, > >Karthik > > This email and any files included with it may contain privileged, proprietary and/or confidential information that is for the sole use of the intended recipient(s). Any disclosure, copying, distribution, posting, or use of the information contained in or attached to this email is prohibited unless permitted by the sender. If you have received this email in error, please immediately notify the sender via return email, telephone, or fax and destroy this original transmission and its included files without reading or saving it in any manner. Thank you.
