Re: GC and High CPU

king about GC black magic) > >> From: Jack Levin >> Subject: Re: GC and High CPU >> To: user@hbase.apache.org >> Date: Monday, May 16, 2011, 5:06 PM >> >> Those are the lines I added: >> >> -XX:+CMSIncrementalMode \  < >> -XX:+CMSIncr

Re: GC and High CPU

Well, our issue was that we take an 8 core box, doing 18MB ParNew GC, do it super fast (10x1sec), receive RPC calls to RS at the same time, pile them up due to CPU contention, do GC even faster, run out of user CPU. This is basically the cycle of cpu death. It seems that larger RAM boxes doing th

Re: GC and High CPU

My rule of thumb is "as big as possible but no bigger". The young gen size determines the pause time and frequency of your young generation collections, which are stop-the-world. In my experience the speed is about 0.5-1 second per GB on modern hardware. You get better throughput and less fragment

Re: GC and High CPU

Isn't a 20mb young generation pretty small these days? For a 4gb heap, you might want to try giving 5-10% of it to the young generation by setting -XX:NewSize=256M Some of the older docshave

Re: GC and High CPU

This is the way I read it. "Low processors" == "high CPU tasks, e.g. high load". So, Incremental takes GC down a number of notches when it comes to competing with CPU for APP threads. That being the case the deadlock is less likely. It would be useful to add code to the RS that will start block

Re: GC and High CPU

I don't understand what of the below made a difference though the difference is plain from the GC logs you show. See below: On Mon, May 16, 2011 at 5:06 PM, Jack Levin wrote: > Those are the lines I added: > > -XX:+CMSIncrementalMode \  < >From the doc., it says about i-cms and java6 "

Re: GC and High CPU

increase the chance of stop-the-world GC and should be avoided. > > - Andy > (who always gets nervous when we start talking about GC black magic) > >> From: Jack Levin >> Subject: Re: GC and High CPU >> To: user@hbase.apache.org >> Date: Monday, May 16,

Re: GC and High CPU

This is interesting because our "conventional wisdom" is those settings should increase the chance of stop-the-world GC and should be avoided. - Andy (who always gets nervous when we start talking about GC black magic) > From: Jack Levin > Subject: Re: GC and Hig

Re: GC and High CPU

Those are the lines I added: -XX:+CMSIncrementalMode \ < -XX:+CMSIncrementalPacing \ <--- -XX:-TraceClassUnloading <-- -Jack (used CMS before) On Mon, May 16, 2011 at 4:19 PM, Stack wrote: > So, the change is that you started using CMS?  You were using the > default GC previous

Re: GC and High CPU

So, the change is that you started using CMS? You were using the default GC previous? ParNew is much bigger now. St.Ack On Mon, May 16, 2011 at 4:11 PM, Jack Levin wrote: > I think this will resolve my issue, here is the output: > >     14 2011-05-16T15:58 >     13 2011-05-16T15:59 >     12 20

Re: GC and High CPU

I think this will resolve my issue, here is the output: 14 2011-05-16T15:58 13 2011-05-16T15:59 12 2011-05-16T16:00 14 2011-05-16T16:01 14 2011-05-16T16:02 13 2011-05-16T16:03 11 2011-05-16T16:04 12 2011-05-16T16:05 11 2011-05-16T16:06 16:06:55 10.103.7

Re: GC and High CPU

hlog rolled, and qps was at 1000, not much at all. I tend to believe that concurrent GC collection is taking way too many threads from the app layer and causing CPU to run up. Setting up incremental mode should help, I will do so, and report what I find. -Jack On Mon, May 16, 2011 at 1:48 PM, S

Re: GC and High CPU

On Mon, May 16, 2011 at 1:36 PM, Jack Levin wrote: > How do you tell?  This is the log entries when we had 100% cpu: > > 2011-05-14T15:48:58.240-0700: 5128.407: [GC 5128.407: [ParNew: > 17723K->780K(19136K), 0.0199350 secs] 4309804K->4292973K(5777060K), > 0.0200660 secs] [Times: user=0.07 sys=0.00

Re: GC and High CPU

How do you tell? This is the log entries when we had 100% cpu: 2011-05-14T15:48:58.240-0700: 5128.407: [GC 5128.407: [ParNew: 17723K->780K(19136K), 0.0199350 secs] 4309804K->4292973K(5777060K), 0.0200660 secs] [Times: user=0.07 sys=0.00, real=0.02 secs] 2011-05-14T15:48:58.349-0700: 5128.515: [GC

Re: GC and High CPU

What is the size of your new gen? Is it growing? Does it level off? St.Ack On Mon, May 16, 2011 at 12:39 PM, Jack Levin wrote: > There HEAP would be 8G used out of 12G total.  The gc-log would be > full of ParNew, no FULL GC at all, ParNew just starts at hight rate > (10-15 lines per second).  

Re: GC and High CPU

There HEAP would be 8G used out of 12G total. The gc-log would be full of ParNew, no FULL GC at all, ParNew just starts at hight rate (10-15 lines per second). Even if you remove queries, the JVM will not recover, and user CPU will remain 100%. Only thing left to do is to kill the process. We a

Re: GC and High CPU

On Sun, May 15, 2011 at 5:37 PM, Jack Levin wrote: > I've added occupancy:  export HBASE_OPTS="$HBASE_OPTS -verbose:gc > -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails > -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError > -Xloggc:$HBASE_HOME/logs/gc-hbase.log" > Does the big CPU u

Re: GC and High CPU

If you have a high insert rate then maybe log rolling (which blocks inserts a little) makes it that the calls get queued enough (occupying heap) to make you enter a GC loop of death? Can you enable RPC logging and see if you can confirm that? Thx, J-D On Sun, May 15, 2011 at 5:37 PM, Jack Levin

Re: GC and High CPU

Somewhere like imageshack :) http://img221.imageshack.us/img221/761/screenshot20110515at537p.png -Jack On Sun, May 15, 2011 at 7:09 PM, Ted Yu wrote: > Jack: > Your attachment got filtered. > Can you upload somewhere ? > > Thanks > > On Sun, May 15, 2011 at 5:37 PM, Jack Levin wrote: > >> I've

Re: GC and High CPU

Jack: Your attachment got filtered. Can you upload somewhere ? Thanks On Sun, May 15, 2011 at 5:37 PM, Jack Levin wrote: > I've added occupancy: export HBASE_OPTS="$HBASE_OPTS -verbose:gc > -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails > -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfM

Re: GC and High CPU

I've added occupancy: export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError -Xloggc:$HBASE_HOME/logs/gc-hbase.log" But we had same issue. Whats strange is that the load on the RS would be around

GC and High CPU

Greetings all -- We had some issues with huge GC counts that drive CPU to 100% and rendering RS basically dead, here is some data: We are normal here, about 1 ParNew per second. 67 2011-05-14T15:35 75 2011-05-14T15:36 70 2011-05-14T15:37 74 2011-05-14T15:38 73 2011-05-14T