Re: X3 slow down after moving from HBase 0.90.3 to HBase 0.92.1

2012-11-21 Thread Vincent Barat
Le 21/11/12 18:39, Stack a écrit : So Vincent, the servers are quiet? Which would match your low CPU observation. Clients are unable to send them load for some reason? How many disks. What is your block cache hit number (see regionserver log -- it gets printed every so often or in the b

Re: X3 slow down after moving from HBase 0.90.3 to HBase 0.92.1

2012-11-21 Thread Stack
On Wed, Nov 21, 2012 at 1:04 AM, Vincent Barat wrote: > Hi, > > I've checked my 30 RPC handlers, they are all in a WAITING state: > > Thread 89 (PRI IPC Server handler 6 on 60020): > State: WAITING > Blocked count: 238 > Waited count: 617 > Waiting on > java.util.concurrent.locks.AbstractQ

Re: X3 slow down after moving from HBase 0.90.3 to HBase 0.92.1

2012-11-21 Thread Vincent Barat
Yes we put and get small amount of data. We already increased handler count to 30, maybe it is not enough... We will try this. Le 21/11/12 05:53, Alok Singh a écrit : Do your PUTs and GETs have small amounts of data? If yes, then you can increase the number of handlers. We have a 8-node cluster

Re: X3 slow down after moving from HBase 0.90.3 to HBase 0.92.1

2012-11-21 Thread Vincent Barat
Le 21/11/12 06:05, Stack a écrit : On Tue, Nov 20, 2012 at 8:21 AM, Vincent Barat wrote: We have changed some parameters on our 16(!) region servers : 1GB more -Xmx, more rpc handler (from 10 to 30) longer timeout, but nothing seems to improve the response time: You have taken a look at the

Re: X3 slow down after moving from HBase 0.90.3 to HBase 0.92.1

2012-11-21 Thread Vincent Barat
Hi, I've checked my 30 RPC handlers, they are all in a WAITING state: Thread 89 (PRI IPC Server handler 6 on 60020): State: WAITING Blocked count: 238 Waited count: 617 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@131f139b Stack: sun.misc.Unsaf

Re: X3 slow down after moving from HBase 0.90.3 to HBase 0.92.1

2012-11-21 Thread Vincent Barat
Hi, I've checked my 30 RPC handler, they are all in a WAITING state: Here is some extract for one of our RS (this is similar to all of them): requestsPerSecond=593, numberOfOnlineRegions=584, numberOfStores=1147, numberOfStorefiles=1980, storef

Re: X3 slow down after moving from HBase 0.90.3 to HBase 0.92.1

2012-11-20 Thread Stack
On Tue, Nov 20, 2012 at 8:21 AM, Vincent Barat wrote: > We have changed some parameters on our 16(!) region servers : 1GB more -Xmx, > more rpc handler (from 10 to 30) longer timeout, but nothing seems to > improve the response time: > You have taken a look at the perf chapter Vincent: http://hba

Re: X3 slow down after moving from HBase 0.90.3 to HBase 0.92.1

2012-11-20 Thread Alok Singh
Do your PUTs and GETs have small amounts of data? If yes, then you can increase the number of handlers. We have a 8-node cluster on 0.92.1, and these are some of the setting we changed from 0.90.4 hbase.regionserver.handler.count = 150 hbase.hregion.max.filesize=2147483648 (2GB) The regions serve

X3 slow down after moving from HBase 0.90.3 to HBase 0.92.1

2012-11-20 Thread Vincent Barat
Hi, We have changed some parameters on our 16(!) region servers : 1GB more -Xmx, more rpc handler (from 10 to 30) longer timeout, but nothing seems to improve the response time: - Scans with HBase 0.92 are x3 SLOWER than with HBase 0.90.3 - A lot of simultaneous gets lead to a huge slow down