On Tue, 2010-02-16 at 00:14 -0600, Stack wrote:
> On Mon, Feb 15, 2010 at 10:05 PM, James Baldassari <[email protected]> wrote:
> > Applying HBASE-2180 isn't really an option at this
> > time because we've been told to stick with the Cloudera distro.
>
> I'm sure the wouldn't mind (smile). Seems to about double throughput.
Hmm, well I might be able to convince them ;)
>
>
> > If I had to guess, I would say the performance issues start to happen
> > around the time the region servers hit max heap size, which occurs
> > within minutes of exposing the app to live traffic. Could GC be killing
> > us? We use the concurrent collector as suggested. I saw on the
> > performance page some mention of limiting the size of the new generation
> > like -XX:NewSize=6m -XX:MaxNewSize=6m. Is that worth trying?
>
> Enable GC logging for a while? See hbase-env.sh. Uncomment this line:
>
> # export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails
> XX:+PrintGCDateStamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
I did uncomment that line, but I can't figure out where the gc-hbase.log
is. It's not with the other logs. When starting HBase the GC output
seems to be going to stdout rather than the file. Maybe a Cloudera
thing. I'll do some digging.
>
> You are using recent JVM? 1.6.0_10 or greater? 1.6.0_18 might have issues.
We're on 1.6.0_16 at the moment.
>
> Whats CPU and iowait or wa in top look like on these machines,
> particularly the loaded machine?
>
> How many disks in the machines?
I'll have to ask our ops guys about the disks. The high load has now
switched from region server 1 to 3. I just saw in our logs that it took
139383.065 milliseconds to do 5000 gets, ~36 gets/second, ouch. Here
are the highlights from top for each region server:
Region Server 1:
top - 01:39:41 up 4 days, 13:44, 4 users, load average: 1.89, 0.99, 1.19
Tasks: 194 total, 1 running, 193 sleeping, 0 stopped, 0 zombie
Cpu(s): 15.6%us, 5.8%sy, 0.0%ni, 76.9%id, 0.0%wa, 0.1%hi, 1.6%si, 0.0%st
Mem: 8166588k total, 8112812k used, 53776k free, 8832k buffers
Swap: 1052248k total, 152k used, 1052096k free, 2831076k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21961 hadoop 19 0 4830m 4.2g 10m S 114.3 53.6 37:26.58 java
21618 hadoop 21 0 4643m 578m 9804 S 66.1 7.3 19:06.89 java
Region Server 2:
top - 01:40:28 up 4 days, 13:43, 4 users, load average: 3.93, 2.17, 1.39
Tasks: 194 total, 1 running, 193 sleeping, 0 stopped, 0 zombie
Cpu(s): 11.3%us, 3.1%sy, 0.0%ni, 83.4%id, 1.2%wa, 0.1%hi, 0.9%si, 0.0%st
Mem: 8166588k total, 7971572k used, 195016k free, 34972k buffers
Swap: 1052248k total, 152k used, 1052096k free, 2944712k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15752 hadoop 18 0 4742m 4.1g 10m S 210.6 53.1 41:52.80 java
15405 hadoop 20 0 4660m 317m 9800 S 114.0 4.0 27:34.17 java
Region Server 3:
top - 01:40:35 up 2 days, 9:04, 4 users, load average: 10.15, 11.05, 11.79
Tasks: 195 total, 1 running, 194 sleeping, 0 stopped, 0 zombie
Cpu(s): 28.7%us, 10.1%sy, 0.0%ni, 25.8%id, 32.9%wa, 0.1%hi, 2.4%si, 0.0%st
Mem: 8166572k total, 8118592k used, 47980k free, 3264k buffers
Swap: 1052248k total, 140k used, 1052108k free, 2099896k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15636 hadoop 18 0 4806m 4.2g 10m S 206.9 53.3 87:48.81 java
15243 hadoop 18 0 4734m 1.3g 9800 S 117.6 16.7 63:46.52 java
-James
>
> St>Ack
>
>
>
> >
> > Here are the new region server stats along with load averages:
> >
> > Region Server 1:
> > request=0.0, regions=16, stores=16, storefiles=33, storefileIndexSize=4,
> > memstoreSize=1, compactionQueueSize=0, usedHeap=2891, maxHeap=4079,
> > blockCacheSize=1403878072, blockCacheFree=307135816, blockCacheCount=21107,
> > blockCacheHitRatio=84, fsReadLatency=0, fsWriteLatency=0, fsSyncLatency=0
> > Load Averages: 10.34, 10.58, 7.08
> >
> > Region Server 2:
> > request=0.0, regions=15, stores=16, storefiles=26, storefileIndexSize=3,
> > memstoreSize=1, compactionQueueSize=0, usedHeap=3257, maxHeap=4079,
> > blockCacheSize=661765368, blockCacheFree=193741576, blockCacheCount=9942,
> > blockCacheHitRatio=77, fsReadLatency=0, fsWriteLatency=0, fsSyncLatency=0
> > Load Averages: 1.90, 1.23, 0.98
> >
> > Region Server 3:
> > request=0.0, regions=16, stores=16, storefiles=41, storefileIndexSize=4,
> > memstoreSize=4, compactionQueueSize=0, usedHeap=1627, maxHeap=4079,
> > blockCacheSize=665117184, blockCacheFree=190389760, blockCacheCount=9995,
> > blockCacheHitRatio=70, fsReadLatency=0, fsWriteLatency=0, fsSyncLatency=0
> > Load Averages: 2.01, 3.56, 4.18
> >
> > That first region server is getting hit much harder than the others.
> > They're identical machines (8-core), and the distribution of keys should
> > be fairly random, so I'm not sure why that would happen. Any other
> > ideas or suggestions would be greatly appreciated.
> >
> > Thanks,
> > James
> >
> >
> > On Mon, 2010-02-15 at 21:51 -0600, Stack wrote:
> >> Yeah, I was going to say that if your loading is mostly read, you can
> >> probably go up from the 0.2 given over to cache. I like Dan's
> >> suggestion of trying it first on one server, if you can.
> >>
> >> St.Ack
> >>
> >> On Mon, Feb 15, 2010 at 5:22 PM, Dan Washusen <[email protected]> wrote:
> >> > So roughly 72% of reads use the blocks held in the block cache...
> >> >
> >> > It would be interesting to see the difference between when it was
> >> > working OK
> >> > and now. Could you try increasing the memory allocated to one of the
> >> > regions and also increasing the "hfile.block.cache.size" to say '0.4' on
> >> > the
> >> > same region?
> >> >
> >> > On 16 February 2010 11:54, James Baldassari <[email protected]> wrote:
> >> >
> >> >> Hi Dan. Thanks for your suggestions. I am doing writes at the same
> >> >> time as reads, but there are usually many more reads than writes. Here
> >> >> are the stats for all three region servers:
> >> >>
> >> >> Region Server 1:
> >> >> request=0.0, regions=15, stores=16, storefiles=34, storefileIndexSize=3,
> >> >> memstoreSize=308, compactionQueueSize=0, usedHeap=3096, maxHeap=4079,
> >> >> blockCacheSize=705474544, blockCacheFree=150032400,
> >> >> blockCacheCount=10606,
> >> >> blockCacheHitRatio=76, fsReadLatency=0, fsWriteLatency=0,
> >> >> fsSyncLatency=0
> >> >>
> >> >> Region Server 2:
> >> >> request=0.0, regions=16, stores=16, storefiles=39, storefileIndexSize=4,
> >> >> memstoreSize=225, compactionQueueSize=0, usedHeap=3380, maxHeap=4079,
> >> >> blockCacheSize=643172800, blockCacheFree=212334144,
> >> >> blockCacheCount=9660,
> >> >> blockCacheHitRatio=69, fsReadLatency=0, fsWriteLatency=0,
> >> >> fsSyncLatency=0
> >> >>
> >> >> Region Server 3:
> >> >> request=0.0, regions=13, stores=13, storefiles=31, storefileIndexSize=4,
> >> >> memstoreSize=177, compactionQueueSize=0, usedHeap=1905, maxHeap=4079,
> >> >> blockCacheSize=682848608, blockCacheFree=172658336,
> >> >> blockCacheCount=10262,
> >> >> blockCacheHitRatio=72, fsReadLatency=0, fsWriteLatency=0,
> >> >> fsSyncLatency=0
> >> >>
> >> >> The average blockCacheHitRatio is about 72. Is this too low? Anything
> >> >> else I can check?
> >> >>
> >> >> -James
> >> >>
> >> >>
> >> >> On Mon, 2010-02-15 at 18:16 -0600, Dan Washusen wrote:
> >> >> > Maybe the block cache is thrashing?
> >> >> >
> >> >> > If you are regularly writing data to your tables then it's possible
> >> >> > that
> >> >> the
> >> >> > block cache is no longer being effective. On the region server web UI
> >> >> check
> >> >> > the blockCacheHitRatio value. You want this value to be high (0 -
> >> >> > 100).
> >> >> If
> >> >> > this value is low it means that HBase has to go to disk to fetch
> >> >> > blocks
> >> >> of
> >> >> > data. You can control the amount of VM memory that HBase allocates to
> >> >> the
> >> >> > block cache using the "hfile.block.cache.size" property (default is
> >> >> > 0.2
> >> >> > (20%)).
> >> >> >
> >> >> > Cheers,
> >> >> > Dan
> >> >> >
> >> >> > On 16 February 2010 10:45, James Baldassari <[email protected]> wrote:
> >> >> >
> >> >> > > Hi,
> >> >> > >
> >> >> > > Does anyone have any tips to share regarding optimization for random
> >> >> > > read performance? For writes I've found that setting a large write
> >> >> > > buffer and setting auto-flush to false on the client side
> >> >> > > significantly
> >> >> > > improved put performance. Are there any similar easy tweaks to
> >> >> > > improve
> >> >> > > random read performance?
> >> >> > >
> >> >> > > I'm using HBase 0.20.3 in a very read-heavy real-time system with 1
> >> >> > > master and 3 region servers. It was working ok for a while, but
> >> >> > > today
> >> >> > > there was a severe degradation in read performance. Restarting
> >> >> > > Hadoop
> >> >> > > and HBase didn't help, are there are no errors in the logs. Read
> >> >> > > performance starts off around 1,000-2,000 gets/second but quickly
> >> >> > > (within minutes) drops to around 100 gets/second.
> >> >> > >
> >> >> > > I've already looked at the performance tuning wiki page. On the
> >> >> > > server
> >> >> > > side I've increased hbase.regionserver.handler.count from 10 to 100,
> >> >> but
> >> >> > > it didn't help. Maybe this is expected because I'm only using a
> >> >> > > single
> >> >> > > client to do reads. I'm working on implementing a client pool now,
> >> >> > > but
> >> >> > > I'm wondering if there are any other settings on the server or
> >> >> > > client
> >> >> > > side that might improve things.
> >> >> > >
> >> >> > > Thanks,
> >> >> > > James
> >> >> > >
> >> >> > >
> >> >> > >
> >> >>
> >> >>
> >> >
> >
> >
