Is there any information from this thread that we should make sure gets into the hbase book? it
seem like Wayne went through a lot of work to get good performance and it would be nice if all the
information he gleaned from the community were recorded somewhere. If it doesn't make sense to put
the information in a general performance section, it might make sense to have a performance "case
study" section where for Wayne's case study these are what the performance settings he used to get
good performance.
~Jeff
On 6/2/2011 9:09 AM, Wayne wrote:
I have finally been able to spend enough time to digest/test
all recommendations and get this under control. I wanted to thank Stack,
Jack Levin, and Ted Dunning for their input.
Basically our memory was being pushed to the limit and the JVM does not
like/can not handle this. We are successfully using Todd's MSLAB enabled on
u25 and have set parnew to 128m, increased the heap to 10g, and increased
our block size 4x to 256k to reduce the size of the store file index (thanks
Jack for pointing this out). The combination of all of these changes reduce
significantly the pressure on memory and now we are getting more throughput
(40k writes/sec/node) sustained with no CMF errors (I expect to see one 5
min after hitting send on this email...).
We are also going to move to bulk inserts where we can, which should help as
well. The decreased performance of reads due to the block size increase is
going to be the next challenge. Thanks everyone for your help, I am
a believer again.
On Fri, May 27, 2011 at 12:17 AM, Erik Onnen<eon...@gmail.com> wrote:
On Thu, May 26, 2011 at 11:01 AM, Stack<st...@duboce.net> wrote:
What JVM configs are you running Erik?
St.Ack
Omitting some of the irrelevant ones...
JAVA_OPTS="-XX:+UseLargePages -Xms8192M -Xmx8192M
-XX:+HeapDumpOnOutOfMemoryError -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
-Xloggc:/mnt/services/hbaseregion/var/log/hbaseregion-gc.log
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
For our Cassandra servers we also explicitly pin the new gen tenuring
and occupancy thresholds but that hasn't been necessary for the HBase
workloads so far:
"-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly"
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
