Nick, I am still prototyping, hence the small instance. I thought XX:MaxDirectMemorySize is subset of Xmx, so I thought the total is 6GB. You are saying they are added. If so, that's not good for performance as the JVM will swap. However, this can't be the reason for the bug.
For now, I guess the only option is to use LruBlockCache. However, I expect it to suffer from significant GC, as I churn memory fast because I want to cache blocks on writes and I do significant writing. Khaled ---------------------------------------- > From: [email protected] > Date: Fri, 10 Oct 2014 15:00:45 -0700 > Subject: Re: HBase read performance > To: [email protected] > > Hang on, how are you using 11G total memory? m1.large only has 7.5G total > RAM. > > On Fri, Oct 10, 2014 at 2:56 PM, Nick Dimiduk <[email protected]> wrote: > >> ByteBuffer position math errors makes me suspect #1 cacheonwrite and #2 >> bucketcache (and #3 their use in combination ;) ) >> >> 11G memory is a small enough footprint that I'd not bother with >> BucketCache; just stay on heap with default LruBlockCache. >> >> On Fri, Oct 10, 2014 at 2:11 PM, Stack <[email protected]> wrote: >> >>> On Fri, Oct 10, 2014 at 10:59 AM, Khaled Elmeleegy <[email protected]> >>> wrote: >>> >>>> Yes, I can reproduce it with some work. >>>> The workload is basically as follows: >>>> There are writers streaming writes to a table. Then, there is a reader >>>> (invoked via a web interface). The reader does a 1000 parallel reverse >>>> scans, all end up hitting the same region in my case. The scans are >>>> effectively "gets" as I just need to get one record off of each of >>> them. I >>>> just need to do a "reverse" get, which is not supported (would be great >>> to >>>> have :)), so I do it via reverse scan. After few tries, the reader >>>> consistently hits this bug. >>>> >>>> This happens with these config changes: >>>> hbase-env:HBASE_REGIONSERVER_OPTS=-Xmx6G -XX:MaxDirectMemorySize=5G >>>> -XX:CMSInitiatingOccupancyFraction=88 -XX:+AggressiveOpts -verbose:gc >>>> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xlog >>>> gc:/tmp/hbase-regionserver-gc.loghbase-site: >>>> hbase.bucketcache.ioengine=offheap >>>> hbase.bucketcache.size=4196 >>>> hbase.rs.cacheblocksonwrite=true >>>> hfile.block.index.cacheonwrite=true >>>> hfile.block.bloom.cacheonwrite=true >>>> >>>> Interestingly, without these config changes, I can't reproduce the >>> problem. >>> >>> >>> >>> How hard to play w/ combinations? Could you eliminate the cacheonwrites >>> on >>> one server and see if that cures the issue? Could trun off block cache on >>> another to see if that the problem? Anything in your .out files related? >>> >>> St.Ack >>> >> >>
