[ https://issues.apache.org/jira/browse/HBASE-17747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907134#comment-15907134 ]
Yu Li commented on HBASE-17747: ------------------------------- Ok, here comes the E2E test report through YCSB (with 100% cache hit ratio): || Case || Off-heap || Round || Throughput || AverageThroughput|| | w/o patch | No | 1 | 275652.627079| | | | | 2 | 248664.333337| | | | | 3 | 264297.580882| | | | | 4 | 253694.307118| | | | | | | 260577.212104 | | w/ patch | No | 1 | 268122.883638| | | | | 2 | 284483.969107| | | | | 3 | 283008.465412| | | | | 4 | 268476.579267| | | | | | | 276022.974356 (+5.9%) | | w/o patch | Yes | 1 | 199073.764764| | | | | 2 | 210060.769909| | | | | 3 | 216729.412051| | | | | 4 | 221610.155932| | | | | | | 211868.525664 | | w/ patch | Yes | 1 | 217398.730766| | | | | 2 | 232581.157272| | | | | 3 | 225765.026296| | | | | 4 | 229071.140011| | | | | | | 226204.013586 (+6.8%) | >From the result we could see with soft reference we get a better performance. More details about testing configuration: {noformat} YCSB: * 4 physical nodes, 8 ycsb processes on each node * recordcount=11000000 * fieldcount=1 * fieldlength=1024 (10.8GB overall) * threadcount=8 * maxexecutiontime=300 * table schema: {NAME => 'cf', DATA_BLOCK_ENCODING => 'DIFF', VERSIONS=> '1', COMPRESSION => 'SNAPPY', IN_MEMORY => 'false', BLOCKCACHE => 'true'},{SPLITS => (1..9).map {|i| "user#{1000+i*(9999-1000)/9}"}, MAX_FILESIZE=>'223372036854775807',DURABILITY=>'SYNC_WAL',METADATA => {'hbase.hstore.block.storage.policy' => 'ALL_SSD'}} HBase: * One single regionserver, 3 datanode * base commit SHA (w/o patch): 31bc94ae60 * hbase.regionserver.handler.count=192 * On-heap: -Xmx42g, hfile.block.cache.size=0.3 * Off-heap: hbase.bucketcache.size=12288, hbase.bucketcache.writer.queuelength=64, hbase.bucketcache.writer.threads=3 {noformat} > Support both weak and soft object pool > -------------------------------------- > > Key: HBASE-17747 > URL: https://issues.apache.org/jira/browse/HBASE-17747 > Project: HBase > Issue Type: Improvement > Reporter: Yu Li > Assignee: Yu Li > Attachments: HBASE-17747.patch, HBASE-17747.v2.patch, > HBASE-17747.v3.patch > > > During YCSB testing on embedded mode after HBASE-17744, we found that under > high read load GC is quite severe even with offheap L2 cache. After some > investigation, we found it's caused by using weak reference for > IdReadWriteLock. In embedded mode the read is so quick that the lock might > already get promoted to the old generation when the weak reference is > cleared, which causes dirty card table thus slowing YGC. > So we proposed to use soft reference for this IdReadWriteLock used in cache, > which won't get cleared until JVM memory is not enough, and could resolve the > issue mentioned above. What's more, we propose to extend the WeakObjectPool > to be more generate to support both weak and soft reference. > Note that this issue only emerges under embedded mode with DirectOperator, in > which case all costs on the wire is removed thus produces extremely high > workloads. -- This message was sent by Atlassian JIRA (v6.3.15#6346)