[ https://issues.apache.org/jira/browse/HBASE-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615214#comment-13615214 ]
Hudson commented on HBASE-8163: ------------------------------- Integrated in hbase-0.95-on-hadoop2 #45 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/45/]) HBASE-8163 MemStoreChunkPool: An improvement for JAVA GC when using MSLAB (Revision 1461399) Result = FAILURE zjushch : Files : * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreChunkPool.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreLAB.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreChunkPool.java > MemStoreChunkPool: An improvement for JAVA GC when using MSLAB > -------------------------------------------------------------- > > Key: HBASE-8163 > URL: https://issues.apache.org/jira/browse/HBASE-8163 > Project: HBase > Issue Type: New Feature > Components: regionserver > Reporter: chunhui shen > Assignee: chunhui shen > Fix For: 0.95.0, 0.98.0 > > Attachments: hbase-0.95-8163v6.patch, hbase-8163v1.patch, > hbase-8163v2.patch, hbase-8163v3.patch, hbase-8163v4.patch, > hbase-8163v5.patch, hbase-8163v6.patch > > > *Usage:* > Disable pool(default): configure "hbase.hregion.memstore.chunkpool.maxsize" > as 0 > Enable pool: configure "hbase.hregion.memstore.chunkpool.maxsize" as a > percentage of global memstore size (between 0.0 and 1.0, recommend to set it > as the gap between min and max sizes of memstore, e.g. 0.5) > *Background*: > When we use mslab,we will copy the keyvalue together in a structure called > *MemStoreLAB$Chunk*, therefore we could decrease the heap fragment. > *Problem*: > When one chunk is full, we will create a new chunk, and then the old chunk > will be reclaimed by JVM if no reference to it. > Mostly the chunk object would be promoted when doing Young GC, cause > increasing the cost of YGC > When a Chunk object has no reference? It should meet two following condition: > 1.Memstore which this chunk belongs to is flushed > 2.No scanner is opening on the memstore which this chunk belongs to > *Solution:* > 1.Create a chunk pool to manage the no-reference chunks, instead of being > reclaimed by JVM > 2.When a Chunk has no reference, put it back to the pool > 3.The pool has a max capacity, it will skip the chunks when achieve the max > size > 4.When we need new Chunk to store KeyValue, get it from the pool if exists, > else create new one by pool, so we could be able to reuse the old chunks > *Test results:* > Environment: > hbase-version:0.94 > -Xms4G -Xmx4G -Xmn2G > Row size=50 bytes, Value size=1024 bytes > 50 concurrent theads per client, insert 10,000,000 rows > Before: > Avg write request per second:12953 > After testing, final result of jstat -gcutil : > YGC YGCT FGC FGCT GCT > 747 36.503 48 2.492 38.995 > After: > Avg write request per second:14025 > After testing, final result of jstat -gcutil : > YGC YGCT FGC FGCT GCT > 711 20.344 4 0.284 20.628 > *Improvement: YGC 40+%; WPS 5%+* > review board : > https://reviews.apache.org/r/10056/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira