[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516625#comment-16516625 ]
Ted Yu commented on HBASE-20542: -------------------------------- When I ran the test locally with patch, I saw the following in test output: {code} 2018-06-18 20:43:32,244 ERROR [Time-limited test] regionserver.HRegion(1249): Asked to modify this region's (foobar,,1529379812191.c0c4ada07a3a9905699278a1b1fd63ff.) memStoreSizing to a negative value which is incorrect. Current memStoreSizing=0, delta=-32 java.lang.Exception at org.apache.hadoop.hbase.regionserver.HRegion.checkNegativeMemStoreDataSize(HRegion.java:1249) at org.apache.hadoop.hbase.regionserver.HRegion.incMemStoreSize(HRegion.java:1229) at org.apache.hadoop.hbase.regionserver.RegionServicesForStores.addMemStoreSize(RegionServicesForStores.java:61) at org.apache.hadoop.hbase.regionserver.CompactionPipeline.swap(CompactionPipeline.java:153) at org.apache.hadoop.hbase.regionserver.CompactingMemStore.swapCompactedSegments(CompactingMemStore.java:332) at org.apache.hadoop.hbase.regionserver.MemStoreCompactor.doCompaction(MemStoreCompactor.java:177) at org.apache.hadoop.hbase.regionserver.MemStoreCompactor.start(MemStoreCompactor.java:110) at org.apache.hadoop.hbase.regionserver.CompactingMemStore.inMemoryCompaction(CompactingMemStore.java:459) at org.apache.hadoop.hbase.regionserver.CompactingMemStore.flushInMemory(CompactingMemStore.java:439) at org.apache.hadoop.hbase.regionserver.TestCompactingToCellFlatMapMemStore.testTimeRange(TestCompactingToCellFlatMapMemStore.java:409) at org.apache.hadoop.hbase.regionserver.TestCompactingToCellFlatMapMemStore.testTimeRangeAfterCompaction(TestCompactingToCellFlatMapMemStore.java:374) {code} > Better heap utilization for IMC with MSLABs > ------------------------------------------- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task > Reporter: Eshcar Hillel > Assignee: Eshcar Hillel > Priority: Major > Attachments: HBASE-20542.branch-2.001.patch, run.sh, workloada, > workloadc, workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)