[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071856#comment-16071856 ]
Anoop Sam John commented on HBASE-18294: ---------------------------------------- When the data size of the memstore is high (choosing one to flush), the heap occupancy of it also will be on higher side no? Am speaking abt the common case. We have basic flattening as the default now and CompactingMemstore is the default. So we can expect all the memstores to be in this way not like some are flattened and some are never. (Am speaking wrt defaults configs as we have now). One can tune the region flush size as per the new changes. May be the default itself we can change now. But considering data size only for the per region flush decision is more inline with a normal user thinking. 128MB size I have configured and the data is flushed at that data size reach. What is the heap overhead (we have some with DefaultMemstore and some thing else when compacting memstore basic mode in place etc. Tomorrow if a new algorithm comes it may even reduce) is not a user headache. Still we have to consider that as we can not make our RS to OOME or have GC bad impacts. That any way at global level we are doing. In normal cases also flushes due to global pressure might be happening. Say we have 100 regions per RS and then as per default settings, the ideal heap size need for global memstores is 100 * 128 MB = 12.5 GB 12.5 * 4 = 50 GB. We allow the memstores size to grow 4 times 128 MB before blocking. So configuring this big size might not be the usual case. Agree that we will kick start the flush of region once the size is 128 MB. But if the write pressure is high the size can grow beyond 2x . > Flush is based on data size instead of heap size > ------------------------------------------------ > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Bug > Reporter: Eshcar Hillel > Assignee: Eshcar Hillel > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v6.4.14#64029)