[ https://issues.apache.org/jira/browse/HBASE-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15704403#comment-15704403 ]
stack commented on HBASE-15787: ------------------------------- One note is that we should allow specifying less than full percentages and/or finder granularity decimals. 1% of 100G is 1G which is a lot of RAM. Can we add being able to specify to two decimal places in percntages and four if decimal at least doing heap calculations? Is this up on rb? > Change the flush related heuristics to work with offheap size configured > ------------------------------------------------------------------------ > > Key: HBASE-15787 > URL: https://issues.apache.org/jira/browse/HBASE-15787 > Project: HBase > Issue Type: Sub-task > Components: regionserver > Reporter: ramkrishna.s.vasudevan > Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15787.patch, HBASE-15787_1.patch > > > With offheap MSLAB in place we may have to change the flush related > heuristics to work with offheap size configured rather than the java heap > size. > Since we now have clear seperation of the memstore data size and memstore > heap size, for offheap memstore > -> Decide if the global.offheap.memstore.size is breached for blocking > updates and force flushes. > -> If the onheap global.memstore.size is breached (due to heap overhead) even > then block updates and force flushes. > -> The global.memstore.size.lower.limit is now by default 95% of the > global.memstore.size. So now we apply this 95% on the > global.offheap.memstore.size and also on global.memstore.size (as it was done > for onheap case). > -> We will have new FlushTypes introduced > {code} > ABOVE_ONHEAP_LOWER_MARK, /* happens due to lower mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_lower_mark*/ > ABOVE_ONHEAP_HIGHER_MARK,/* happens due to higher mark breach of onheap > memstore settings > An offheap memstore can even breach the > onheap_higher_mark*/ > ABOVE_OFFHEAP_LOWER_MARK,/* happens due to lower mark breach of offheap > memstore settings*/ > ABOVE_OFFHEAP_HIGHER_MARK; > {code} > -> regionServerAccounting does all the accounting. > -> HeapMemoryTuner is what is litte tricky here. First thing to note is that > at no point it will try to increase or decrease the > global.offheap.memstore.size. If there is a heap pressure then it will try to > increase the memstore heap limit. > In case of offheap memstore there is always a chance that the heap pressure > does not increase. In that case we could ideally decrease the heap limit for > memstore. The current logic of heapmemory tuner is such that things will > naturally settle down. But on discussion what we thought is let us include > the flush count that happens due to offheap pressure but give that a lesser > weightage and thus ensure that the initial decrease on memstore heap limit > does not happen. Currently that fraction is set as 0.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)