We've also observed more blocking updates happening with the new policy (flush decision made on data size), but could work-around it by reducing the hbase.hregion.memstore.flush.size setting. The advantage of current policy is we could control the flushed file size more accurately, but meanwhile losing some "compatibility" (requires configuration updating during rolling upgrade).
I'm not sure whether we should rollback, but if stick on current policy there should be more documents, metrics (monitoring heap/data occupancy separately) and log message refinements, etc. Attaching some of the logs we observed, which is pretty confusing w/o knowing the details of implementation: 2017-07-03 16:11:54,724 INFO [B.defaultRpcServer.handler=182,queue=11,port=16020] regionserver.MemStoreFlusher: Blocking updates on hadoop0528.et2.tbsite.net,16020,1497336978160: global memstore heapsize 7.2 G is >= than blocking 7.2 G size 2017-07-03 16:11:54,754 INFO [B.defaultRpcServer.handler=186,queue=15,port=16020] regionserver.MemStoreFlusher: Blocking updates on hadoop0528.et2.tbsite.net,16020,1497336978160: global memstore heapsize 7.2 G is >= than blocking 7.2 G size 2017-07-03 16:11:57,571 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher: Flush of region mainv7_main_result_c,1496,1499062935573.02adfa7cbdc606dce5b79a516e16492a. due to global heap pressure. Total Memstore size=3.2 G, Region memstore size=331.4 M 2017-07-03 16:11:57,571 WARN [B.defaultRpcServer.handler=49,queue=11,port=16020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 2892ms Best Regards, Yu On 6 July 2017 at 00:56, Stack <st...@duboce.net> wrote: > On Wed, Jul 5, 2017 at 6:30 AM, Eshcar Hillel <esh...@yahoo-inc.com.invalid > > > wrote: > > > Hi All, > > I opened a new Jira https://issues.apache.org/jira/browse/HBASE-18294 to > > discuss this question. > > Flush decisions are taken at the region level and also at the region > > server level - there is the question of when to trigger a flush and then > > which region/store to flush.Regions track both their data size (key-value > > size only) and their total heap occupancy (including index and additional > > metadata).One option (which was the past policy) is to trigger flushes > and > > choose flush subjects based on regions heap size - this gives a better > > estimation for sysadmin of how many regions can a RS carry.Another option > > (which is the current policy) is to look at the data size - this gives a > > better estimation of the size of the files that are created by the flush. > > > > > Sounds like we should be doing the former, heap occupancy. An > OutOfMemoryException puts a nail in any benefit other accountings might > have. > > St.Ack > > > > > I see this is as critical to HBase performance and usability, namely > > meeting the user expectation from the system, hence I would like to hear > as > > many voices as possible.Please join the discussion in the Jira and let us > > know what you think. > > Thanks,Eshcar > > > > >
