>>I'm not sure whether we should rollback, but if stick on current policy there should be more documents, metrics (monitoring heap/data occupancy separately) and log message refinements, etc.
Agree to this. >>Sounds like we should be doing the former, heap occupancy Stack, so do you mean we need to roll back this new change in trunk? The background is https://issues.apache.org/jira/browse/HBASE-16747. Regards Ram On Thu, Jul 6, 2017 at 8:40 AM, Yu Li <car...@gmail.com> wrote: > We've also observed more blocking updates happening with the new policy > (flush decision made on data size), but could work-around it by reducing > the hbase.hregion.memstore.flush.size setting. The advantage of current > policy is we could control the flushed file size more accurately, but > meanwhile losing some "compatibility" (requires configuration updating > during rolling upgrade). > > I'm not sure whether we should rollback, but if stick on current policy > there should be more documents, metrics (monitoring heap/data occupancy > separately) and log message refinements, etc. Attaching some of the logs we > observed, which is pretty confusing w/o knowing the details of > implementation: > > 2017-07-03 16:11:54,724 INFO > [B.defaultRpcServer.handler=182,queue=11,port=16020] > regionserver.MemStoreFlusher: Blocking updates on > hadoop0528.et2.tbsite.net,16020,1497336978160: > global memstore heapsize 7.2 G is >= than blocking 7.2 G size > 2017-07-03 16:11:54,754 INFO > [B.defaultRpcServer.handler=186,queue=15,port=16020] > regionserver.MemStoreFlusher: Blocking updates on > hadoop0528.et2.tbsite.net,16020,1497336978160: > global memstore heapsize 7.2 G is >= than blocking 7.2 G size > 2017-07-03 16:11:57,571 INFO [MemStoreFlusher.0] > regionserver.MemStoreFlusher: Flush of region > mainv7_main_result_c,1496,1499062935573.02adfa7cbdc606dce5b79a516e16492a. > due to global heap pressure. Total Memstore size=3.2 G, Region memstore > size=331.4 M > 2017-07-03 16:11:57,571 WARN > [B.defaultRpcServer.handler=49,queue=11,port=16020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 2892ms > > Best Regards, > Yu > > On 6 July 2017 at 00:56, Stack <st...@duboce.net> wrote: > > > On Wed, Jul 5, 2017 at 6:30 AM, Eshcar Hillel > <esh...@yahoo-inc.com.invalid > > > > > wrote: > > > > > Hi All, > > > I opened a new Jira https://issues.apache.org/jira/browse/HBASE-18294 > to > > > discuss this question. > > > Flush decisions are taken at the region level and also at the region > > > server level - there is the question of when to trigger a flush and > then > > > which region/store to flush.Regions track both their data size > (key-value > > > size only) and their total heap occupancy (including index and > additional > > > metadata).One option (which was the past policy) is to trigger flushes > > and > > > choose flush subjects based on regions heap size - this gives a better > > > estimation for sysadmin of how many regions can a RS carry.Another > option > > > (which is the current policy) is to look at the data size - this gives > a > > > better estimation of the size of the files that are created by the > flush. > > > > > > > > > Sounds like we should be doing the former, heap occupancy. An > > OutOfMemoryException puts a nail in any benefit other accountings might > > have. > > > > St.Ack > > > > > > > > > I see this is as critical to HBase performance and usability, namely > > > meeting the user expectation from the system, hence I would like to > hear > > as > > > many voices as possible.Please join the discussion in the Jira and let > us > > > know what you think. > > > Thanks,Eshcar > > > > > > > > >
