[ https://issues.apache.org/jira/browse/HBASE-28257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Manning resolved HBASE-28257. ----------------------------------- Resolution: Duplicate > Memstore flushRequest can be blocked by a delayed flush scheduled by > PeriodicMemstoreFlusher, RegionTooBusyExceptions for up to 5 minutes > ----------------------------------------------------------------------------------------------------------------------------------------- > > Key: HBASE-28257 > URL: https://issues.apache.org/jira/browse/HBASE-28257 > Project: HBase > Issue Type: Bug > Components: regionserver > Affects Versions: 2.0.0, 3.0.0 > Reporter: David Manning > Priority: Minor > > *Steps to reproduce:* > # Make an edit to a region. > # Wait 1 hour + 10 seconds (default value of > {{hbase.regionserver.optionalcacheflushinterval}} plus > {{hbase.regionserver.flush.check.period}}.) > # Make a very large number of edits to the region (i.e. >= 1GB, pressure the > memstore.) > *Expected:* > Memstore pressure leads to flushes. > *Result:* > The PeriodicMemstoreFlusher has scheduled a refresh with a random delay of > 0-5 minutes (default for > {{hbase.regionserver.periodicmemstoreflusher.rangeofdelayseconds}}.) Memstore > pressure flushes are blocked by the scheduled delayed flush. Client receives > 0-5 minutes of {{RegionTooBusyExceptions}} until the delayed flush executes. > *Logs:* > 2023-12-13 06:00:13,573 INFO regionserver.HRegionServer - > MemstoreFlusherChore requesting flush of <REGION> because <CF> has an old > edit so flush to free WALs after random delay 166761 ms > 2023-12-13 06:00:53,219 DEBUG regionserver.HRegion - Flush requested on > <REGION> > 2023-12-13 06:01:47,694 WARN regionserver.HRegion - Region is too busy due to > exceeding memstore size limit. > org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=1.0 G, > regionName=<REGION>, server=<SERVER> > at org.apache.hadoop.hbase.regionserver.HRegion.checkResources > ... > repeats > ... > 2023-12-13 06:01:52,223 WARN regionserver.HRegion - Region is too busy due to > exceeding memstore size limit. > org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=1.0 G, > regionName=<REGION>, server=<SERVER> > at org.apache.hadoop.hbase.regionserver.HRegion.checkResources > ... > 2023-12-13 06:03:00,340 INFO regionserver.HRegion - Flushing <REGION> 1/1 > column families, dataSize=534.77 MB heapSize=1.00 GB > 2023-12-13 06:03:09,634 INFO regionserver.HRegion - Finished flush of > dataSize ~534.77 MB/560744948, heapSize ~1.00 GB/1073816296, currentSize=0 > B/0 for <REGION> in 9294ms, sequenceid=21310753, compaction requested=false > Note also this is the same cause as discussed in HBASE-16030 conversation > https://issues.apache.org/jira/browse/HBASE-16030?focusedCommentId=15340153&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15340153 -- This message was sent by Atlassian Jira (v8.20.10#820010)