[ https://issues.apache.org/jira/browse/HBASE-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088938#comment-14088938 ]
Lars Hofhansl commented on HBASE-11695: --------------------------------------- We can do what we do what the CompactionChecker does and add a multiplier. (As an aside the default multiplier for the CompactionChecker is 1000, so it would only check every 10000s = 2h 46m, isn't that too rarely?) Another option is to set the period like this: max(wakeFrequency, 2*jitter, flushInteral/10). I.e. we # do not wake up more often that wakeFrequency # do not wake up such that we would request flush of the same region multiple times (2*jitter) # only wakeup often enough to satisfy the flushInterval with an accuracy of 10% The jitter is hardcoded to 20s. wakeFrequency defaults to 10s (it's not actually a frequency, btw), and flushInterval defaults to 1h. So with these defaults we'd wake up to check every 360s, which seems more like it. Or maybe just max(wakeFrequency, 2*jitter)... I.e. every 40s with default settings. But maybe that's too complicate and we just define another multiplier, or a complete new setting - mean another config option, though. > PeriodicFlusher and WakeFrequency issues > ---------------------------------------- > > Key: HBASE-11695 > URL: https://issues.apache.org/jira/browse/HBASE-11695 > Project: HBase > Issue Type: Bug > Affects Versions: 0.94.21 > Reporter: Lars Hofhansl > > We just ran into a flush storm caused by the PeriodicFlusher. > Many memstore became eligible for flushing at exactly the same time, the > effect we've seen is that the exact same region was flushed multiple times, > because the flusher wakes up too often (every 10s). The jitter of 20s is > larger than that and it takes some time to actually flush the memstore. > Here's one example. We've seen 100's of these, monopolizing the flush queue > and preventing "important" flushes from happening. > {code} > 06-Aug-2014 20:11:56 [regionserver60020.periodicFlusher] INFO > org.apache.hadoop.hbase.regionserver.HRegionServer[1397]-- > regionserver60020.periodicFlusher requesting flush for region > tsdb,\x00\x00\x0AO\xCF* > \x00\x00\x01\x00\x01\x1F\x00\x00\x03\x00\x00\x0C,1340147003629.ef4a680b962592de910d0fdeb376dfc2. > after a delay of 13449 > 06-Aug-2014 20:12:06 [regionserver60020.periodicFlusher] INFO > org.apache.hadoop.hbase.regionserver.HRegionServer[1397]-- > regionserver60020.periodicFlusher requesting flush for region > tsdb,\x00\x00\x0AO\xCF* > \x00\x00\x01\x00\x01\x1F\x00\x00\x03\x00\x00\x0C,1340147003629.ef4a680b962592de910d0fdeb376dfc2. > after a delay of 14060 > {code} > So we need to increase the period of the PeriodicFlusher to at least the > random jitter, also increase the default random jitter (20s does not help > with many regions). -- This message was sent by Atlassian JIRA (v6.2#6252)