Hi David, Thanks for the suggestion, we are already throttling compaction output to reduce impact on other jobs. We just want to move the time so that the compaction can be done faster.
Unfortunately, we are using an older version of HBase provided by Hortonwork where CompactionWindowFactory is not pluggable. We will try your tips when we upgrade to a newer HBase, which hopefully can be done soon. Best regards, Kien On Mar 4, 2020 at 23:44, David Latham <[email protected]> wrote: I helped with the date tiered compaction work, but it's been awhile. I'll chime in in case it's helpful. If you merely want to shift the storm to a different time of day, you may be able to accomplish that by changing the CompactionWindowFactory being used to slide all the windows by X hours. You should be able to plug in an adjusted implementation via the configuration "hbase.hstore.compaction.date.tiered.window.factory.class". If instead you want to smooth out the storm, you may need to look at how compactions are being queued and throttled on the compaction threads. Good luck, and let us know how it goes! Dave On Tue, Mar 3, 2020 at 8:09 PM Kien Truong <[email protected]> wrote: > Hi all, > > We are testing Date Tiered Compaction but encounter a problem with the > promotion time. In the default configuration, at 00:00 UTC every day, there > is a compaction storm when all the store files of the previous day is > promoted and compacted, and possibly all the store files for the previous 4 > days as well. We would like to move this promotion time so that it would > fall into our offpeak hours. How can we achieve this ? > > Thanks, > > Kien >
