-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Stack Sent: Thursday, June 09, 2011 1:53 PM To: [email protected] Cc: Yanlijun; Chenjian Subject: Re: Outdated data can not be cleaned in time
On Wed, Jun 8, 2011 at 9:14 PM, Zhoushuaifeng <[email protected]> wrote: > //zhou: I agree regions with more files should take higher priority, but > there are other factors important should be considered. In our test case, we > found some regions sent to the queue by major compact checker hunging in the > queue for more than 2 days! Some scanners on these regions cannot get > availably data for a long time and lease expired. > I think set these regions priority to hbase.hstore.blockingStoreFiles - > hbase.hstore.compactionThreshold -1 as default may be a good way to solve > this problem. If regions have less than 3 files, it's priority is lower than > outdated regions, but if it has more than 4 files, it's priority will be > higher. This settings can both solve the outdated problem and will not block > flush and put. > I would not build on hbase.hstore.blockingStoreFiles. That should be going away. What else would you suggest? //zhou: other way is when major compact checker send regions to the compact queue, consider how long the oldest timestamp passed the TTL. The older get the higher priority. And what do you mean by 'That should be going away' ? I mean change like this in the compaction checker: + majorCompactPriority = this.instance.conf.getInt("hbase.regionserver.compactionChecker.majorCompactPriority", + this.instance.conf.getInt("hbase.hstore.blockingStoreFiles", 7) + - this.instance.conf.getInt("hbase.hstore.compactionThreshold", 3) + -1); if (s.isMajorCompaction()) { + // Queue a compaction. Will recognize if major is needed. + this.instance.compactSplitThread.requestCompaction(r, s, + getName() + " requests major compaction", this.majorCompactPriority); + } > Maybe the compaction algorithm should consider age of compactions too? > If a compaction has been hanging the queue a good while, its priority > gets bumped a level? > //zhou: This is good, I totally agree. This is another good way to solve the > problem. But by now, I don't know how to make the patch, may be we can dig > more on it. > Want to file an issue on this Zhou? //ok, I will file an issue, and provide the possible solution and patch. St.Ack
