On Thu, Aug 16, 2018 at 2:27 AM, Vladimir Ozerov <voze...@gridgain.com> wrote:
> Dima, > > None database I know use separate regions for index pages due to the reason > I expressed above. Instead, they split all pages into two groups - hot and > cold. With certain rules on how to move pages inside and between these > groups. None of these algorithms are complex enough. In fact, they are > pretty straightforward and battle-tested. When implemented properly it > doesn't matter whether the page is index page or data page. The only thing > that matter is how often it is accessed. This is critical piece that we > lack in the product - our policy is called "random *LRU*", while in reality > is not LRU at all. > Aerospike keeps index pages in memory, so there is at least one database that does that. I am sure if we research around, there will be more. > As far as index pages replacement we do not know whether this is problem at > all. We heard some complaints that it might be a problem. But we didn't see > any proofs (thanks to lack of monitoring) and even if this is a problem, we > do not understand how severe it is. May be it adds 1% overhead and can be > ignored for years, may be it adds 1000% overhead and must be fixed > immediately. > I remember one test case with a potential user where we had to change our eviction algorithm to avoid evicting index pages and because of that improved performance by about 10x. > This is sensitive piece of a product. Let's use objective data, not > assumptions. > I agree. The difference is that we need a solution in the mean time. I am suggesting a very straight forward approach that can be added fairly quickly and will solve majority performance problems associated with index page eviction. Once we have that, we can take our time and investigate further.
