Igniters, One of the main complaints I hear from users is that whenever the persistence is turned on, index creation can really slow down the performance, because of massive amounts of writes to disk. The reason Ignite is writing indexes to disk is to support fast restarts - nothing needs to be rebuilt on startup, and Ignite can become operational right away.
However, as far as I can tell, most users care about faster operations after the system is started and much less about the startup speed. What if we added a mode where we do not persist indexes at all? This way data ingestion and overall throughput will significantly increase (of course, at the cost of startup type getting longer because we have to rebuild the indexes). There are 2 ways to achieve this in Ignite. The simplest way is not mark index pages dirty in memory, so they will never participate in check-pointing process. We also have to make sure that index pages never get evicted form memory. This can be done fairly quickly. The disadvantage of this approach is that if indexes fill up most of the memory, then it will be very difficult to find a page to evict, which may hurt the performance. The other way is to have a separate in-memory off-heap region for indexes. This region should never be persisted. It maybe somewhat bigger refactoring, as we currently do not separate between index and data pages. However, the advantage of this approach is that this region can be flushed to disk practically as is during a graceful shutdown of the node, and hence shorten the restart time. I think we should start from the 1st approach and then think about the 2nd one. What do you think? D.
