[ https://issues.apache.org/jira/browse/IGNITE-15818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kirill Tkalenko updated IGNITE-15818: ------------------------------------- Fix Version/s: 3.0.0-alpha6 > [Native Persistence 3.0] Checkpoint, lifecycle and file store refactoring and > re-implementation > ----------------------------------------------------------------------------------------------- > > Key: IGNITE-15818 > URL: https://issues.apache.org/jira/browse/IGNITE-15818 > Project: Ignite > Issue Type: Task > Reporter: Sergey Chugunov > Assignee: Kirill Tkalenko > Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-alpha6 > > > h2. Goal > Port and refactor core classes implementing page-based persistent store in > Ignite 2.x: GridCacheOffheapManager, GridCacheDatabaseSharedManager, > PageMemoryImpl, Checkpointer, FileWriteAheadLogManager. > New checkpoint implementation to avoid excessive logging. > Store lifecycle clarification to avoid complicated and invasive code of > custom lifecycle managed mostly by DatabaseSharedManager. > h2. Items to pay attention to > New checkpoint implementation based on split-file storage, new page index > structure to maintain disk-memory page mapping. > File page store implementation should be extracted from > GridCacheOffheapManager to a separate entity, target implementation should > support new version of checkpoint (split-file store to enable > always-consistent store and to eliminate binary recovery phase). > Support of big pages (256+ kB). > Support of throttling algorithms. > h2. References > New checkpoint design overview is available > [here|https://github.com/apache/ignite-3/blob/ignite-14647/modules/vault/README.md] > h2. Thoughts > Although there is a technical opportunity to have independent checkpoints for > different data regions, managing them could be a nightmare and it's > definitely in the realm of optimizations and out of scope right now. > So, let's assume that there's one good old checkpoint process. There's still > a requirement to have checkpoint markers, but they will not have a reference > to WAL, because there's no WAL. Instead, we will have to store RAFT log > revision per partition. Or not, I'm not that familiar with a recovery > procedure that's currently in development. > Unlike checkpoints in Ignite 2.x, that had DO and REDO operations, new > version will have DO and UNDO. This drastically simplifies both checkpoint > itself and node recovery. But is complicates data access. > There will be two process that will share storage resource: "checkpointer" > and "compactor". Let's examine what compactor should or shouldn't do: > * it should not work in parallel with checkpointer, except for cases when > there are too many layers (more on that later) > * it should merge later checkpoint delta files into main partition files > * it should delete checkpoint markers once all merges are completed for it, > thus markers are decoupled from RAFT log > About "cases when there are too many layers" - too many layers could > compromise reading speed. Number of layers should not increase > uncontrollably. So, when a threshold is exceeded, compactor should start > working no mater what. If anything, writing load can be throttled, reading > matters more. > Recovery procedure: > * read the list of checkpoint markers on engines start > * remove all data from unfinished checkpoint, if it's there > * trim main partition files to their proper size (should check it it's > actually beneficial) > Table start procedure: > * read all layer files headers according to the list of checkpoints > * construct a list oh hash tables (pageId -> pageIndex) for all layers, make > it as effective as possible > * everything else is just like before > Partition removal might be tricky, but we'll see. It's tricky in Ignite 2.x > after all. "Restore partition states" procedure could be revisited, I don't > know how this will work yet. > How to store hashmaps: > regular maps might be too much, we should consider roaring map implementation > or something similar that'll occupy less space. This is only a concern for > in-memory structures. Files on disk may have a list of pairs, that's fine. > Generally speaking, checkpoints with a size of 100 thousand pages are close > to the top limit for most users. Splitting that to 500 partitions, for > example, gives us 200 pages per partition. Entire map should fit into a > single page. > The only exception to these calculations is index.bin. Amount of pages per > checkpoint can be an orders of magnitudes higher, so we should keep an eye on > it, It'll be the main target for testing/benchmarking. Anyway, 4 kilobytes is > enough to fit 512 integer pairs, scaling to 2048 for regular 16 kilobytes > pages. Map won't be too big IMO. > Another important moment - we should enable direct IO, it's supported by Java > natively since version 9 (I guess). There's a chance that not only regular > disk operations will become somewhat faster, but fsync will become > drastically faster as a result. Which is good, fsync can easily take half a > time of the checkpoint, which is just unacceptable. > h2. Thoughts 2.0 > With high likelihood, we'll get rid of index.bin. This will remove the > requirement of having checkpoint markers. > All that we need is a consistently growing local counter that will be used to > mark partition delta files. But, it doesn't need to be global even on a level > of local node, it can be a local counter per partition, that's persisted in > the meta page. This should be further discussed during the implementation. -- This message was sent by Atlassian Jira (v8.20.7#820007)