[
https://issues.apache.org/jira/browse/IGNITE-18721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikolay Izhikov updated IGNITE-18721:
-------------------------------------
Fix Version/s: 2.15
> Improve WAL writing during index rebuilding
> -------------------------------------------
>
> Key: IGNITE-18721
> URL: https://issues.apache.org/jira/browse/IGNITE-18721
> Project: Ignite
> Issue Type: Improvement
> Affects Versions: 2.14
> Reporter: Ilya Shishkov
> Assignee: Nikolay Izhikov
> Priority: Major
> Labels: ise
> Fix For: 2.15
>
> Attachments: WalDuringIndexRebuildTest.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Currently, Ignite writes WAL segments during index rebuilding. For large
> index files this process can last for a several hours and can lead to very
> intensive of WAL segments. It seems, that we could optimize or even turn off
> WAL segments writing during index rebuild process in order to minimize index
> rebuilding time.
> In attached test ([^WalDuringIndexRebuildTest.patch]) rebuilding process
> generates 37-38 segments of a 64MB size more than 2GB for an 1,3GB index file.
> In my environment, index rebuild for {{LOG_ONLY}} estimates 28-33 seconds,
> for {{NONE}}: 11-14 seconds.
> Also, there are 2 possible ways now to force index rebuilding:
> # Force index rebuild by a {{control.sh}} (which leads to
> {{IgniteCacheDatabaseSharedManager#forceRebuildIndexes}} calls).
> # Manually delete index.bin files.
> Both rebuilding processes generates different WALRecords in different
> proportions, as you can see in a tables below (results derived from the
> attached test).
> *Summary for force index rebuild:*
> ||Record type ||Cnt||
> |BTREE_PAGE_REPLACE |5045341|
> |PAGE_RECORD |389204|
> |CHECKPOINT_RECORD |4|
> |INIT_NEW_PAGE_RECORD |1|
> |DATA_PAGE_SET_FREE_LIST_PAGE |2|
> |PAGES_LIST_ADD_PAGE |1|
> |METASTORE_DATA_RECORD |1|
> *Summary for index.bin remove:*
> ||Record type ||Count||
> |PAGE_RECORD |386487|
> |BTREE_PAGE_INSERT |5075253|
> |BTREE_FIX_COUNT |20216|
>
> |BTREE_EXISTING_PAGE_SPLIT |351907|
> |CHECKPOINT_RECORD |5|
> |INIT_NEW_PAGE_RECORD |133|
> |DATA_PAGE_INSERT_FRAGMENT_RECORD |2|
> |DATA_PAGE_SET_FREE_LIST_PAGE |3|
> |BTREE_META_PAGE_ADD_ROOT |103|
> |BTREE_INIT_NEW_ROOT |154|
> |BTREE_PAGE_RECYCLE |2|
> |PAGES_LIST_ADD_PAGE |4|
> |PAGES_LIST_REMOVE_PAGE |1|
> |METASTORE_DATA_RECORD |3|
> |BTREE_META_PAGE_INIT_ROOT_V3 |52|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
