[
https://issues.apache.org/jira/browse/IGNITE-18721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ilya Shishkov updated IGNITE-18721:
-----------------------------------
Description:
Currently, Ignite writes WAL segments during index rebuilding. For large index
files this process can last for a several hours and can lead to a generation of
very excessive WAL segments amount. It seems, that we could optimize or even
turn off WAL segments writing during index rebuild process in order to minimize
index rebuilding time.
In attached test ([^WalDuringIndexRebuildTest.patch]) rebuilding process
generates 37-38 segments of a 64MB size more than 2GB for an 1,3GB index file.
In my environment, index rebuild for {{LOG_ONLY}} estimates 28-33 seconds, for
{{NONE}} - 11-14 seconds.
Also, there are 2 possible ways now to force index rebuilding:
# Force index rebuild by a {{control.sh}}.
# Manually delete index.bin files.
Both rebuilding processes generates different WALRecords in different
proportions, as you can see in a tables below (results derived from the
attached test).
*Summary for index rebuild task:*
||Record type ||Cnt||
|BTREE_PAGE_REPLACE |5045341|
|PAGE_RECORD |389204|
|CHECKPOINT_RECORD |4|
|INIT_NEW_PAGE_RECORD |1|
|DATA_PAGE_SET_FREE_LIST_PAGE |2|
|PAGES_LIST_ADD_PAGE |1|
|METASTORE_DATA_RECORD |1|
*Summary for index.bin remove:*
||Record type ||Count||
|PAGE_RECORD |386487|
|BTREE_PAGE_INSERT |5075253|
|BTREE_FIX_COUNT |20216|
|BTREE_EXISTING_PAGE_SPLIT |351907|
|CHECKPOINT_RECORD |5|
|INIT_NEW_PAGE_RECORD |133|
|DATA_PAGE_INSERT_FRAGMENT_RECORD |2|
|DATA_PAGE_SET_FREE_LIST_PAGE |3|
|BTREE_META_PAGE_ADD_ROOT |103|
|BTREE_INIT_NEW_ROOT |154|
|BTREE_PAGE_RECYCLE |2|
|PAGES_LIST_ADD_PAGE |4|
|PAGES_LIST_REMOVE_PAGE |1|
|METASTORE_DATA_RECORD |3|
|BTREE_META_PAGE_INIT_ROOT_V3 |52|
was:
Currently, Ignite writes WAL segments during index rebuilding. For large index
files this process can last for a several hours and can lead to a generation of
very excessive WAL segments amount. It seems, that we could optimize or even
turn off WAL segments writing during index rebuild process in order to minimize
index rebuilding time.
In attached test ([^WalDuringIndexRebuildTest.patch]) rebuilding process
generates 37-38 segments of a 64MB size more than 2GB for an 1,3GB index file.
In my environment, index rebuild for {{LOG_ONLY}} estimates 28-33 seconds, for
{{NONE}} - 11-14 seconds.
Also, there are 2 possible ways now to force index rebuilding:
# Force index rebuild by a {{control.sh}}.
# Manually delete index.bin files.
Both rebuilding processes generates different WALRecords in different
proportions, as you can see in a tables below (results derived from .
*Summary for index rebuild task:*
||Record type ||Cnt||
|BTREE_PAGE_REPLACE |5045341|
|PAGE_RECORD |389204|
|CHECKPOINT_RECORD |4|
|INIT_NEW_PAGE_RECORD |1|
|DATA_PAGE_SET_FREE_LIST_PAGE |2|
|PAGES_LIST_ADD_PAGE |1|
|METASTORE_DATA_RECORD |1|
*Summary for index.bin remove:*
||Record type ||Count||
|PAGE_RECORD |386487|
|BTREE_PAGE_INSERT |5075253|
|BTREE_FIX_COUNT |20216|
|BTREE_EXISTING_PAGE_SPLIT |351907|
|CHECKPOINT_RECORD |5|
|INIT_NEW_PAGE_RECORD |133|
|DATA_PAGE_INSERT_FRAGMENT_RECORD |2|
|DATA_PAGE_SET_FREE_LIST_PAGE |3|
|BTREE_META_PAGE_ADD_ROOT |103|
|BTREE_INIT_NEW_ROOT |154|
|BTREE_PAGE_RECYCLE |2|
|PAGES_LIST_ADD_PAGE |4|
|PAGES_LIST_REMOVE_PAGE |1|
|METASTORE_DATA_RECORD |3|
|BTREE_META_PAGE_INIT_ROOT_V3 |52|
> Improve WAL duing index rebuild
> --------------------------------
>
> Key: IGNITE-18721
> URL: https://issues.apache.org/jira/browse/IGNITE-18721
> Project: Ignite
> Issue Type: Improvement
> Reporter: Ilya Shishkov
> Priority: Minor
> Attachments: WalDuringIndexRebuildTest.patch
>
>
> Currently, Ignite writes WAL segments during index rebuilding. For large
> index files this process can last for a several hours and can lead to a
> generation of very excessive WAL segments amount. It seems, that we could
> optimize or even turn off WAL segments writing during index rebuild process
> in order to minimize index rebuilding time.
> In attached test ([^WalDuringIndexRebuildTest.patch]) rebuilding process
> generates 37-38 segments of a 64MB size more than 2GB for an 1,3GB index file.
> In my environment, index rebuild for {{LOG_ONLY}} estimates 28-33 seconds,
> for {{NONE}} - 11-14 seconds.
> Also, there are 2 possible ways now to force index rebuilding:
> # Force index rebuild by a {{control.sh}}.
> # Manually delete index.bin files.
> Both rebuilding processes generates different WALRecords in different
> proportions, as you can see in a tables below (results derived from the
> attached test).
> *Summary for index rebuild task:*
> ||Record type ||Cnt||
> |BTREE_PAGE_REPLACE |5045341|
> |PAGE_RECORD |389204|
> |CHECKPOINT_RECORD |4|
> |INIT_NEW_PAGE_RECORD |1|
> |DATA_PAGE_SET_FREE_LIST_PAGE |2|
> |PAGES_LIST_ADD_PAGE |1|
> |METASTORE_DATA_RECORD |1|
> *Summary for index.bin remove:*
> ||Record type ||Count||
> |PAGE_RECORD |386487|
> |BTREE_PAGE_INSERT |5075253|
> |BTREE_FIX_COUNT |20216|
>
> |BTREE_EXISTING_PAGE_SPLIT |351907|
> |CHECKPOINT_RECORD |5|
> |INIT_NEW_PAGE_RECORD |133|
> |DATA_PAGE_INSERT_FRAGMENT_RECORD |2|
> |DATA_PAGE_SET_FREE_LIST_PAGE |3|
> |BTREE_META_PAGE_ADD_ROOT |103|
> |BTREE_INIT_NEW_ROOT |154|
> |BTREE_PAGE_RECYCLE |2|
> |PAGES_LIST_ADD_PAGE |4|
> |PAGES_LIST_REMOVE_PAGE |1|
> |METASTORE_DATA_RECORD |3|
> |BTREE_META_PAGE_INIT_ROOT_V3 |52|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
