[
https://issues.apache.org/jira/browse/IGNITE-17613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikolay Izhikov updated IGNITE-17613:
-------------------------------------
Description:
Incremental snapshot is a lightweight alternative to full snapshot. It bases on
the non-blocking Consistent Cut algorithm and provides a collection of WAL
segments that hold logical changes since previous snapshot (full or
incremental).
Incremental snapshot should contain:
* compacted WAL segments
* meta file with Consistent Cut to restore on
* binary_meta if it has changed since previous snapshot.
Incremental snapshot is stored within full snapshot directory.
Incremental snapshot before creation checks:
* Base snapshot (at least its metafile) exists. Exists metafile for all
incremental snapshots.
* Validate that no misses in WAL segments since previous snapshot.
* Check that new _ConsistentCutVersion_ is greater than version of previous
snapshots.
* Check that baseline topology and cacheGroups are the same (relatively to
base snapshot).
More info in IEP:
[https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211884314]
----
Highlevel process of creation of incremental snapshot(without consistent cut
algorithm):
* Incremental snapshot started with create snapshot command with --incremental
flag.
* Incremental snapshot consists of:
** WAL segments from previous increment snapshot (or full snapshot in case
first incremental snapshot).
** Changed binary meta and marshaller files.
** Snapshot metafile
* Incremental snapshot are placed to
{{work/snapshots/mybackup/increments/node01/0001}} folder where
** mybackup - name of the full snapshot.
** node01 - consistent id of the node.
** 0001, 0002, etc - number of incremental snapshot.
* Incremental snapshot creation consists of the following actions executed on
each node. The whole process orchestrated by the {{DistributedProcess}} in the
same manner as the full snapshot creation:
** creation of the snapshot folder
** awaits while required WAL segments will be archived
** copy (hard-linked) required WAL segments to the incremental snapshot
folder.
** creates snapshot metafile.
* Failover guarantees (remove partially created snapshot, etc) should be the
same as for the full snapshot.
* Removal of the full snapshot must also removes all the incremental snapshot
based on the full one.
* Removal of the incremental snapshot may be done only for the last one. If
there are next incremental snapshot (0003, for example) then removal of any
previous (0001 or 0002) must be restricted.
was:
Incremental snapshot is a lightweight alternative to full snapshot. It bases on
the non-blocking Consistent Cut algorithm and provides a collection of WAL
segments that hold logical changes since previous snapshot (full or
incremental).
Incremental snapshot should contain:
* compacted WAL segments
* meta file with Consistent Cut to restore on
* binary_meta if it has changed since previous snapshot.
Incremental snapshot is stored within full snapshot directory.
Incremental snapshot before creation checks:
* Base snapshot (at least its metafile) exists. Exists metafile for all
incremental snapshots.
* Validate that no misses in WAL segments since previous snapshot.
* Check that new _ConsistentCutVersion_ is greater than version of previous
snapshots.
* Check that baseline topology and cacheGroups are the same (relatively to
base snapshot).
More info in IEP:
[https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211884314]
----
Highlevel process of creation of incremental snapshot(without consistent cut
algorithm):
* Incremental snapshot started with create snapshot command with --incremental
flag.
* Incremental snapshot consists of:
** WAL segments from previous increment snapshot (or full snapshot in case
first incremental snapshot).
** Changed binary meta and marshaller files.
** Snapshot metafile
* Incremental snapshot are placed to
{{work/snapshots/mybackup/increments/node01/0001}} folder where
** mybackup - name of the full snapshot.
** node01 - consistent id of the node.
** 0001, 0002, etc - number of incremental snapshot.
* Incremental snapshot creation consists of the following actions executed on
each node. The whole process orchestrated by the {{DistributedProcess}} in the
same manner as the full snapshot creation:
** creation of the snapshot folder
** awaits while required WAL segments will be archived
** copy (hard-linked) required WAL segments to the incremental snapshot
folder.
** creates snapshot metafile.
> Create incremental snapshot
> ---------------------------
>
> Key: IGNITE-17613
> URL: https://issues.apache.org/jira/browse/IGNITE-17613
> Project: Ignite
> Issue Type: Sub-task
> Reporter: Maksim Timonin
> Assignee: Maksim Timonin
> Priority: Major
> Labels: IEP-89, ise
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Incremental snapshot is a lightweight alternative to full snapshot. It bases
> on the non-blocking Consistent Cut algorithm and provides a collection of WAL
> segments that hold logical changes since previous snapshot (full or
> incremental).
> Incremental snapshot should contain:
> * compacted WAL segments
> * meta file with Consistent Cut to restore on
> * binary_meta if it has changed since previous snapshot.
> Incremental snapshot is stored within full snapshot directory.
> Incremental snapshot before creation checks:
> * Base snapshot (at least its metafile) exists. Exists metafile for all
> incremental snapshots.
> * Validate that no misses in WAL segments since previous snapshot.
> * Check that new _ConsistentCutVersion_ is greater than version of previous
> snapshots.
> * Check that baseline topology and cacheGroups are the same (relatively to
> base snapshot).
> More info in IEP:
> [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211884314]
> ----
> Highlevel process of creation of incremental snapshot(without consistent cut
> algorithm):
> * Incremental snapshot started with create snapshot command with
> --incremental flag.
> * Incremental snapshot consists of:
> ** WAL segments from previous increment snapshot (or full snapshot in case
> first incremental snapshot).
> ** Changed binary meta and marshaller files.
> ** Snapshot metafile
> * Incremental snapshot are placed to
> {{work/snapshots/mybackup/increments/node01/0001}} folder where
> ** mybackup - name of the full snapshot.
> ** node01 - consistent id of the node.
> ** 0001, 0002, etc - number of incremental snapshot.
> * Incremental snapshot creation consists of the following actions executed on
> each node. The whole process orchestrated by the {{DistributedProcess}} in
> the same manner as the full snapshot creation:
> ** creation of the snapshot folder
> ** awaits while required WAL segments will be archived
> ** copy (hard-linked) required WAL segments to the incremental snapshot
> folder.
> ** creates snapshot metafile.
> * Failover guarantees (remove partially created snapshot, etc) should be the
> same as for the full snapshot.
> * Removal of the full snapshot must also removes all the incremental snapshot
> based on the full one.
> * Removal of the incremental snapshot may be done only for the last one. If
> there are next incremental snapshot (0003, for example) then removal of any
> previous (0001 or 0002) must be restricted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
