Hi,
So the way I am thinking to use it is if we lose the EBS volume and we need
to restore the cluster state back. I would have a secondary EBS as my
snapshot directory so I can restore from it.
It means Application would need to be restarted after EBS data is copied
back to the work directory. I see two options here
1. Manual as described in previous reply. manually copy data from snapshot
directory to work/db and restart cluster
2. Use control script : I am not clear on how will this work because If I
restart cluster, it is going to create directory structure again and then
when we run restore command, it does not copy data
Could you please suggest how it would work. directory structure is
attached.
Also, can you suggest a better way to copy snapshot directory data to S3 .
I am thinking of using a kubernetes CSI driver to do it. Any objections to
it
On Thu, Feb 3, 2022 at 4:23 PM Maxim Muzafarov <mmu...@apache.org> wrote:
> Hello,
>
> You don't need to stop the cluster or delete/move any snapshot files
> in case you are using the restore procedure from the control.sh, so
> the following should work:
> - create snapshot
> - stop the caches you are intended to restore
> - run ./control.sh --snapshot restore snapshot_1 --start
>
> Can you provide the directory structure of the Ignite working
> directory? (use `tree` command)
>
> On Wed, 2 Feb 2022 at 22:15, Surinder Mehra <redni...@gmail.com> wrote:
> >
> > Hi,
> > Could you please point out if i missed something?
> >
> > On Wed, Feb 2, 2022, 13:39 Surinder Mehra <redni...@gmail.com> wrote:
> >>
> >> Hey thanks for your suggestions.
> >>
> >> I tried restoring using control.sh but it doesn't seem to work. Below
> are steps
> >> 1. Started 3 nodes and added data using a thick client
> >> 2. created a snapshot using with ./control.sh --snapshot create
> snapshot_1
> >> 3. I verified, the snapshot directory has data
> >> 4. Stopped the cluster and cleared binary_data, marshaler and nodes
> directory /db
> >> 5. Started the cluster again, all 3 nodes
> >> 6. Activate the cluster using ./control.sh --set-state ACTIVE
> >> 7. Run restore command : ./control.sh --snapshot restore snapshot_1
> --start
> >> 8. Command was successful but data is not copied to cluster nodes.
> >>
> >> Please note that when I restarted the cluster, it created binary_data,
> marshaler and nodes directories by default.
> >>
> >> Did I miss anything ?
> >>
> >>
> >> On Tue, Feb 1, 2022 at 8:21 PM Maxim Muzafarov <mmu...@apache.org>
> wrote:
> >>>
> >>> Hello,
> >>>
> >>> Your case looks correct to me, however, I'd like to mention some
> >>> important points that may help you:
> >>> - the directories structure of the snapshot has the same structure as
> >>> the Ignite native persistence, so you may backup the original cluster
> >>> node directory (for binary_data, marshaller and db) and move all the
> >>> files right from the snapshot.
> >>> - do not forget to backup and clear the original wal directory in case
> >>> of restoration.
> >>> - you may use control.sh --snapshot restore command to restore from a
> >>> snapshot (this was added in 2.11)
> >>>
> >>> [1] https://issues.apache.org/jira/browse/IGNITE-13805
> >>>
> >>> On Tue, 1 Feb 2022 at 16:28, Surinder Mehra <redni...@gmail.com>
> wrote:
> >>> >
> >>> > Hi,
> >>> > After a few hiccups, I managed to restore the cluster state from the
> snapshot. Please confirm if they look correct. If so documentation page
> needs to be updated
> >>> >
> >>> > Create N nodes
> >>> > Add some data to them
> >>> > Create snapshot
> >>> > Stop all nodes(cluster)
> >>> > Delete binary_data, marsheller and sub directories of /work/db
> >>> > Copy snapshots/snapshotname/db/binary_data to /work/db/,
> >>> > Copy snapshots/snapshotname/db/marshaller to /work/db/
> >>> > Copy snapshots/snapshotname/db/{nodeid} dir to /work/db/
> >>> > Start cluster
> >>> > Cluster should auto activate after all nodes join it
> >>> > Cluster is ready
> >>> >
> >>> >
> >>> > On Mon, Jan 31, 2022 at 7:14 PM Surinder Mehra <redni...@gmail.com>
> wrote:
> >>> >>
> >>> >> Hi,
> >>> >> We are using ignite 2.11.1 to experiment with ignite snapshots. We
> tried steps mentioned on below page to restore ignite data from snapshot
> >>> >> https://ignite.apache.org/docs/latest/snapshots/snapshots
> >>> >>
> >>> >> But we get the below error when we start a cluster after copying
> data manually as mentioned on the page.
> >>> >>
> >>> >> Steps:
> >>> >> 1.Created 3 nodes and added 3 records
> >>> >>
> >>> >> 2.Created snapshot.
> >>> >> 3. Stopped the cluster and removed files from binary_data and
> marshellar, not the directories. they are present but empty
> >>> >> 4. removed nodeId directories and files under them from /work/db/
> >>> >>
> >>> >> 5. Copied node id directories from snapshot directory to /work/db/.
> I guess the below step meant to say $IGNITE_HOME/work/db/ right ?
> >>> >>
> >>> >> Copy the files belonging to a node with the {node_id} from the
> snapshot into the $IGNITE_HOME/work/ directory. If the db/{node_id}
> directory is not located under the Ignite work dir then you need to copy
> data files there.
> >>> >>
> >>> >> Error : do we need to copy binary_data and marshaler files as well
> or something else missing ?
> >>> >>
> >>> >> Caused by: class org.apache.ignite.IgniteCheckedException: Cannot
> find metadata for object with compact footer (Ignite work directory might
> have been cleared after restart. Make sure that IGNITE_HOME does not point
> to a temp folder or any other folder that is destroyed/cleared on restarts)
> [typeId=-88020438, IGNITE_HOME='null']
> >>> >>
> >>> >> Please note that ignite HOEM/work/db directory has all nodes data
> copied from snapshot, it is not cleared as indicated by error above
> >>> >>
> >>> >>
>
.
├── db
│ ├── binary_meta
│ │ ├── node00-cc75876d-ef52-4e75-9067-bc8816113ea4
│ │ │ ├── -663680413.bin
│ │ │ └── -88020438.bin
│ │ ├── node01-f8123b78-b6c6-4b8b-8383-5b1c73be9512
│ │ │ ├── -663680413.bin
│ │ │ └── -88020438.bin
│ │ └── node02-f87ed92a-d375-413d-aa74-3c44943c1ee3
│ │ ├── -663680413.bin
│ │ └── -88020438.bin
│ ├── marshaller
│ │ ├── -1603532548.classname0
│ │ ├── -663680413.classname0
│ │ └── -88020438.classname0
│ ├── node00-cc75876d-ef52-4e75-9067-bc8816113ea4
│ │ ├── cache-deptCache
│ │ │ ├── cache_data.dat
│ │ │ ├── index.bin
│ │ │ ├── part-1.bin
│ │ │ └── part-7.bin
│ │ ├── cache-ignite-sys-cache
│ │ │ ├── cache_data.dat
│ │ │ └── index.bin
│ │ ├── cache-studentCache
│ │ │ ├── cache_data.dat
│ │ │ ├── index.bin
│ │ │ └── part-112.bin
│ │ ├── cp
│ │ │ ├── 1643811779816-dc9742a6-d697-487e-8813-5e185fe167dd-END.bin
│ │ │ └── 1643811779816-dc9742a6-d697-487e-8813-5e185fe167dd-START.bin
│ │ ├── lock
│ │ ├── maintenance_tasks.mntc
│ │ ├── metastorage
│ │ │ ├── part-0.bin
│ │ │ └── part-1.bin
│ │ ├── snp
│ │ └── TxLog
│ ├── node01-f8123b78-b6c6-4b8b-8383-5b1c73be9512
│ │ ├── cache-deptCache
│ │ │ ├── cache_data.dat
│ │ │ └── index.bin
│ │ ├── cache-ignite-sys-cache
│ │ │ ├── cache_data.dat
│ │ │ └── index.bin
│ │ ├── cache-studentCache
│ │ │ ├── cache_data.dat
│ │ │ ├── index.bin
│ │ │ └── part-870.bin
│ │ ├── cp
│ │ │ ├── 1643811779953-01b6df17-ae58-4c45-b1db-7b065f3a9f78-END.bin
│ │ │ └── 1643811779953-01b6df17-ae58-4c45-b1db-7b065f3a9f78-START.bin
│ │ ├── lock
│ │ ├── maintenance_tasks.mntc
│ │ ├── metastorage
│ │ │ ├── part-0.bin
│ │ │ └── part-1.bin
│ │ ├── snp
│ │ └── TxLog
│ ├── node02-f87ed92a-d375-413d-aa74-3c44943c1ee3
│ │ ├── cache-deptCache
│ │ │ ├── cache_data.dat
│ │ │ ├── index.bin
│ │ │ ├── part-2.bin
│ │ │ ├── part-3.bin
│ │ │ ├── part-5.bin
│ │ │ └── part-6.bin
│ │ ├── cache-ignite-sys-cache
│ │ │ ├── cache_data.dat
│ │ │ └── index.bin
│ │ ├── cache-studentCache
│ │ │ ├── cache_data.dat
│ │ │ ├── index.bin
│ │ │ └── part-513.bin
│ │ ├── cp
│ │ │ ├── 1643811780327-58cdbeeb-6add-436e-8b08-37ca41af1e29-END.bin
│ │ │ └── 1643811780327-58cdbeeb-6add-436e-8b08-37ca41af1e29-START.bin
│ │ ├── lock
│ │ ├── maintenance_tasks.mntc
│ │ ├── metastorage
│ │ │ ├── part-0.bin
│ │ │ └── part-1.bin
│ │ ├── snp
│ │ └── TxLog
│ └── wal
│ ├── archive
│ │ ├── node00-cc75876d-ef52-4e75-9067-bc8816113ea4
│ │ ├── node01-f8123b78-b6c6-4b8b-8383-5b1c73be9512
│ │ └── node02-f87ed92a-d375-413d-aa74-3c44943c1ee3
│ ├── node00-cc75876d-ef52-4e75-9067-bc8816113ea4
│ │ ├── 0000000000000000.wal
│ │ ├── 0000000000000001.wal
│ │ ├── 0000000000000002.wal
│ │ ├── 0000000000000003.wal
│ │ ├── 0000000000000004.wal
│ │ ├── 0000000000000005.wal
│ │ ├── 0000000000000006.wal
│ │ ├── 0000000000000007.wal
│ │ ├── 0000000000000008.wal
│ │ └── 0000000000000009.wal
│ ├── node01-f8123b78-b6c6-4b8b-8383-5b1c73be9512
│ │ ├── 0000000000000000.wal
│ │ ├── 0000000000000001.wal
│ │ ├── 0000000000000002.wal
│ │ ├── 0000000000000003.wal
│ │ ├── 0000000000000004.wal
│ │ ├── 0000000000000005.wal
│ │ ├── 0000000000000006.wal
│ │ ├── 0000000000000007.wal
│ │ ├── 0000000000000008.wal
│ │ └── 0000000000000009.wal
│ └── node02-f87ed92a-d375-413d-aa74-3c44943c1ee3
│ ├── 0000000000000000.wal
│ ├── 0000000000000001.wal
│ ├── 0000000000000002.wal
│ ├── 0000000000000003.wal
│ ├── 0000000000000004.wal
│ ├── 0000000000000005.wal
│ ├── 0000000000000006.wal
│ ├── 0000000000000007.wal
│ ├── 0000000000000008.wal
│ └── 0000000000000009.wal
├── diagnostic
├── log
│ ├── control-utility-0.log
│ ├── control-utility-1.log
│ ├── control-utility-2.log
│ ├── ignite-1098ce8a.0.log
│ ├── ignite-1098ce8a.0.log.lck
│ ├── ignite-909db4ed.0.log
│ ├── ignite-909db4ed.0.log.lck
│ ├── ignite-9e6bbfdf.0.log
│ ├── ignite-9e6bbfdf.0.log.lck
│ ├── ignite-9ef91657.0.log
│ ├── ignite-a49407ba.0.log
│ └── ignite-f7802fc4.0.log
└── snapshots
├── snapshot_1
│ ├── cc75876d_ef52_4e75_9067_bc8816113ea4.smf
│ ├── db
│ │ ├── binary_meta
│ │ │ ├── node00-cc75876d-ef52-4e75-9067-bc8816113ea4
│ │ │ │ ├── -663680413.bin
│ │ │ │ └── -88020438.bin
│ │ │ ├── node01-f8123b78-b6c6-4b8b-8383-5b1c73be9512
│ │ │ │ ├── -663680413.bin
│ │ │ │ └── -88020438.bin
│ │ │ └── node02-f87ed92a-d375-413d-aa74-3c44943c1ee3
│ │ │ ├── -663680413.bin
│ │ │ └── -88020438.bin
│ │ ├── marshaller
│ │ │ ├── -1603532548.classname0
│ │ │ ├── -663680413.classname0
│ │ │ └── -88020438.classname0
│ │ ├── node00-cc75876d-ef52-4e75-9067-bc8816113ea4
│ │ │ ├── cache-deptCache
│ │ │ │ ├── cache_data.dat
│ │ │ │ ├── index.bin
│ │ │ │ └── part-1.bin
│ │ │ ├── cache-studentCache
│ │ │ │ ├── cache_data.dat
│ │ │ │ ├── index.bin
│ │ │ │ └── part-112.bin
│ │ │ └── metastorage
│ │ │ └── part-1.bin
│ │ ├── node01-f8123b78-b6c6-4b8b-8383-5b1c73be9512
│ │ │ ├── cache-deptCache
│ │ │ │ ├── cache_data.dat
│ │ │ │ └── index.bin
│ │ │ ├── cache-studentCache
│ │ │ │ ├── cache_data.dat
│ │ │ │ ├── index.bin
│ │ │ │ └── part-870.bin
│ │ │ └── metastorage
│ │ │ └── part-1.bin
│ │ └── node02-f87ed92a-d375-413d-aa74-3c44943c1ee3
│ │ ├── cache-deptCache
│ │ │ ├── cache_data.dat
│ │ │ ├── index.bin
│ │ │ ├── part-2.bin
│ │ │ └── part-3.bin
│ │ ├── cache-studentCache
│ │ │ ├── cache_data.dat
│ │ │ ├── index.bin
│ │ │ └── part-513.bin
│ │ └── metastorage
│ │ └── part-1.bin
│ ├── f8123b78_b6c6_4b8b_8383_5b1c73be9512.smf
│ └── f87ed92a_d375_413d_aa74_3c44943c1ee3.smf
└── snapshot_2
├── cc75876d_ef52_4e75_9067_bc8816113ea4.smf
├── db
│ ├── binary_meta
│ │ ├── node00-cc75876d-ef52-4e75-9067-bc8816113ea4
│ │ │ ├── -663680413.bin
│ │ │ └── -88020438.bin
│ │ ├── node01-f8123b78-b6c6-4b8b-8383-5b1c73be9512
│ │ │ ├── -663680413.bin
│ │ │ └── -88020438.bin
│ │ └── node02-f87ed92a-d375-413d-aa74-3c44943c1ee3
│ │ ├── -663680413.bin
│ │ └── -88020438.bin
│ ├── marshaller
│ │ ├── -1603532548.classname0
│ │ ├── -663680413.classname0
│ │ └── -88020438.classname0
│ ├── node00-cc75876d-ef52-4e75-9067-bc8816113ea4
│ │ ├── cache-deptCache
│ │ │ ├── cache_data.dat
│ │ │ ├── index.bin
│ │ │ ├── part-1.bin
│ │ │ └── part-7.bin
│ │ ├── cache-studentCache
│ │ │ ├── cache_data.dat
│ │ │ ├── index.bin
│ │ │ └── part-112.bin
│ │ └── metastorage
│ │ └── part-1.bin
│ ├── node01-f8123b78-b6c6-4b8b-8383-5b1c73be9512
│ │ ├── cache-deptCache
│ │ │ ├── cache_data.dat
│ │ │ └── index.bin
│ │ ├── cache-studentCache
│ │ │ ├── cache_data.dat
│ │ │ ├── index.bin
│ │ │ └── part-870.bin
│ │ └── metastorage
│ │ └── part-1.bin
│ └── node02-f87ed92a-d375-413d-aa74-3c44943c1ee3
│ ├── cache-deptCache
│ │ ├── cache_data.dat
│ │ ├── index.bin
│ │ ├── part-2.bin
│ │ ├── part-3.bin
│ │ ├── part-5.bin
│ │ └── part-6.bin
│ ├── cache-studentCache
│ │ ├── cache_data.dat
│ │ ├── index.bin
│ │ └── part-513.bin
│ └── metastorage
│ └── part-1.bin
├── f8123b78_b6c6_4b8b_8383_5b1c73be9512.smf
└── f87ed92a_d375_413d_aa74_3c44943c1ee3.smf
