Re: Ignite Snapshots take minutes to complete

2022-05-31 Thread Surinder Mehra
Hey, I have one more query on this one. Currently statefulset "podManagementPolicy" is "OrderedReady". It means kubernetes will start one pod at a time until it reaches the replica count given. If we change "podManagementPolicy" to "Parallel", all pods will start in parallel and hence I can contro

Re: Ignite Snapshots take minutes to complete

2022-05-27 Thread Surinder Mehra
Hi Maxim, As I explained in original email, I do cleanup as part of init container. Since ignite nodes starts after one another, init container would also run in sequence. So when ignite node 1 completes startup(init container clean up work directory and copy data from snapshot) but node 2 is still

Re: Ignite Snapshots take minutes to complete

2022-05-27 Thread Maxim Muzafarov
Hello, If you're copying a snapshot part to the new node, then you have to be sure that the /ignite/work/cp, /ignite/wal, /ignite/walarchive directories are empty prior to the node start. Is it true for your case? On Fri, 27 May 2022 at 10:29, Surinder Mehra wrote: > > Hi, > Please find ignite c

Re: Ignite Snapshots take minutes to complete

2022-05-27 Thread Surinder Mehra
Hi, Please find ignite config and error log below config :

Re: Ignite Snapshots take minutes to complete

2022-05-26 Thread Николай Ижиков
Can you, please, send your config and full log file that contains error message. > 26 мая 2022 г., в 17:50, Surinder Mehra написал(а): > > Hello, > I upgraded to 2.13.0 and I am able to take sync snapshots now. However, I ran > into another problem while restoring from snapshot using manual ste

Re: Ignite Snapshots take minutes to complete

2022-05-26 Thread Surinder Mehra
Hello, I upgraded to 2.13.0 and I am able to take sync snapshots now. However, I ran into another problem while restoring from snapshot using manual steps mentioned in documentation. We run ignite statefulset on kubernetes cluster so when we scale it to N nodes, it brings up one node at a time. N

Re: Ignite Snapshots take minutes to complete

2022-05-24 Thread Surinder Mehra
Thanks a lot. I will try this. On Tue, 24 May 2022, 23:50 Николай Ижиков, wrote: > > Does it ensure consistency while copying data which is parallely getting > updated by application writes > > Yes. > > From the documentation: > > «An Ignite snapshot includes a consistent cluster-wide copy of al

Re: Ignite Snapshots take minutes to complete

2022-05-24 Thread Николай Ижиков
> Does it ensure consistency while copying data which is parallely getting > updated by application writes Yes. From the documentation: «An Ignite snapshot includes a consistent cluster-wide copy of all data records persisted on disk and some other files needed for a restore procedure.» > wi

Re: Ignite Snapshots take minutes to complete

2022-05-24 Thread Surinder Mehra
Hi Thanks for reply. #1: So it's not a stop the world task. Does it ensure consistency while copying data which is parallely getting updated by application writes. Or does it mark the data to copied and ignore further updates on it. #2: I will try sync snapshot. But just to confirm, will this be

Re: Ignite Snapshots take minutes to complete

2022-05-24 Thread Николай Ижиков
Hello, Mehra. > 1. Is it stop the world process. No, you can perform any actions. Note, topology changes will cancel snapshot create process. > 2. If so, is it stop the world only during command execution (500millis) or > until snapshot Dara is fully copied(takes many minutes) to complete. Ple

Ignite Snapshots take minutes to complete

2022-05-24 Thread Surinder Mehra
Hi, I have 3 node ignite cluster each node contains 60G work directory(ebs) and I need to create snapshots. I followed steps to create snapshots and run create snapshot command using control utility. Command completed in 500millis but snapshot directory only had 400Mb data. Later I realised directo