Re: Ever increasing startup times as data grow in persistent storage

We have been continuing to monitor this issue and have experimented more with deactivation versus Ingition.Stop(). There is some evidence that using graceful shutdown does not perform a checkpoint in that startup times are not necessarily improved on restart. We'll continue to investigate this but

Re: Ever increasing startup times as data grow in persistent storage

Hi, I don't think there are any other options at the moment other than the ones you mentioned. However, you can also create your own application that will check the topology and activate it when all nodes from the baseline are online. For example, additional java code when starting a server

Re: Ever increasing startup times as data grow in persistent storage

Hi Andre, I would like to see Ignite support a graceful shutdown scenario you get with deactivation, but which does not need to be manually reactivated. We run a pretty agile process and it is not uncommon to have multiple deploys to production throughout a week. This is a pretty automated affair

Re: Ever increasing startup times as data grow in persistent storage

Hi, Yes, that was to be expected. The main autoactivation scenario is cluster restart. If you are using manual deactivation, you should also manually activate your cluster. BR, Andrei 1/20/2021 5:50 AM, Raymond Wilson пишет: We have been experimenting with using deactivation to shutdown the

Re: Ever increasing startup times as data grow in persistent storage

We have been experimenting with using deactivation to shutdown the grid to reduce the time for the grid to start up again. It appears there is a downside to this: once deactivated the grid does not appear to auto-activate once baseline topology is achieved, which means we will need to run through

Re: Ever increasing startup times as data grow in persistent storage

Hi Raymond It does block writes until the checkpoint is complete, but this only happens when we restart our nodes, that time all the piled up requests (during the shutdown) gets processed, thats when bulk data ingestion happens, otherwise for normal day to day real time operations it does not real

Re: Ever increasing startup times as data grow in persistent storage

Raymond, Please use ICluster.SetActive [1] instead, the API linked above is obsolete [1] https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.Cluster.ICluster.html?#Apache_Ignite_Core_Cluster_ICluster_SetActive_System_Boolean_ On Wed, Jan 13, 2021 at 11:54 AM Raymond Wilso

Re: Ever increasing startup times as data grow in persistent storage

Of course. Obvious! :) Sent from my iPhone On 13/01/2021, at 9:15 PM, Zhenya Stanilovsky wrote:  Is there an API version of the cluster deactivation? https://github.com/apache/ignite/blob/master/modules/platforms/dotnet/Apache.Ignite.Core.Tests/Cache/PersistentStoreTestObsolete.cs#L131

Re[4]: Ever increasing startup times as data grow in persistent storage

  >Is there an API version of the cluster deactivation?   https://github.com/apache/ignite/blob/master/modules/platforms/dotnet/Apache.Ignite.Core.Tests/Cache/PersistentStoreTestObsolete.cs#L131   >On Wed, Jan 13, 2021 at 8:28 PM Zhenya Stanilovsky < arzamas...@mail.ru > >wrote: >> >> >>  >>>H

Re: Re[2]: Ever increasing startup times as data grow in persistent storage

Is there an API version of the cluster deactivation? On Wed, Jan 13, 2021 at 8:28 PM Zhenya Stanilovsky wrote: > > > > > Hi Zhenya, > > Thanks for confirming performing checkpoints more often will help here. > > Hi Raymond ! > > > I have established this configuration so will experiment with set

Re: Re[2]: Ever increasing startup times as data grow in persistent storage

Thank Zhenya. Currently we call Ignition.Stop() with the flag to allow jobs to complete. I assume when using deactivate we don;t need to call that, or is it still a good idea as a belt and braces shut down for the grid? Raymond On Wed, Jan 13, 2021 at 8:28 PM Zhenya Stanilovsky wrote: > > > >

Re[2]: Ever increasing startup times as data grow in persistent storage

  >Hi Zhenya, >  >Thanks for confirming performing checkpoints more often will help here. Hi Raymond ! >  >I have established this configuration so will experiment with settings little. >  >On a related note, is there any way to automatically trigger a checkpoint, for >instance as a pre-shutdow

Re: Ever increasing startup times as data grow in persistent storage

Hi Zhenya, Thanks for confirming performing checkpoints more often will help here. I have established this configuration so will experiment with settings little. On a related note, is there any way to automatically trigger a checkpoint, for instance as a pre-shutdown activity? Checkpoints seem t

Re: Ever increasing startup times as data grow in persistent storage

We are currently using AI 2.8.1 with the c# client. On Wed, Jan 13, 2021 at 8:12 PM Kirill Tkalenko wrote: > Hello, Raymond! What version are you using? > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ > -- Raymond Wilson Solution Architect, Ci

Re: Ever increasing startup times as data grow in persistent storage

Hi Naveen, We currently have two data regions. A small one (for ingest), set to 128 Mb, and a larger one for requests (4Gb). We leave the checkpoint page buffer size at the default value, so this will be 1Gb for the larger region, and possibly 128Mb for the smaller region (if I recall the rules co

Re: Ever increasing startup times as data grow in persistent storage

Hello, Raymond! What version are you using? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ever increasing startup times as data grow in persistent storage

  >We have noticed that startup time for our server nodes has been slowly >increasing in time as the amount of data stored in the persistent store grows. >  >This appears to be closely related to recovery of WAL changes that were not >checkpointed at the time the node was stopped. >  >After e

Re: Ever increasing startup times as data grow in persistent storage

Hi Raymond Did you try checkpointPageBufferSize instead of time interval, we have used 24MB as checkpointPageBufferSize , working fine for us, we also have close to 12 TB of data and does take good 6 to 10 mts to bring up the node and become cluster active Regarding the no of partitions also, 128

Ever increasing startup times as data grow in persistent storage

We have noticed that startup time for our server nodes has been slowly increasing in time as the amount of data stored in the persistent store grows. This appears to be closely related to recovery of WAL changes that were not checkpointed at the time the node was stopped. After enabling debug log