Re: Possible WAL corruption on running system during K8s update

2023-07-19 Thread Raymond Wilson
As a follow up to this: We tried removing both those in the walstore and walarchive. Problem is that somewhere there is a checkpoint that says its up to wal index 2414...yet we only have 2413...2412...etc We need to find where it stores this checkpoint index and change it, it seems. On Wed, 19

Re: Possible WAL corruption on running system during K8s update

2023-07-18 Thread Raymond Wilson
Hi Alex, Here is the log from the Ignite startup. It's fairly short but shows everything I think: 2023-07-17 22:38:55,061 [1] DBG [ImmutableCacheComputeServer] Starting Ignite.NET 2.15.0.23172 2023-07-17 22:38:55,065 [1] DBG [ImmutableCacheComputeServer] 2023-07-17 22:38:55,068 [1] DBG

Re: Possible WAL corruption on running system during K8s update

2023-07-18 Thread Raymond Wilson
Hi Alex, We are using Ignite v2.15. I will track down the additional log information and reply on this thread. Raymond. On Wed, Jul 19, 2023 at 2:55 AM Alex Plehanov wrote: > Hello, > > Which Ignite version do you use? > Please share exception details after "Exception during start

Re: Possible WAL corruption on running system during K8s update

2023-07-18 Thread Alex Plehanov
Hello, Which Ignite version do you use? Please share exception details after "Exception during start processors, node will be stopped and close connections" (there should be a reason in the log, why the page delta can't be applied). вт, 18 июл. 2023 г. в 05:05, Raymond Wilson : > Hi, > > We run

Possible WAL corruption on running system during K8s update

2023-07-17 Thread Raymond Wilson
Hi, We run a dev/alpha stack of our application in Azure Kubernetes. Persistent storage is contained in Azure Files NAS storage volumes, one per server node. We ran an upgrade of Kubernetes today (from 1.24.9 to 1.26.3). During the update various pods were stopped and restarted as is normal for