Thanks Slavka, I will attempt to clean up some old snaps and see if the backups start working again.
Andrei ----- Original Message ----- > From: "Slavka Peleva" <slav...@storpool.com.INVALID> > To: "users" <users@cloudstack.apache.org> > Sent: Monday, 14 June, 2021 13:28:01 > Subject: Re: Snapshots are not working after upgrading to 4.15.0 > Hi, Andrei, > > After a quick check of the code, I guess that the backup of the snapshot > fails because the secondary storage space should be less than 90%, and you > pointed that yours is 90.6% full. > > Regards, > Slavka > > On Mon, Jun 14, 2021 at 1:10 PM Andrei Mikhailovsky > <and...@arhont.com.invalid> wrote: > >> Hi Harikrishna, >> >> I've done some more testing testing just now. Please see below the answers >> to your questions/comments: >> >> >> >> ----- Original Message ----- >> > From: "Harikrishna Patnala" <harikrishna.patn...@shapeblue.com> >> > To: "users" <users@cloudstack.apache.org> >> > Sent: Tuesday, 8 June, 2021 03:33:57 >> > Subject: Re: Snapshots are not working after upgrading to 4.15.0 >> >> > Hi Andrei, >> > >> > Can you check the following things and let us know? >> > >> > >> > 1. Can you try creating a new volume and then create snapshot of that, >> to check >> > if this an issue with old entries >> >> AM: I've done testing with a newly created image which was attached to a >> vm. The same problem happens with new images as well as old images. Please >> see the link below for the management server log. >> >> >> https://zerobin.net/?5781e4b65d9e3605#+GtIC7JBtp70Q0cw65cypJDiSyba/r/JldRsAyOI8l4= >> >> >> > 2. For the snapshots which are failing can you check if you are seeing >> any >> > error messages like this "Can't find an image storage in zone with less >> than". >> > This is to check if secondary storage free space check failed. >> >> >> AM: I do not see any such message in the logs. tried grepping the logs for >> a couple of weeks and nothing comes up. Having said this, the secondary >> storage is about 90.6% full. >> >> >> > 3. For the snapshots which are failing and if it is delta snapshot can >> you >> > check if its parent's snapshot entry exists in "snapshot_store_ref" >> table with >> > 'parent_snapshot_id' of the current snapshot with 'store_role' "Image". >> This is >> > to find the secondary storage where the parent snapshot backup is >> located. >> > >> >> AM: all snapshots are failing, not just selective few. Some volumes are >> brand new, as I've indicated above, others do have previous snapshots. I >> only have a single secondary storage, so all snaps should be in one place. >> >> >> >> >> > Regards, >> > Harikrishna >> > ________________________________ >> > From: Andrei Mikhailovsky <and...@arhont.com.INVALID> >> > Sent: Monday, June 7, 2021 7:00 PM >> > To: users <users@cloudstack.apache.org> >> > Subject: Snapshots are not working after upgrading to 4.15.0 >> > >> > Hello everyone, >> > >> > I am having an issue with volume snapshots since I've upgraded to >> 4.15.0. None >> > of the volumes are being snapshotted regardless if the snapshot is >> initiated >> > manually or from the schedule. The strange thing is that if I manually >> take the >> > snapshot, the GUI shows Success status, but the Storage>Snapshots show >> an Error >> > status. Here is what I see in the management server logs: >> > >> > 2021-06-07 13:55:20,022 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> > (Work-Job-Executor-81:ctx-08dd4222 job-86141/job-86143) (logid:be34ce01) >> Done >> > executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-86143 >> > 2021-06-07 13:55:20,024 INFO [o.a.c.f.j.i.AsyncJobMonitor] >> > (Work-Job-Executor-81:ctx-08dd4222 job-86141/job-86143) (logid:be34ce01) >> Remove >> > job-86143 from job monitoring >> > 2021-06-07 13:55:20,094 DEBUG [o.a.c.s.s.SnapshotServiceImpl] >> > (BackupSnapshotTask-3:ctx-744796da) (logid:607dbb0e) Failed to copy >> snapshot >> > com.cloud.utils.exception.CloudRuntimeException: can not find an image >> stores >> > at >> > >> org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:271) >> > at >> > >> org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:171) >> > at >> > >> com.cloud.storage.snapshot.SnapshotManagerImpl$BackupSnapshotTask.runInContext(SnapshotManagerImpl.java:1238) >> > at >> > >> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48) >> > at >> > >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) >> > at >> > >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) >> > at >> > >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) >> > at >> > >> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45) >> > at >> > >> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) >> > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) >> > at >> > >> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) >> > at >> > >> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) >> > at >> > >> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) >> > at java.base/java.lang.Thread.run(Thread.java:829) >> > 2021-06-07 13:55:20,152 DEBUG [c.c.s.s.SnapshotManagerImpl] >> > (BackupSnapshotTask-3:ctx-744796da) (logid:607dbb0e) Backing up of >> snapshot >> > failed, for snapshot with ID 53531, left with 2 more attempts >> > >> > >> > I've checked and the Secondary storage is configured and visible in the >> GUI. I >> > can also mount it manually from the management server and a couple of >> host >> > servers that I've tested. In addition, I can successfully upload an ISO >> image >> > and that registers just fine and I can create new VMs using the newly >> uploaded >> > ISO image. >> > >> > I've had no such problems with 4.13.x ACS, so the issue seems to have >> been >> > introduced after doing the upgrade to 4.15.0. >> > >> > Could you please let me know how do I fix the issue? >> > >> > Cheers >> > >> > andrei
