Hi Harikrishna, I've done some more testing testing just now. Please see below the answers to your questions/comments:
----- Original Message ----- > From: "Harikrishna Patnala" <harikrishna.patn...@shapeblue.com> > To: "users" <users@cloudstack.apache.org> > Sent: Tuesday, 8 June, 2021 03:33:57 > Subject: Re: Snapshots are not working after upgrading to 4.15.0 > Hi Andrei, > > Can you check the following things and let us know? > > > 1. Can you try creating a new volume and then create snapshot of that, to > check > if this an issue with old entries AM: I've done testing with a newly created image which was attached to a vm. The same problem happens with new images as well as old images. Please see the link below for the management server log. https://zerobin.net/?5781e4b65d9e3605#+GtIC7JBtp70Q0cw65cypJDiSyba/r/JldRsAyOI8l4= > 2. For the snapshots which are failing can you check if you are seeing any > error messages like this "Can't find an image storage in zone with less > than". > This is to check if secondary storage free space check failed. AM: I do not see any such message in the logs. tried grepping the logs for a couple of weeks and nothing comes up. Having said this, the secondary storage is about 90.6% full. > 3. For the snapshots which are failing and if it is delta snapshot can you > check if its parent's snapshot entry exists in "snapshot_store_ref" table > with > 'parent_snapshot_id' of the current snapshot with 'store_role' "Image". This > is > to find the secondary storage where the parent snapshot backup is located. > AM: all snapshots are failing, not just selective few. Some volumes are brand new, as I've indicated above, others do have previous snapshots. I only have a single secondary storage, so all snaps should be in one place. > Regards, > Harikrishna > ________________________________ > From: Andrei Mikhailovsky <and...@arhont.com.INVALID> > Sent: Monday, June 7, 2021 7:00 PM > To: users <users@cloudstack.apache.org> > Subject: Snapshots are not working after upgrading to 4.15.0 > > Hello everyone, > > I am having an issue with volume snapshots since I've upgraded to 4.15.0. None > of the volumes are being snapshotted regardless if the snapshot is initiated > manually or from the schedule. The strange thing is that if I manually take > the > snapshot, the GUI shows Success status, but the Storage>Snapshots show an > Error > status. Here is what I see in the management server logs: > > 2021-06-07 13:55:20,022 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (Work-Job-Executor-81:ctx-08dd4222 job-86141/job-86143) (logid:be34ce01) Done > executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-86143 > 2021-06-07 13:55:20,024 INFO [o.a.c.f.j.i.AsyncJobMonitor] > (Work-Job-Executor-81:ctx-08dd4222 job-86141/job-86143) (logid:be34ce01) > Remove > job-86143 from job monitoring > 2021-06-07 13:55:20,094 DEBUG [o.a.c.s.s.SnapshotServiceImpl] > (BackupSnapshotTask-3:ctx-744796da) (logid:607dbb0e) Failed to copy snapshot > com.cloud.utils.exception.CloudRuntimeException: can not find an image stores > at > org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:271) > at > org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:171) > at > com.cloud.storage.snapshot.SnapshotManagerImpl$BackupSnapshotTask.runInContext(SnapshotManagerImpl.java:1238) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45) > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > 2021-06-07 13:55:20,152 DEBUG [c.c.s.s.SnapshotManagerImpl] > (BackupSnapshotTask-3:ctx-744796da) (logid:607dbb0e) Backing up of snapshot > failed, for snapshot with ID 53531, left with 2 more attempts > > > I've checked and the Secondary storage is configured and visible in the GUI. I > can also mount it manually from the management server and a couple of host > servers that I've tested. In addition, I can successfully upload an ISO image > and that registers just fine and I can create new VMs using the newly uploaded > ISO image. > > I've had no such problems with 4.13.x ACS, so the issue seems to have been > introduced after doing the upgrade to 4.15.0. > > Could you please let me know how do I fix the issue? > > Cheers > > andrei
