Hi Andrei, Can you check the following things and let us know?
1. Can you try creating a new volume and then create snapshot of that, to check if this an issue with old entries 2. For the snapshots which are failing can you check if you are seeing any error messages like this "Can't find an image storage in zone with less than". This is to check if secondary storage free space check failed. 3. For the snapshots which are failing and if it is delta snapshot can you check if its parent's snapshot entry exists in "snapshot_store_ref" table with 'parent_snapshot_id' of the current snapshot with 'store_role' "Image". This is to find the secondary storage where the parent snapshot backup is located. Regards, Harikrishna ________________________________ From: Andrei Mikhailovsky <and...@arhont.com.INVALID> Sent: Monday, June 7, 2021 7:00 PM To: users <users@cloudstack.apache.org> Subject: Snapshots are not working after upgrading to 4.15.0 Hello everyone, I am having an issue with volume snapshots since I've upgraded to 4.15.0. None of the volumes are being snapshotted regardless if the snapshot is initiated manually or from the schedule. The strange thing is that if I manually take the snapshot, the GUI shows Success status, but the Storage>Snapshots show an Error status. Here is what I see in the management server logs: 2021-06-07 13:55:20,022 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-81:ctx-08dd4222 job-86141/job-86143) (logid:be34ce01) Done executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-86143 2021-06-07 13:55:20,024 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-81:ctx-08dd4222 job-86141/job-86143) (logid:be34ce01) Remove job-86143 from job monitoring 2021-06-07 13:55:20,094 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (BackupSnapshotTask-3:ctx-744796da) (logid:607dbb0e) Failed to copy snapshot com.cloud.utils.exception.CloudRuntimeException: can not find an image stores at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:271) at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:171) at com.cloud.storage.snapshot.SnapshotManagerImpl$BackupSnapshotTask.runInContext(SnapshotManagerImpl.java:1238) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2021-06-07 13:55:20,152 DEBUG [c.c.s.s.SnapshotManagerImpl] (BackupSnapshotTask-3:ctx-744796da) (logid:607dbb0e) Backing up of snapshot failed, for snapshot with ID 53531, left with 2 more attempts I've checked and the Secondary storage is configured and visible in the GUI. I can also mount it manually from the management server and a couple of host servers that I've tested. In addition, I can successfully upload an ISO image and that registers just fine and I can create new VMs using the newly uploaded ISO image. I've had no such problems with 4.13.x ACS, so the issue seems to have been introduced after doing the upgrade to 4.15.0. Could you please let me know how do I fix the issue? Cheers andrei
