Hello Andrei,
As yourself noticed, ACS has a hardcoded threshold for secondary
storages. In cases that the secondary storage has large capacities, 10%
can mean a lot of storage. There is an open PR
(https://github.com/apache/cloudstack/pull/4790) that externalize this
threshold to allow operators decide how much they need. Also, the logs
of secondary storage management were improved with PR
https://github.com/apache/cloudstack/pull/4955.
With respect to KVM snapshots, volume snapshots are taken in a quite
peculiar way. Instead of taking volume snapshots directly, ACS takes a
full snapshot of the VM, which may cause freeze on it (VM) due to memory
snapshot, and then extracts the disk from the VM snapshot. Due to it, it
was open an issue https://github.com/apache/cloudstack/issues/5124 to
discuss a new workflow to snapshots on KVM.
I am already implementing the solution for the issue 5124, and to
improve this whole snapshot process for KVM; However, it is a complex
and long standing job. As soon as we have something, I would appreciate
to receive some feedback from you.
Regards,
Guto
On 2021/06/16 16:15:51, Andrei Mikhailovsky wrote:
> Hello,>
>
> I've done some more investigation and indeed, the snapshots were not
taken because the secondary storage was over 90% used. I have started
cleaning some of the older volumes and noticed another problem. After
removing snapshots, they do not seem to be removed from the secondary
storage. I've removed all snapshots over 24 hours ago and it looks like
the disk space hasn't been freed up at all.>
>
> Looks like there are issues with snapshotting function after all.>
>
> Andrei>
>
>
>
> ----- Original Message ----->
> > From: "Harikrishna Patnala" >
> > To: "users" >
> > Sent: Tuesday, 8 June, 2021 03:33:57>
> > Subject: Re: Snapshots are not working after upgrading to 4.15.0>
>
> > Hi Andrei,>
> > >
> > Can you check the following things and let us know?>
> > >
> > >
> > 1. Can you try creating a new volume and then create snapshot of
that, to check>
> > if this an issue with old entries>
> > 2. For the snapshots which are failing can you check if you are
seeing any>
> > error messages like this "Can't find an image storage in zone with
less than".>
> > This is to check if secondary storage free space check failed.>
> > 3. For the snapshots which are failing and if it is delta snapshot
can you>
> > check if its parent's snapshot entry exists in "snapshot_store_ref"
table with>
> > 'parent_snapshot_id' of the current snapshot with 'store_role'
"Image". This is>
> > to find the secondary storage where the parent snapshot backup is
located.>
> > >
> > Regards,>
> > Harikrishna>
> > ________________________________>
> > From: Andrei Mikhailovsky >
> > Sent: Monday, June 7, 2021 7:00 PM>
> > To: users >
> > Subject: Snapshots are not working after upgrading to 4.15.0>
> > >
> > Hello everyone,>
> > >
> > I am having an issue with volume snapshots since I've upgraded to
4.15.0. None>
> > of the volumes are being snapshotted regardless if the snapshot is
initiated>
> > manually or from the schedule. The strange thing is that if I
manually take the>
> > snapshot, the GUI shows Success status, but the Storage>Snapshots
show an Error>
> > status. Here is what I see in the management server logs:>
> > >
> > 2021-06-07 13:55:20,022 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]>
> > (Work-Job-Executor-81:ctx-08dd4222 job-86141/job-86143)
(logid:be34ce01) Done>
> > executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-86143>
> > 2021-06-07 13:55:20,024 INFO [o.a.c.f.j.i.AsyncJobMonitor]>
> > (Work-Job-Executor-81:ctx-08dd4222 job-86141/job-86143)
(logid:be34ce01) Remove>
> > job-86143 from job monitoring>
> > 2021-06-07 13:55:20,094 DEBUG [o.a.c.s.s.SnapshotServiceImpl]>
> > (BackupSnapshotTask-3:ctx-744796da) (logid:607dbb0e) Failed to copy
snapshot>
> > com.cloud.utils.exception.CloudRuntimeException: can not find an
image stores>
> > at>
> >
org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:271)>
> > at>
> >
org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:171)>
> > at>
> >
com.cloud.storage.snapshot.SnapshotManagerImpl$BackupSnapshotTask.runInContext(SnapshotManagerImpl.java:1238)>
> > at>
> >
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)>
> > at>
> >
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)>
> > at>
> >
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)>
> > at>
> >
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)>
> > at>
> >
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)>
> > at>
> >
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)>
> > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)>
> > at>
> >
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)>
> > at>
> >
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)>
> > at>
> >
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)>
> > at java.base/java.lang.Thread.run(Thread.java:829)>
> > 2021-06-07 13:55:20,152 DEBUG [c.c.s.s.SnapshotManagerImpl]>
> > (BackupSnapshotTask-3:ctx-744796da) (logid:607dbb0e) Backing up of
snapshot>
> > failed, for snapshot with ID 53531, left with 2 more attempts>
> > >
> > >
> > I've checked and the Secondary storage is configured and visible in
the GUI. I>
> > can also mount it manually from the management server and a couple
of host>
> > servers that I've tested. In addition, I can successfully upload an
ISO image>
> > and that registers just fine and I can create new VMs using the
newly uploaded>
> > ISO image.>
> > >
> > I've had no such problems with 4.13.x ACS, so the issue seems to
have been>
> > introduced after doing the upgrade to 4.15.0.>
> > >
> > Could you please let me know how do I fix the issue?>
> > >
> > Cheers>
> > >
> > andrei>
>
