akoskuczi-bw opened a new issue, #13972:
URL: https://github.com/apache/cloudstack/issues/13972
### problem
When using KVM with RBD primary storage and
kvm.vmstoragesnapshot.enabled=true, an Instance Snapshot that fails during
creation before the underlying storage snapshot is created cannot be deleted
while the VM is stopped.
The failed Instance Snapshot remains in CloudStack in an error state. When
CloudStack later attempts to delete the snapshot, the snapshot is handled by
DefaultVMSnapshotStrategy instead of StorageVMSnapshotStrategy.
For a stopped VM backed by RAW/RBD storage, DeleteVMSnapshotCommand then
fails because the libvirt domain does not exist:
Delete Instance Snapshot failed due to org.libvirt.LibvirtException:
Domain not found: no domain with matching name 'i-2-43-VM'
Starting the VM allows the same failed Instance Snapshot to be deleted
successfully.
This appears to be a RAW/RBD variant of #11673. PR #11687 fixes a similar
case for QCOW2/stopped VMs, but the fallback implemented there only handles
QCOW2 volumes.
### versions
Apache CloudStack: 4.22.1
Hypervisor: KVM
Primary storage: Ceph RBD
Volume format: RAW
Instance Snapshot type: Disk
kvm.vmstoragesnapshot.enabled=true
### The steps to reproduce the bug
Deploy a KVM VM whose ROOT volume is stored on Ceph RBD.
Enable: kvm.vmstoragesnapshot.enabled=true
Run the VM without a working QEMU guest agent.
Create a disk-only Instance Snapshot with VM quiescing enabled.
Snapshot creation fails during the freeze operation:
StorageVMSnapshotStrategy
Failed to frozen VM - i-2-43-VM due to Guest agent is not responding:
QEMU guest agent is not connected
Failed to create Instance Snapshot
The Instance Snapshot remains in CloudStack in a failed/error state.
Stop the VM.
Try to delete the failed Instance Snapshot, or destroy the VM so that
CloudStack runs VmWorkDeleteAllVMSnapshots.
**Actual result**
CloudStack does not use StorageVMSnapshotStrategy to clean up the failed
snapshot.
The deletion is handled by DefaultVMSnapshotStrategy:
KvmFileBasedStorageVmSnapshotStrategy ... cannot handle [...]
...
DeleteVMSnapshotCommand
...
DefaultVMSnapshotStrategy
The command sent to the KVM agent contains an RBD/RAW ROOT volume:
"poolType":"RBD"
"format":"RAW"
"volumeType":"ROOT"
"vmName":"i-2-43-VM"
The agent then fails because the stopped VM has no libvirt domain:
DeleteVMSnapshotAnswer:
result=false
Delete Instance Snapshot failed due to org.libvirt.LibvirtException:
Domain not found: no domain with matching name 'i-2-43-VM'
The VM destroy operation also fails because
orchestrateDeleteAllVMSnapshots() cannot remove the failed Instance Snapshot:
CloudRuntimeException:
Failed to destroy vm with specified vmId
Caused by:
Delete Instance Snapshot ... failed due to
org.libvirt.LibvirtException:
Domain not found: no domain with matching name 'i-2-43-VM'
**Expected result**
A failed storage-based Instance Snapshot whose creation stopped before any
underlying storage snapshot was created should be removable while the VM is
stopped.
The cleanup should not require a libvirt domain to exist.
In particular, if:
the Instance Snapshot is in Error state,
it was created using the storage-based KVM snapshot strategy,
there are no kvmStorageSnapshot details / underlying volume snapshots to
remove,
then deleting the Instance Snapshot should clean up the CloudStack metadata
and return success.
VM destruction should also not be blocked by such a failed Instance Snapshot.
Additional observations
The Instance Snapshot creation originally uses StorageVMSnapshotStrategy:
StorageVMSnapshotStrategy
Freeze of virtual machine took 0 milliseconds.
Failed to create Instance Snapshot ...
Could not freeze Instance.
QEMU guest agent is not connected
The failure happens during the VM freeze operation, before the actual RBD
snapshot creation.
As a result, there is no underlying RBD snapshot associated with the failed
Instance Snapshot.
During a later delete, however, CloudStack falls back to
DefaultVMSnapshotStrategy.
For stopped RAW/RBD VMs, the KVM LibvirtDeleteVMSnapshotCommandWrapper
cannot use the QCOW2 offline fallback, because the ROOT volume format is RAW.
**Workaround**
Starting the VM creates the libvirt domain again.
After starting the VM, deleting the exact same failed Instance Snapshot
succeeds:
DeleteVMSnapshotAnswer: result=true
No RBD snapshot is actually being deleted in this case. The presence of the
libvirt domain merely allows the wrapper to determine that the requested
libvirt snapshot does not exist and return success.
**The current workaround is therefore:**
Start VM
- delete failed Instance Snapshot
- stop/destroy VM
This should not be necessary for a failed storage-based Instance Snapshot.
### What to do about it?
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
