JoaoJandre commented on issue #14030: URL: https://github.com/apache/cloudstack/issues/14030#issuecomment-5618649842
> [@JoaoJandre](https://github.com/JoaoJandre) what's the recovery process if it happens with anyone on 4.22, is it documented? > > If I understand the code correctly, Snapshot delete causes merge down which will copy all the diverged data to the base file. So the snapshot delete time is storage bandwidth dependent. If the diversion is large or their are active high throughput writes happening to the guest OS, the process can further slow down. Snapshot delete has the default timeout of 1 hour. Beyond which the management server job will be timed out but the underlying block job will still be running. This will cause the mentioned inconsistency between the volume.path in DB and on the storage. And this causes VM outage. > > So, this looks like very much possible as it depends on the guest writes and IO stress. If the fix is not possible by the time, at least we should document this possible issue and recovery procedure very clearly in the upcoming 4.22.2 release. Let me know your thoughts. @abh1sar, in the specific case you mentioned, where the user tried to delete the latest VM snapshot, which would change the volume path in the process, there are two possible routes: 1. Recreate the deltas manually, adjust the timeout, and ask ACS to delete the VM snapshot again. Since you just created the deltas, it should merge quickly. 2. Make a series of updates in the database to fix the metadata. I would rather users avoid this path. To recreate the deltas manually, assuming the VM is running, the easiest route is to look at the creation logs for the VM snapshot, where you will see something like: ``` 2026-09-10 09:05:52,571 INFO [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-2:[]) (logid:294a3af8) Creating disk-only VM snapshot for VM [i-2-341-VM] using parameters: snapshotXml [<domainsnapshot> <name>i-2-341-VM_VS_20260910120550</name> <memory snapshot='no'/> <disks> <disk name='hda' snapshot='external'> <source file='/mnt/a66f48f5-13c6-3676-b320-a15d32bbf32f/a6c733a7-1868-4887-a8c3-ef067ec940c0'/> </disk> </disks> </domainsnapshot>]; flags [148]. ``` You can save the XML to a file and use the below command to create the snapshot again. **I want to stress that this will create a snapshot at the top of the VM's volumes, so this should only be done for failures to delete the current VM snapshot, you can check which snapshot is the current in the GUI** ``` virsh snapshot-create --disk-only --atomic --no-metadata --domain <vm_name> --xmlfile <path_to_xml_file> ``` There are some other cases of situations that can occur. We can eventually document the most common ones. In any case, I strongly recommend adjusting the timeout to be longer than 1 hour, using the `commands.timeout` setting. Regarding extending the VM snapshot delete process to detect problems with earlier attempts and try to fix them, I don't know if I will have the time to create a PR for 4.22.2. But I'll try to open it when possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
