GitHub user midhunpjos closed a discussion: Unexpected Deletion of Valid Disks
by CloudStack + XCP-ng 8.2.1
We recently encountered an unusual issue with CloudStack, where some valid
disks assigned to stopped instances were unexpectedly deleted.
Upon investigation, logs from the XCP-ng Center indicated that the delete
requests for the VDIs originated from CloudStack. Interestingly, this coincided
with the execution of a volume cleanup thread within CloudStack.
However, during this cleanup process, CloudStack also issued a delete command
for a disk that was actually in a "Ready" state. we could not find any
corresponding delete request for that valid disk in the CloudStack management
server logs.
This raises concerns about a potential issue in the volume cleanup logic or
logging mechanisms within CloudStack.
Has anyone else experienced a similar scenario, or is aware of what might cause
such behavior?
audit logs from xen getting requests from cloudstack
`May 20 06:39:12 dhcld-az1c2-xcph01 xapi:
[20250520T00:54:12.925Z|audit||178683230 HTTPS 10.140.1.13->:::80|VDI.destroy
R:c81eccd0efa4|audit] ('trackid=47508e295ef7610e1fc86ae9a02090b2'
'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'ERROR:VDI_IN_USE: [
OpaqueRef:03e5cebb-c885-4ed9-a7db-27a40bcbee6d; destroy ]' 'API' 'VDI.destroy'
(('self' 'ROOT-546' 'e9253423-48eb-471d-be43-4e0f46d44593'
'OpaqueRef:03e5cebb-c885-4ed9-a7db-27a40bcbee6d')))
May 20 06:42:18 dhcld-az1c2-xcph01 xapi:
[20250520T00:57:18.072Z|audit||178683230 HTTPS 10.140.1.13->:::80|VDI.destroy
R:a5223d94b8f5|audit] ('trackid=47508e295ef7610e1fc86ae9a02090b2'
'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'VDI.destroy' (('self'
'ROOT-12558' '4d7f2a7d-637e-446c-aa5f-57c867d7848a'
'OpaqueRef:fe55984f-df0f-4a92-ab57-2f381f67c669')))
May 20 06:42:18 dhcld-az1c2-xcph01 xapi:
[20250520T00:57:18.107Z|audit||178683230 HTTPS 10.140.1.13->:::80|VDI.destroy
R:70523a4fd400|audit] ('trackid=47508e295ef7610e1fc86ae9a02090b2'
'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'ERROR:VDI_IN_USE: [
OpaqueRef:fcb01b3f-98a5-4d13-a1ac-fd5337fe2ec0; destroy ]' 'API' 'VDI.destroy'
(('self' 'ROOT-12776' '6b03ea00-657b-4567-8063-5b8370e6bfda'
'OpaqueRef:fcb01b3f-98a5-4d13-a1ac-fd5337fe2ec0')))
May 20 06:42:19 dhcld-az1c2-xcph01 xapi:
[20250520T00:57:19.883Z|audit||178683230 HTTPS 10.140.1.13->:::80|VDI.destroy
R:ef1a9d18f7d6|audit] ('trackid=47508e295ef7610e1fc86ae9a02090b2'
'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'ERROR:VDI_IN_USE: [
OpaqueRef:aa7e8124-a669-4edc-b8b8-1610b395b7cd; destroy ]' 'API' 'VDI.destroy'
(('self' 'ROOT-13929' 'a0b4f644-bc40-4b82-9f2a-dafcaf6db0ae'
'OpaqueRef:aa7e8124-a669-4edc-b8b8-1610b395b7cd')))
May 20 11:41:28 dhcld-az1c2-xcph01 xapi:
[20250520T05:56:28.476Z|audit||178964351 HTTPS 10.140.1.13->:::80|VDI.destroy
R:92b87bb5e694|audit] ('trackid=47508e295ef7610e1fc86ae9a02090b2'
'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'VDI.destroy' (('self'
'ROOT-14400' 'e3bad622-578e-4d29-924e-47b66a6439d6'
'OpaqueRef:e8db1761-e15d-45c3-891e-d7766a867cee')))
May 20 12:05:42 dhcld-az1c2-xcph01 xapi:
[20250520T06:20:42.998Z|audit||178983321 HTTPS 10.140.1.13->:::80|VDI.destroy
R:ef9634e5414d|audit] ('trackid=47508e295ef7610e1fc86ae9a02090b2'
'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'VDI.destroy' (('self'
'ROOT-12820' '0f707aa3-d858-4231-b6dc-e92dfe910ba3'
'OpaqueRef:00b6bd67-21a5-43d9-ab86-2442ad13516f')`
Cleanup log reference at the time of incident:
`2025-05-20 06:39:12,873 DEBUG [c.c.a.t.Request]
(StorageManager-Scavenger-1:ctx-6ed39762) (logid:69e084ff) Seq
32-4469822630165282172: Sending { Cmd , MgmtId: 108462341209969, via:
32(DHCLD-AZ1C2-XCPH04), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"649ab11c-ff50-4a81-8e8b-7fe1110a8608","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"PUREC-DHCLD-AZ1C2-VOL1","name":"PUREC-DHCLD-AZ1C2-VOL1","id":"5","poolType":"PreSetup","host":"localhost","path":"/PUREC-DHCLD-AZ1C2-VOL1","port":"0","url":"PreSetup://localhost/PUREC-DHCLD-AZ1C2-VOL1/?ROLE=Primary&STOREUUID=PUREC-DHCLD-AZ1C2-VOL1","isManaged":"false"}},"name":"ROOT-546","size":"(4.89
GB)
5253365760","path":"e9253423-48eb-471d-be43-4e0f46d44593","volumeId":"2164","accountId":"74","format":"VHD","provisioningType":"THIN","poolId":"5","id":"2164","deviceId":"0","hypervisorType":"XenServer"
,"directDownload":"false","deployAsIs":"false","followRedirects":"true"}},"wait":"0","bypassHostMaintenance":"false"}}]
}
2025-05-20 06:39:12,875 DEBUG [c.c.a.t.Request]
(StorageManager-Scavenger-1:ctx-6ed39762) (logid:69e084ff) Seq
32-4469822630165282172: Executing: { Cmd , MgmtId: 108462341209969, via:
32(DHCLD-AZ1C2-XCPH04), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"649ab11c-ff50-4a81-8e8b-7fe1110a8608","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"PUREC-DHCLD-AZ1C2-VOL1","name":"PUREC-DHCLD-AZ1C2-VOL1","id":"5","poolType":"PreSetup","host":"localhost","path":"/PUREC-DHCLD-AZ1C2-VOL1","port":"0","url":"PreSetup://localhost/PUREC-DHCLD-AZ1C2-VOL1/?ROLE=Primary&STOREUUID=PUREC-DHCLD-AZ1C2-VOL1","isManaged":"false"}},"name":"ROOT-546","size":"(4.89
GB)
5253365760","path":"e9253423-48eb-471d-be43-4e0f46d44593","volumeId":"2164","accountId":"74","format":"VHD","provisioningType":"THIN","poolId":"5","id":"2164","deviceId":"0","hypervisorType":"XenServe
r","directDownload":"false","deployAsIs":"false","followRedirects":"true"}},"wait":"0","bypassHostMaintenance":"false"}}]
}`
GitHub link: https://github.com/apache/cloudstack/discussions/10911
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
