rozaqi opened a new issue, #13054:
URL: https://github.com/apache/cloudstack/issues/13054
### problem
We are experiencing a VM migration failure in CloudStack where the migration
process fails with a "Domain not found" error on the source host.
During migration, CloudStack reports "Migration was unsuccessful" without
detailed error information on the management server. However, on the source
host (KVM/libvirt), the agent fails to locate the VM domain during execution of
the MigrateCommand.
**log management-server
`2026-04-21 15:21:29,314 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(qtp652176954-26811:[ctx-82bf6948, ctx-c49af74b]) (logid:8f7aa9da) submit async
job-303869, details: AsyncJob
{"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd","cmdInfo":"{\"virtualmachineid\":\"bf5f467f-c17a-4e6e-b3e2-1beb97941324\",\"response\":\"json\",\"ctxUserId\":\"2\",\"sessionkey\":\"V3TXcG6Qnbi2kWSZ5DM5AT2x088\",\"hostid\":\"2995de01-4d98-49c9-b379-ff8adf0f0722\",\"httpmethod\":\"POST\",\"ctxStartEventId\":\"6185591\",\"ctxDetails\":\"{\\\"interface
com.cloud.host.Host\\\":\\\"2995de01-4d98-49c9-b379-ff8adf0f0722\\\",\\\"interface
com.cloud.vm.VirtualMachine\\\":\\\"bf5f467f-c17a-4e6e-b3e2-1beb97941324\\\"}\",\"ctxAccountId\":\"2\",\"cmdEventType\":\"VM.MIGRATE\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":303869,"initMsid":345050754440,"instanceId":4127,"instanceType":"VirtualMachine","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":nul
l,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"569f38b1-bc6c-422c-a4ee-5fc8e3ebb7c3"}
2026-04-21 15:21:29,316 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5]
(API-Job-Executor-57:[ctx-964ae7d1, job-303869]) (logid:569f38b1) Executing
AsyncJob
{"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd","cmdInfo":"{\"virtualmachineid\":\"bf5f467f-c17a-4e6e-b3e2-1beb97941324\",\"response\":\"json\",\"ctxUserId\":\"2\",\"sessionkey\":\"V3TXcG6Qnbi2kWSZ5DM5AT2x088\",\"hostid\":\"2995de01-4d98-49c9-b379-ff8adf0f0722\",\"httpmethod\":\"POST\",\"ctxStartEventId\":\"6185591\",\"ctxDetails\":\"{\\\"interface
com.cloud.host.Host\\\":\\\"2995de01-4d98-49c9-b379-ff8adf0f0722\\\",\\\"interface
com.cloud.vm.VirtualMachine\\\":\\\"bf5f467f-c17a-4e6e-b3e2-1beb97941324\\\"}\",\"ctxAccountId\":\"2\",\"cmdEventType\":\"VM.MIGRATE\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":303869,"initMsid":345050754440,"instanceId":4127,"instanceType":"VirtualMachine","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status
":"IN_PROGRESS","userId":2,"uuid":"569f38b1-bc6c-422c-a4ee-5fc8e3ebb7c3"}
2026-04-21 15:21:29,353 DEBUG [c.c.v.UserVmManagerImpl]
(API-Job-Executor-57:[ctx-964ae7d1, job-303869, ctx-ff0b825e]) (logid:569f38b1)
Checking if there are any ongoing snapshots volumes associated with VM VM
instance
{"id":4127,"instanceName":"i-2-4127-VM","state":"Running","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"}
2026-04-21 15:21:29,354 DEBUG [c.c.v.UserVmManagerImpl]
(API-Job-Executor-57:[ctx-964ae7d1, job-303869, ctx-ff0b825e]) (logid:569f38b1)
Found 1 no. of volumes of type null for vm with VM ID VM instance
{"id":4127,"instanceName":"i-2-4127-VM","state":"Running","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"}
2026-04-21 15:21:29,355 DEBUG [c.c.v.UserVmManagerImpl]
(API-Job-Executor-57:[ctx-964ae7d1, job-303869, ctx-ff0b825e]) (logid:569f38b1)
The count of ongoing Snapshots for VM VM instance
{"id":4127,"instanceName":"i-2-4127-VM","state":"Running","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"}
and disk type null is 0
2026-04-21 15:21:29,355 DEBUG [c.c.v.UserVmManagerImpl]
(API-Job-Executor-57:[ctx-964ae7d1, job-303869, ctx-ff0b825e]) (logid:569f38b1)
Found no ongoing snapshots on volumes associated with the vm VM instance
{"id":4127,"instanceName":"i-2-4127-VM","state":"Running","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"}
2026-04-21 15:21:30,999 INFO [c.c.v.ClusteredVirtualMachineManagerImpl]
(Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07])
(logid:569f38b1) Migrating VM instance
{"id":4127,"instanceName":"i-2-4127-VM","state":"Running","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"}
to
Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
: Dest[Zone(2)-Pod(2)-Cluster(6)-Host(324)-Storage()]
2026-04-21 15:21:31,022 DEBUG [o.a.c.e.o.VolumeOrchestrator]
(Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07])
(logid:569f38b1) Preparing to migrate [1] volumes for VM [VM instance
{"id":4127,"instanceName":"i-2-4127-VM","state":"Running","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"}].
2026-04-21 15:21:31,106 DEBUG [c.c.a.m.ClusteredAgentAttache]
(Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07])
(logid:569f38b1) Seq 446-3275242829005214246: Forwarding Seq
324-3275242829005214246: { Cmd , MgmtId: 345050754440, via: 324(com11), Ver:
v1, Flags: 100111,
[{"com.cloud.agent.api.PrepareForMigrationCommand":{"vm":{"id":"4127","name":"i-2-4127-VM","state":"Running","type":"User","cpus":"2","minSpeed":"400","maxSpeed":"2000","minRam":"(2.00
GB) 2147483648","maxRam":"(2.00 GB) 2147483648","arch":"x86_64","os":"Other
Ubuntu (64-bit)","platformEmulator":"Other
Ubuntu","bootArgs":"","enableHA":"true","limitCpuUse":"false","enableDynamicallyScaleVm":"true","details":{"cpuOvercommitRatio":"5.0","rootdisksize":"20","rootDiskController":"osdefault"},"uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324","enterHardwareSetup":"false","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"c9e66d60-139c-4e90-ab9a-f06d6486b63d","volumeType":"ROOT
","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"ssd;36b6ca0e-eb0f-4d11-9491-937d592f36e9","name":"SSD","id":"66","poolType":"StorPool","host":"n/a","path":"/dev/storpool-byid/","port":"0","url":"StorPool://n/a/dev/storpool-byid//?ROLE=Primary&STOREUUID=ssd;36b6ca0e-eb0f-4d11-9491-937d592f36e9","isManaged":"false"}},"name":"ROOT-4127","size":"(20.00
GB)
21474836480","path":"/dev/storpool-byid/n4en.b.bgxk","volumeId":"15599","vmName":"i-2-4127-VM","accountId":"2","format":"QCOW2","provisioningType":"THIN","poolId":"66","id":"15599","deviceId":"0","bytesReadRate":"(0
bytes) 0","bytesWriteRate":"(0 bytes) 0","iopsReadRate":"(0 bytes)
0","iopsWriteRate":"(0 bytes)
0","cacheMode":"NONE","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false","checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":"true"}},"diskSeq":"0","path":"/dev/storpool-byid/n4en.b.bgxk","type":"ROOT","details":{"storageHost":"n/a","managed":"false","storagePort"
:"0","protocoltype":"StorPool","storage.pool.disk.wait":"60","volumeSize":"(20.00
GB)
21474836480"}},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":"0","format":"ISO","accountId":"0","hvm":"false","bootable":"false","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"diskSeq":"3","type":"ISO"}],"nics":[{"deviceId":"0","networkRateMbps":"-1","defaultNic":"true","pxeDisable":"false","nicUuid":"b1aee509-ad6b-4dd5-ade5-c1d44f995615","details":{"MacLearning":"false","PromiscuousMode":"false","internetProtocol":"IPv4","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":"false","networkId":"783","networkSegmentName":"D1-A2-Z2-V1124-S783","uuid":"958d365a-d975-419f-830a-b13b7f29389d","ip":"192.168.10.125","netmask":"255.255.255.0","gateway":"192.168.10.1","mac":"02:02:03:0f:00:3c","dns1":"8.8.8.8","broadcastType":"Vxlan","type":"Guest","broadcastUri":"vxlan://17742","isolationUri":"vxlan://17742","securityGroupEnabled":"false","name"
:"cloudbr2"}],"vcpuMaxLimit":"2","configDriveLabel":"config-2","configDriveLocation":"SECONDARY","guestOsDetails":{},"extraConfig":{},"networkIdToNetworkNameMap":{"783":"D1-A2-Z2-V1124-S783"}},"rollback":"false","wait":"0","bypassHostMaintenance":"false"}}]
} to 345050719314
2026-04-21 15:21:31,368 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07])
(logid:569f38b1) VM instance
{"id":4127,"instanceName":"i-2-4127-VM","state":"Migrating","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"}
state transited from [Running] to [Migrating] with event [MigrationRequested].
VM's original host: Host
{"id":318,"name":"com10","type":"Routing","uuid":"c8b05f7c-09cc-467e-b626-74048a6a870b"},
new host: Host
{"id":324,"name":"com11","type":"Routing","uuid":"2995de01-4d98-49c9-b379-ff8adf0f0722"},
host before state transition: Host
{"id":318,"name":"com10","type":"Routing","uuid":"c8b05f7c-09cc-467e-b626-74048a6a870b"}
2026-04-21 15:21:31,396 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl]
(Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07])
(logid:569f38b1) Setting VLAN persistence to [{"17742":true}] as part of
migrate command for VM [VM {id: "4127", name: "i-2-4127-VM", uuid:
"bf5f467f-c17a-4e6e-b3e2-1beb97941324", type: "User"}].
2026-04-21 15:21:31,396 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl]
(Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07])
(logid:569f38b1) Setting CPU shares to [800] as part of migrate command for VM
[VM {id: "4127", name: "i-2-4127-VM", uuid:
"bf5f467f-c17a-4e6e-b3e2-1beb97941324", type: "User"}].
2026-04-21 15:21:31,399 DEBUG [c.c.a.m.ClusteredAgentAttache]
(Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07])
(logid:569f38b1) Seq 446-8848728842852947950: Forwarding Seq
318-8848728842852947950: { Cmd , MgmtId: 345050754440, via: 318(com10), Ver:
v1, Flags: 100011,
[{"com.cloud.agent.api.MigrateCommand":{"vmName":"i-2-4127-VM","destinationIp":"10.xx.xx.61","migrateStorageManaged":"false","migrateNonSharedInc":"false","autoConvergence":"false","hostGuid":"dca8e648-8706-3504-adb2-7f20ea59398c-LibvirtComputingResource","windows":"false","virtualMachine":{"id":"4127","name":"i-2-4127-VM","state":"Running","type":"User","cpus":"2","minSpeed":"400","maxSpeed":"2000","minRam":"(2.00
GB) 2147483648","maxRam":"(2.00 GB) 2147483648","arch":"x86_64","os":"Other
Ubuntu (64-bit)","platformEmulator":"Other
Ubuntu","bootArgs":"","enableHA":"true","limitCpuUse":"false","enableDynamicallyScaleVm":"true","details":{"cpuOvercommitRatio":"5.0","rootdisksize":"20","rootDis
kController":"osdefault"},"uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324","enterHardwareSetup":"false","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"c9e66d60-139c-4e90-ab9a-f06d6486b63d","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"ssd;36b6ca0e-eb0f-4d11-9491-937d592f36e9","name":"SSD","id":"66","poolType":"StorPool","host":"n/a","path":"/dev/storpool-byid/","port":"0","url":"StorPool://n/a/dev/storpool-byid//?ROLE=Primary&STOREUUID=ssd;36b6ca0e-eb0f-4d11-9491-937d592f36e9","isManaged":"false"}},"name":"ROOT-4127","size":"(20.00
GB)
21474836480","path":"/dev/storpool-byid/n4en.b.bgxk","volumeId":"15599","vmName":"i-2-4127-VM","accountId":"2","format":"QCOW2","provisioningType":"THIN","poolId":"66","id":"15599","deviceId":"0","bytesReadRate":"(0
bytes) 0","bytesWriteRate":"(0 bytes) 0","iopsReadRate":"(0 bytes)
0","iopsWriteRate":"(0 bytes)
0","cacheMode":"NONE","hypervisorType":"KVM","directDownload":"fa
lse","deployAsIs":"false","checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":"true"}},"diskSeq":"0","path":"/dev/storpool-byid/n4en.b.bgxk","type":"ROOT","details":{"storageHost":"n/a","managed":"false","storagePort":"0","protocoltype":"StorPool","storage.pool.disk.wait":"60","volumeSize":"(20.00
GB)
21474836480"}},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":"0","format":"ISO","accountId":"0","hvm":"false","bootable":"false","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"diskSeq":"3","type":"ISO"}],"nics":[{"deviceId":"0","networkRateMbps":"-1","defaultNic":"true","pxeDisable":"false","nicUuid":"b1aee509-ad6b-4dd5-ade5-c1d44f995615","details":{"MacLearning":"false","PromiscuousMode":"false","internetProtocol":"IPv4","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":"false","networkId":"783","networkSegmentName":"D1-A2-Z2-V1124-S783","uuid":"958d365a-d975-419f-830a-b13b7f29389d","ip":"192.168.10.125
","netmask":"255.255.255.0","gateway":"192.168.10.1","mac":"02:02:03:0f:00:3c","dns1":"8.8.8.8","broadcastType":"Vxlan","type":"Guest","broadcastUri":"vxlan://17742","isolationUri":"vxlan://17742","securityGroupEnabled":"false","name":"cloudbr2"}],"vcpuMaxLimit":"2","configDriveLabel":"config-2","configDriveLocation":"SECONDARY","guestOsDetails":{},"extraConfig":{},"networkIdToNetworkNameMap":{"783":"D1-A2-Z2-V1124-S783"}},"executeInSequence":"false","migrateDiskInfoList":[],"dpdkInterfaceMapping":{},"newVmCpuShares":"800","vlanToPersistenceMap":{"17742":"true"},"wait":"0","bypassHostMaintenance":"false"}}]
} to 345050719314
2026-04-21 15:21:31,447 INFO [c.c.v.ClusteredVirtualMachineManagerImpl]
(Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07])
(logid:569f38b1) Migration was unsuccessful. Cleaning up: VM instance
{"id":4127,"instanceName":"i-2-4127-VM","state":"Migrating","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"}
2026-04-21 15:21:31,454 DEBUG [o.a.c.e.o.VolumeOrchestrator]
(Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07])
(logid:569f38b1) Releasing [1] volumes for VM [VM instance
{"id":4127,"instanceName":"i-2-4127-VM","state":"Migrating","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"}]
from host [Host
{"id":324,"name":"com11","type":"Routing","uuid":"2995de01-4d98-49c9-b379-ff8adf0f0722"}].
2026-04-21 15:21:31,463 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl]
(Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07])
(logid:569f38b1) VM instance
{"id":4127,"instanceName":"i-2-4127-VM","state":"Migrating","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"}
is a [User], returning null for control Nic IP.
2026-04-21 15:21:31,482 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07])
(logid:569f38b1) VM instance
{"id":4127,"instanceName":"i-2-4127-VM","state":"Running","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"}
state transited from [Migrating] to [Running] with event [OperationFailed].
VM's original host: Host
{"id":318,"name":"com10","type":"Routing","uuid":"c8b05f7c-09cc-467e-b626-74048a6a870b"},
new host: Host
{"id":318,"name":"com10","type":"Routing","uuid":"c8b05f7c-09cc-467e-b626-74048a6a870b"},
host before state transition: Host
{"id":324,"name":"com11","type":"Routing","uuid":"2995de01-4d98-49c9-b379-ff8adf0f0722"}`
### Additional Context
From the libvirt/qemu logs on the source host, it is observed that the VM
domain **did exist previously** as part of an earlier migration.
At around 15:20, the VM was successfully started on the source host and
participated in a migration process:
**log cloudstack-agent (source host)
`2026-04-21 15:21:29,362 DEBUG [cloud.agent.Agent]
(AgentRequest-Handler-3:[]) (logid:569f38b1) Processing command:
com.cloud.agent.api.GetVmDiskStatsCommand
2026-04-21 15:21:29,362 DEBUG [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-3:[]) (logid:569f38b1) Looking for libvirtd connection
at: qemu:///system
2026-04-21 15:21:29,374 WARN
[resource.wrapper.LibvirtGetVmDiskStatsCommandWrapper]
(AgentRequest-Handler-3:[]) (logid:569f38b1) Can't get vm disk stats:
org.libvirt.LibvirtException: Domain not found: no domain with matching name
'i-2-4127-VM', continue
2026-04-21 15:21:29,375 DEBUG [cloud.agent.Agent]
(AgentRequest-Handler-3:[]) (logid:569f38b1) Seq 318-8848728842852947947: {
Ans: , MgmtId: 345050754440, via: 318, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.GetVmDiskStatsAnswer":{"hostName":"tepub03com10","vmDiskStatsMap":{},"result":"true","details":"","wait":"0","bypassHostMaintenance":"false"}}]
}
2026-04-21 15:21:29,422 DEBUG [cloud.agent.Agent]
(AgentRequest-Handler-2:[]) (logid:569f38b1) Processing command:
com.cloud.agent.api.GetVmNetworkStatsCommand
2026-04-21 15:21:29,422 DEBUG [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-2:[]) (logid:569f38b1) Looking for libvirtd connection
at: qemu:///system
2026-04-21 15:21:29,435 WARN
[resource.wrapper.LibvirtGetVmNetworkStatsCommandWrapper]
(AgentRequest-Handler-2:[]) (logid:569f38b1) Can't get vm network stats:
org.libvirt.LibvirtException: Domain not found: no domain with matching name
'i-2-4127-VM', continue
2026-04-21 15:21:29,436 DEBUG [cloud.agent.Agent]
(AgentRequest-Handler-2:[]) (logid:569f38b1) Seq 318-8848728842852947948: {
Ans: , MgmtId: 345050754440, via: 318, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.GetVmNetworkStatsAnswer":{"hostName":"tepub03com10","vmNetworkStatsMap":{},"result":"true","details":"","wait":"0","bypassHostMaintenance":"false"}}]
}
2026-04-21 15:21:31,403 DEBUG [cloud.agent.Agent]
(AgentRequest-Handler-5:[]) (logid:569f38b1) Request:Seq
318-8848728842852947950: { Cmd , MgmtId: 345050754440, via: 318, Ver: v1,
Flags: 100011,
[{"com.cloud.agent.api.MigrateCommand":{"vmName":"i-2-4127-VM","destinationIp":"10.120.25.61","migrateStorageManaged":"false","migrateNonSharedInc":"false","autoConvergence":"false","hostGuid":"dca8e648-8706-3504-adb2-7f20ea59398c-LibvirtComputingResource","windows":"false","virtualMachine":{"id":"4127","name":"i-2-4127-VM","state":"Running","type":"User","cpus":"2","minSpeed":"400","maxSpeed":"2000","minRam":"(2.00
GB) 2147483648","maxRam":"(2.00 GB) 2147483648","arch":"x86_64","os":"Other
Ubuntu (64-bit)","platformEmulator":"Other
Ubuntu","bootArgs":"","enableHA":"true","limitCpuUse":"false","enableDynamicallyScaleVm":"true","details":{"cpuOvercommitRatio":"5.0","rootdisksize":"20","rootDiskController":"osdefault"},"uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324","enterHardwareSetup":"fal
se","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"c9e66d60-139c-4e90-ab9a-f06d6486b63d","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"ssd;36b6ca0e-eb0f-4d11-9491-937d592f36e9","name":"SSD","id":"66","poolType":"StorPool","host":"n/a","path":"/dev/storpool-byid/","port":"0","url":"StorPool://n/a/dev/storpool-byid//?ROLE=Primary&STOREUUID=ssd;36b6ca0e-eb0f-4d11-9491-937d592f36e9","isManaged":"false"}},"name":"ROOT-4127","size":"(20.00
GB)
21474836480","path":"/dev/storpool-byid/n4en.b.bgxk","volumeId":"15599","vmName":"i-2-4127-VM","accountId":"2","format":"QCOW2","provisioningType":"THIN","poolId":"66","id":"15599","deviceId":"0","bytesReadRate":"(0
bytes) 0","bytesWriteRate":"(0 bytes) 0","iopsReadRate":"(0 bytes)
0","iopsWriteRate":"(0 bytes)
0","cacheMode":"NONE","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false","checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":"t
rue"}},"diskSeq":"0","path":"/dev/storpool-byid/n4en.b.bgxk","type":"ROOT","details":{"storageHost":"n/a","managed":"false","storagePort":"0","protocoltype":"StorPool","storage.pool.disk.wait":"60","volumeSize":"(20.00
GB)
21474836480"}},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":"0","format":"ISO","accountId":"0","hvm":"false","bootable":"false","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"diskSeq":"3","type":"ISO"}],"nics":[{"deviceId":"0","networkRateMbps":"-1","defaultNic":"true","pxeDisable":"false","nicUuid":"b1aee509-ad6b-4dd5-ade5-c1d44f995615","details":{"MacLearning":"false","PromiscuousMode":"false","internetProtocol":"IPv4","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":"false","networkId":"783","networkSegmentName":"D1-A2-Z2-V1124-S783","uuid":"958d365a-d975-419f-830a-b13b7f29389d","ip":"192.168.10.125","netmask":"255.255.255.0","gateway":"192.168.10.1","mac":"02:02:03:0f:00:3c","dns1":"8.8.8.8","
broadcastType":"Vxlan","type":"Guest","broadcastUri":"vxlan://17742","isolationUri":"vxlan://17742","securityGroupEnabled":"false","name":"cloudbr2"}],"vcpuMaxLimit":"2","configDriveLabel":"config-2","configDriveLocation":"SECONDARY","guestOsDetails":{},"extraConfig":{},"networkIdToNetworkNameMap":{"783":"D1-A2-Z2-V1124-S783"}},"executeInSequence":"false","migrateDiskInfoList":[],"dpdkInterfaceMapping":{},"newVmCpuShares":"800","vlanToPersistenceMap":{"17742":"true"},"wait":"0","bypassHostMaintenance":"false"}}]
}
2026-04-21 15:21:31,403 DEBUG [cloud.agent.Agent]
(AgentRequest-Handler-5:[]) (logid:569f38b1) Processing command:
com.cloud.agent.api.MigrateCommand
2026-04-21 15:21:31,419 DEBUG
[resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-5:[])
(logid:569f38b1) Trying to migrate VM [i-2-4127-VM] to destination host:
[qemu+tls://10.120.25.61/system].
2026-04-21 15:21:31,419 DEBUG [agent.properties.AgentPropertiesFileHandler]
(AgentRequest-Handler-5:[]) (logid:569f38b1) Property [hypervisor.uri] has
empty or null value. Using default value [null].
2026-04-21 15:21:31,419 DEBUG [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-5:[]) (logid:569f38b1) Looking for libvirtd connection
at: qemu:///system
2026-04-21 15:21:31,431 DEBUG [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-5:[]) (logid:569f38b1) Can not find KVM connection for
Instance: i-2-4127-VM, continuing.
2026-04-21 15:21:31,432 DEBUG [agent.properties.AgentPropertiesFileHandler]
(AgentRequest-Handler-5:[]) (logid:569f38b1) Property [hypervisor.uri] has
empty or null value. Using default value [null].
2026-04-21 15:21:31,432 DEBUG [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-5:[]) (logid:569f38b1) Looking for libvirtd connection
at: lxc:///
2026-04-21 15:21:31,432 INFO [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-5:[]) (logid:569f38b1) No existing libvirtd connection
found. Opening a new one
2026-04-21 15:21:31,432 DEBUG [agent.properties.AgentPropertiesFileHandler]
(AgentRequest-Handler-5:[]) (logid:569f38b1) Property [libvirt.events.enabled]
has empty or null value. Using default value [false].
2026-04-21 15:21:31,432 DEBUG [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-5:[]) (logid:569f38b1) Libvirt event listening is
disabled, not setting up event loop
2026-04-21 15:21:31,433 DEBUG [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-5:[]) (logid:569f38b1) Can not find LXC connection for
Instance: i-2-4127-VM, continuing.
2026-04-21 15:21:31,433 WARN [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-5:[]) (logid:569f38b1) Can not find a connection for
Instance i-2-4127-VM. Assuming the default connection.
2026-04-21 15:21:31,433 DEBUG [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-5:[]) (logid:569f38b1) Looking for libvirtd connection
at: qemu:///system
2026-04-21 15:21:31,445 DEBUG [kvm.resource.LibvirtComputingResource]
(AgentRequest-Handler-5:[]) (logid:569f38b1) Failed to get dom xml:
org.libvirt.LibvirtException: Domain not found: no domain with matching name
'i-2-4127-VM'
2026-04-21 15:21:31,446 DEBUG [kvm.resource.LibvirtComputingResource]
(AgentRequest-Handler-5:[]) (logid:569f38b1) Failed to get dom xml:
org.libvirt.LibvirtException: Domain not found: no domain with matching name
'i-2-4127-VM'
2026-04-21 15:21:31,446 DEBUG
[resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-5:[])
(logid:569f38b1) Found domain with name [i-2-4127-VM]. Starting VM migration to
host [qemu+tls://10.120.25.61/system].
2026-04-21 15:21:31,446 ERROR
[resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-5:[])
(logid:569f38b1) Can't migrate domain [i-2-4127-VM] due to: [Domain not found:
no domain with matching name 'i-2-4127-VM']. org.libvirt.LibvirtException:
Domain not found: no domain with matching name 'i-2-4127-VM'
2026-04-21 15:21:31,447 DEBUG [cloud.agent.Agent]
(AgentRequest-Handler-5:[]) (logid:569f38b1) Seq 318-8848728842852947950: {
Ans: , MgmtId: 345050754440, via: 318, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.MigrateAnswer":{"result":"false","details":"Domain not
found: no domain with matching name
'i-2-4127-VM'","wait":"0","bypassHostMaintenance":"false"}}] }`
**log libvirt (source host)
`2026-04-21 08:20:25.095+0000: starting up libvirt version: 8.0.0, package:
23.2.module_el8.10.0+3869+b8959270 (AlmaLinux Packaging Team
<[email protected]>, 2024-07-10-04:57:33, ), qemu version:
6.2.0qemu-kvm-6.2.0-53.module_el8.10.0+3897+eb84924d, kernel:
4.18.0-553.22.1.el8_10.x86_64, hostname: com10
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \
HOME=/var/lib/libvirt/qemu/domain-21617-i-2-4127-VM \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-21617-i-2-4127-VM/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-21617-i-2-4127-VM/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-21617-i-2-4127-VM/.config \
/usr/libexec/qemu-kvm \
-name guest=i-2-4127-VM,debug-threads=on \
-S \
-object
'{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-21617-i-2-4127-VM/master-key.aes"}'
\
-machine
pc-i440fx-rhel7.6.0,usb=off,dump-guest-core=off,memory-backend=pc.ram \
-accel kvm \
-cpu
Cascadelake-Server-noTSX,mpx=off,hypervisor=on,pku=on,arch-capabilities=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on
\
-m 2048 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":2147483648}' \
-overcommit mem-lock=off \
-smp 2,sockets=2,dies=1,cores=1,threads=1 \
-uuid bf5f467f-c17a-4e6e-b3e2-1beb97941324 \
-smbios 'type=1,manufacturer=Apache Software Foundation,product=CloudStack
KVM
Hypervisor,serial=bf5f467f-c17a-4e6e-b3e2-1beb97941324,uuid=bf5f467f-c17a-4e6e-b3e2-1beb97941324'
\
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=70,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 \
-blockdev
'{"driver":"host_device","filename":"/dev/storpool-byid/n4en.b.bgxk","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}'
\
-blockdev
'{"node-name":"libvirt-2-format","read-only":false,"discard":"unmap","cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}'
\
-device
virtio-blk-pci,bus=pci.0,addr=0x5,drive=libvirt-2-format,id=virtio-disk0,bootindex=2,write-cache=on,serial=c9e66d60139c4e90ab9a
\
-device ide-cd,bus=ide.1,unit=0,id=ide0-1-0,bootindex=1 \
-netdev tap,fd=76,id=hostnet0,vhost=on,vhostfd=86 \
-device
virtio-net-pci,netdev=hostnet0,id=net0,mac=02:02:03:0f:00:3c,bus=pci.0,addr=0x3
\
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=67,server=on,wait=off \
-device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0
\
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-audiodev '{"id":"audio1","driver":"none"}' \
-object
'{"qom-type":"tls-creds-x509","id":"vnc-tls-creds0","dir":"/etc/pki/libvirt-vnc","endpoint":"server","verify-peer":true}'
\
-vnc 10.xx.xx.60:0,password=on,tls-creds=vnc-tls-creds0,audiodev=audio1 \
-device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \
-device i6300esb,id=watchdog0,bus=pci.0,addr=0x7 \
-watchdog-action none \
-incoming defer \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 \
-sandbox
on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2026-04-21 08:20:25.095+0000: Domain id=21617 is tainted: high-privileges
2026-04-21 08:20:25.096+0000: Domain id=21617 is tainted: hook-script
char device redirected to /dev/pts/25 (label charserial0)
2026-04-21T08:20:25.719385Z qemu-kvm: -device
cirrus-vga,id=video0,bus=pci.0,addr=0x2: warning: 'cirrus-vga' is deprecated,
please use a different VGA card instead
2026-04-21 08:20:56.634+0000: initiating migration
2026-04-21 08:20:57.921+0000: shutting down, reason=migrated
2026-04-21T08:20:57.922592Z qemu-kvm: terminating on signal 15 from pid
3930964 (/usr/sbin/libvirtd)
`
**log libvirt (dest host)
`2026-04-21 08:20:54.470+0000: starting up libvirt version: 8.0.0, package:
23.2.module_el8.10.0+3869+b8959270 (AlmaLinux Packaging Team
<[email protected]>, 2024-07-10-04:57:33, ), qemu version:
6.2.0qemu-kvm-6.2.0-53.module_el8.10.0+3897+eb84924d, kernel:
4.18.0-553.22.1.el8_10.x86_64, hostname: com11
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \
HOME=/var/lib/libvirt/qemu/domain-321-i-2-4127-VM \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-321-i-2-4127-VM/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-321-i-2-4127-VM/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-321-i-2-4127-VM/.config \
/usr/libexec/qemu-kvm \
-name guest=i-2-4127-VM,debug-threads=on \
-S \
-object
'{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-321-i-2-4127-VM/master-key.aes"}'
\
-machine
pc-i440fx-rhel7.6.0,usb=off,dump-guest-core=off,memory-backend=pc.ram \
-accel kvm \
-cpu
Cascadelake-Server-noTSX,mpx=off,hypervisor=on,pku=on,arch-capabilities=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on
\
-m 2048 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":2147483648}' \
-overcommit mem-lock=off \
-smp 2,sockets=2,dies=1,cores=1,threads=1 \
-uuid bf5f467f-c17a-4e6e-b3e2-1beb97941324 \
-smbios 'type=1,manufacturer=Apache Software Foundation,product=CloudStack
KVM
Hypervisor,serial=bf5f467f-c17a-4e6e-b3e2-1beb97941324,uuid=bf5f467f-c17a-4e6e-b3e2-1beb97941324'
\
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=42,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 \
-blockdev
'{"driver":"host_device","filename":"/dev/storpool-byid/n4en.b.bgxk","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}'
\
-blockdev
'{"node-name":"libvirt-2-format","read-only":false,"discard":"unmap","cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}'
\
-device
virtio-blk-pci,bus=pci.0,addr=0x5,drive=libvirt-2-format,id=virtio-disk0,bootindex=2,write-cache=on,serial=c9e66d60139c4e90ab9a
\
-device ide-cd,bus=ide.1,unit=0,id=ide0-1-0,bootindex=1 \
-netdev tap,fd=46,id=hostnet0,vhost=on,vhostfd=55 \
-device
virtio-net-pci,netdev=hostnet0,id=net0,mac=02:02:03:0f:00:3c,bus=pci.0,addr=0x3
\
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=41,server=on,wait=off \
-device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0
\
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-audiodev '{"id":"audio1","driver":"none"}' \
-object
'{"qom-type":"tls-creds-x509","id":"vnc-tls-creds0","dir":"/etc/pki/libvirt-vnc","endpoint":"server","verify-peer":true}'
\
-vnc 10.xx.xx.61:1,password=on,tls-creds=vnc-tls-creds0,audiodev=audio1 \
-device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \
-device i6300esb,id=watchdog0,bus=pci.0,addr=0x7 \
-watchdog-action none \
-incoming defer \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 \
-sandbox
on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2026-04-21 08:20:54.470+0000: Domain id=321 is tainted: high-privileges
2026-04-21 08:20:54.470+0000: Domain id=321 is tainted: hook-script
char device redirected to /dev/pts/3 (label charserial0)
2026-04-21T08:20:55.588925Z qemu-kvm: -device
cirrus-vga,id=video0,bus=pci.0,addr=0x2: warning: 'cirrus-vga' is deprecated,
please use a different VGA card instead
2026-04-21T08:21:32.547875Z qemu-kvm: terminating on signal 15 from pid
1223112 (/usr/sbin/libvirtd)
2026-04-21 08:21:32.806+0000: shutting down, reason=shutdown`
- After the failed migration:
- The VM is **not running on the source host**
- The VM is also **not running on the destination host**
### versions
CloudStack version: 4.22
Hypervisor: KVM (libvirt 8.0.0, qemu-kvm 6.2.0)
OS: AlmaLinux / RHEL-based
### The steps to reproduce the bug
1. Perform VM migration sequentially across multiple hosts (e.g., com01 →
com02 → com03 → … → com24)
2. Repeat the migration process multiple times across the cluster until the
issue occurs
3. this problem occurs when migrating from com10 to com11
### What to do about it?
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
