GitHub user joan-911 created a discussion: Recurring Instance Creation Failures
in CloudStack 4.19.3 on XCP-ng 8.2.1 – Management Service Restart Required
Environment Details:
CloudStack Version: 4.19.3
Hypervisor: XCP-ng 8.2.1
Primary Storage: Pre-configured iSCSI on XCP-ng
**Issue Observed:**
All instance creation requests are timing out, and new instances fail to
deploy. This problem tends to recur every 5–6 days. Once it occurs, restarting
the management service temporarily resolves the issue.
We would like to know if anyone has faced a similar issue in ACS with XCP-ng.
Are there any configuration tweaks available to prevent having to restart the
management service each time?
Additional Information:
Logs related to disk creation are attached for reference.
2025-10-22 10:24:33,667 DEBUG [c.c.a.m.ClusteredAgentAttache]
(Work-Job-Executor-62:ctx-a00ad243 job-133845/job-133846 ctx-50f38497)
(logid:13a10f8a) Seq 20-6844627008673079170: Forwarding Seq
20-6844627008673079170: { Cmd , MgmtId: 271367514881978, via:
20(CLD-Z1-XCPH01), Ver: v1, Flags: 100111,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"9f1a2c02-4c9b-4910-b3e8-c1026e42addf","origUrl":"http://download.cloudstack.org/systemvm/4.19/systemvmtemplate-4.19.1-xen.vhd.bz2","uuid":"1036c354-65b7-492f-96fc-848ee19f49ab","id":"380","format":"VHD","accountId":"6","checksum":"{SHA-512}e7df5f0fc04318bf4bf2b28fa0e630cfb3b7cd3f783202382202c8404fc8deaadc5c03e8b0c2137a2fb7d998ee3901e3d94f280587999fee6659399c064b59ea","hvm":"true","displayText":"systemvm-xenserver-4.19.1","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"CLD-Z1-VOL2","name":"CLD-Z1-VOL2","id":"7","poolType":"PreSetup","ho
st":"localhost","path":"/CLD-Z1-VOL2","port":"0","url":"PreSetup://localhost/CLD-Z1-VOL2/?ROLE=Primary&STOREUUID=CLD-Z1-VOL2","isManaged":"false"}},"name":"380-6-45c2d735-1393-3ebb-9b43-21248166660f","size":"(4.88
GB)
5242880000","hypervisorType":"XenServer","bootable":"false","uniqueName":"380-6-45c2d735-1393-3ebb-9b43-21248166660f","directDownload":"false","deployAsIs":"false","followRedirects":"true"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"2cf5fcb9-9570-4263-95ea-98004c0efa98","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"CLD-Z1-VOL2","name":"CLD-Z1-VOL2","id":"7","poolType":"PreSetup","host":"localhost","path":"/CLD-Z1-VOL2","port":"0","url":"PreSetup://localhost/CLD-Z1-VOL2/?ROLE=Primary&STOREUUID=CLD-Z1-VOL2","isManaged":"false"}},"name":"ROOT-15673","size":"(4.88
GB)
5242880000","volumeId":"6223","vmName":"r-15673-VM","accountId":"1028","format":"VHD","provisioningType":"THIN","poolId":"7","id":"
6223","deviceId":"0","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"}},"executeInSequence":"true","options":{},"options2":{},"wait":"0","bypassHostMaintenance":"false"}}]
} to 108462341209969
2025-10-22 10:54:33,668 DEBUG [c.c.a.m.AgentAttache]
(Work-Job-Executor-62:ctx-a00ad243 job-133845/job-133846 ctx-50f38497)
(logid:13a10f8a) Seq 20-6844627008673079170: Waiting some more time because
this is the current command
2025-10-22 11:24:33,668 DEBUG [c.c.a.m.AgentAttache]
(Work-Job-Executor-62:ctx-a00ad243 job-133845/job-133846 ctx-50f38497)
(logid:13a10f8a) Seq 20-6844627008673079170: Waiting some more time because
this is the current command
2025-10-22 11:24:33,674 WARN [c.c.a.m.AgentAttache]
(Work-Job-Executor-62:ctx-a00ad243 job-133845/job-133846 ctx-50f38497)
(logid:13a10f8a) Seq 20-6844627008673079170: Timed out on Seq
20-6844627008673079170: { Cmd , MgmtId: 271367514881978, via:
20(CLD-Z1-XCPH01), Ver: v1, Flags: 100111,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"9f1a2c02-4c9b-4910-b3e8-c1026e42addf","origUrl":"http://download.cloudstack.org/systemvm/4.19/systemvmtemplate-4.19.1-xen.vhd.bz2","uuid":"1036c354-65b7-492f-96fc-848ee19f49ab","id":"380","format":"VHD","accountId":"6","checksum":"{SHA-512}e7df5f0fc04318bf4bf2b28fa0e630cfb3b7cd3f783202382202c8404fc8deaadc5c03e8b0c2137a2fb7d998ee3901e3d94f280587999fee6659399c064b59ea","hvm":"true","displayText":"systemvm-xenserver-4.19.1","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"CLD-Z1-VOL2","name":"CLD-Z1-VOL2","id":"7","poolType":"PreSetup","host":"lo
calhost","path":"/CLD-Z1-VOL2","port":"0","url":"PreSetup://localhost/CLD-Z1-VOL2/?ROLE=Primary&STOREUUID=CLD-Z1-VOL2","isManaged":"false"}},"name":"380-6-45c2d735-1393-3ebb-9b43-21248166660f","size":"(4.88
GB)
5242880000","hypervisorType":"XenServer","bootable":"false","uniqueName":"380-6-45c2d735-1393-3ebb-9b43-21248166660f","directDownload":"false","deployAsIs":"false","followRedirects":"true"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"2cf5fcb9-9570-4263-95ea-98004c0efa98","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"CLD-Z1-VOL2","name":"CLD-Z1-VOL2","id":"7","poolType":"PreSetup","host":"localhost","path":"/CLD-Z1-VOL2","port":"0","url":"PreSetup://localhost/CLD-Z1-VOL2/?ROLE=Primary&STOREUUID=CLD-Z1-VOL2","isManaged":"false"}},"name":"ROOT-15673","size":"(4.88
GB)
5242880000","volumeId":"6223","vmName":"r-15673-VM","accountId":"1028","format":"VHD","provisioningType":"THIN","poolId":"7","id":"6223","
deviceId":"0","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"}},"executeInSequence":"true","options":{},"options2":{},"wait":"0","bypassHostMaintenance":"false"}}]
}
2025-10-22 11:24:33,674 DEBUG [c.c.a.m.ClusteredAgentAttache]
(Work-Job-Executor-62:ctx-a00ad243 job-133845/job-133846 ctx-50f38497)
(logid:13a10f8a) Seq 20-6844627008673079170: Forwarding to peer to cancel due
to timeout
2025-10-22 11:24:33,675 DEBUG [c.c.a.m.AgentAttache]
(Work-Job-Executor-62:ctx-a00ad243 job-133845/job-133846 ctx-50f38497)
(logid:13a10f8a) Seq 20-6844627008673079170: Cancelling.
2025-10-22 11:24:33,675 DEBUG [o.a.c.s.RemoteHostEndPoint]
(Work-Job-Executor-62:ctx-a00ad243 job-133845/job-133846 ctx-50f38497)
(logid:13a10f8a) Failed to send command, due to Agent:20,
com.cloud.exception.OperationTimedoutException: Commands 6844627008673079170 to
Host 20 timed out after 3600
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to
Agent:20, com.cloud.exception.OperationTimedoutException: Commands
6844627008673079170 to Host 20 timed out after 3600
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to send
command, due to Agent:20, com.cloud.exception.OperationTimedoutException:
Commands 6844627008673079170 to Host 20 timed out after 3600
GitHub link: https://github.com/apache/cloudstack/discussions/11885
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
