GitHub user PPisz added a comment to the discussion: CloudStack fails to start
more VMs
Similar issue afer upgrade to 4.20 (Ubuntu 22), but I couldn't run any VM
(unfortunately, we had to withdraw the upgrade, I cannot provide more detailed
information) agent.log:
``2025-01-10 12:28:30,179 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:null) (logid:fb2d5891) Exit value of process [3708958]
for command [/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
update_config.py 169.254.93.185
vm_dhcp_entry.json.82a5c90f-4744-4bba-abe0-df28dfaa3c0c ] is [1].
2025-01-10 12:28:30,180 WARN [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:null) (logid:fb2d5891) Exception [Stream closed]
occurred when attempting to run command
[/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
update_config.py 169.254.93.185
vm_dhcp_entry.json.82a5c90f-4744-4bba-abe0-df28dfaa3c0c ].
java.io.IOException: Stream closed
at
java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:168)
at
java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:281)
at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:324)
at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:189)
at java.base/java.io.InputStreamReader.read(InputStreamReader.java:177)
at java.base/java.io.BufferedReader.fill(BufferedReader.java:162)
at java.base/java.io.BufferedReader.readLine(BufferedReader.java:329)
at java.base/java.io.BufferedReader.readLine(BufferedReader.java:396)
at
com.cloud.utils.script.OutputInterpreter.processError(OutputInterpreter.java:38)
at com.cloud.utils.script.Script.execute(Script.java:313)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:546)
at
com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfigToVR(VirtualRoutingResource.java:302)
at
com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfig(VirtualRoutingResource.java:317)
at
com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:164)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:35)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:29)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1929)
at com.cloud.agent.Agent.processRequest(Agent.java:683)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1106)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
2025-01-10 12:28:30,181 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:null) (logid:fb2d5891) Executing script in VR:
update_config.py
2025-01-10 12:28:30,181 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-1:null) (logid:fb2d5891) Processing ScriptConfigItem,
executing update_config.py
vm_dhcp_entry.json.82a5c90f-4744-4bba-abe0-df28dfaa3c0c took 363ms
2025-01-10 12:28:30,190 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null)
(logid:fb2d5891) Seq 5-2621376458106339390: { Ans: , MgmtId: 222320990092816,
via: 5, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success:
Creating file in VR, with ip: 169.254.93.185, file:
vm_dhcp_entry.json.82a5c90f-4744-4bba-abe0-df28dfaa3c0c","null - failed:
java.io.IOException: Stream closed
at
java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:168)
at
java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:281)
at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:324)
at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:189)
at java.base/java.io.InputStreamReader.read(InputStreamReader.java:177)
at java.base/java.io.BufferedReader.fill(BufferedReader.java:162)
at java.base/java.io.BufferedReader.readLine(BufferedReader.java:329)
at java.base/java.io.BufferedReader.readLine(BufferedReader.java:396)
at
java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:168)
at
java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:281)
at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:324)
at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:189)
at java.base/java.io.InputStreamReader.read(InputStreamReader.java:177)
at java.base/java.io.BufferedReader.fill(BufferedReader.java:162)
at java.base/java.io.BufferedReader.readLine(BufferedReader.java:329)
at java.base/java.io.BufferedReader.readLine(BufferedReader.java:396)
at
com.cloud.utils.script.OutputInterpreter.processError(OutputInterpreter.java:38)
at com.cloud.utils.script.Script.execute(Script.java:313)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:546)
at
com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfigToVR(VirtualRoutingResource.java:302)
at
com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfig(VirtualRoutingResource.java:317)
at
com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:164)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:35)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:29)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1929)
at com.cloud.agent.Agent.processRequest(Agent.java:683)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1106)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
"],"result":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2025-01-10 12:28:31,311 DEBUG [kvm.resource.LibvirtComputingResource]
(UgentTask-5:null) (logid:) Executing command
[/usr/share/cloudstack-common/scripts/vm/network/security_group.py
get_rule_logs_for_vms ].
2025-01-10 12:28:31,429 DEBUG [kvm.resource.LibvirtComputingResource]
(UgentTask-5:null) (logid:) Successfully executed process [3708972] for command
[/usr/share/cloudstack-common/scripts/vm/network/security_group.py
get_rule_logs_for_vms ].
2025-01-10 12:28:31,430 DEBUG [agent.properties.AgentPropertiesFileHandler]
(UgentTask-5:null) (logid:) Property [hypervisor.uri] has empty or null value.
Using default value [null].
2025-01-10 12:28:31,430 DEBUG [kvm.resource.LibvirtConnection]
(UgentTask-5:null) (logid:) Looking for libvirtd connection at: qemu:///system`
and managemen.log:
`2025-01-10 12:28:30,981 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-1:[ctx-6edbbbab, job-69848/job-69851, ctx-ce398d8d])
(logid:fb2d5891) Invocation exception, caused by:
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM instance
{"id":557,"instanceName":"i-39-557-VM","type":"User","uuid":"90f85f42-ed81-42e3-953f-20edba0b1304"}Scope=interface
com.cloud.dc.DataCenter; id=1
2025-01-10 12:28:30,981 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-1:[ctx-6edbbbab, job-69848/job-69851, ctx-ce398d8d])
(logid:fb2d5891) Rethrow exception
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM instance
{"id":557,"instanceName":"i-39-557-VM","type":"User","uuid":"90f85f42-ed81-42e3-953f-20edba0b1304"}Scope=interface
com.cloud.dc.DataCenter; id=1
2025-01-10 12:28:30,981 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-1:[ctx-6edbbbab, job-69848/job-69851]) (logid:fb2d5891) Done
with run of VM work job: com.cloud.vm.VmWorkStart for VM 557, job origin: 69848
2025-01-10 12:28:30,981 ERROR
[c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-1:[ctx-6edbbbab,
job-69848/job-69851]) (logid:fb2d5891) Unable to complete AsyncJobVO:
{id:69851, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd:
com.cloud.vm.VmWorkStart, cmdInfo:
rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVy
cQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAACLXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAXBzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAAAAAAAAFxAH4ACnBwcHEAfgAKcHBw,
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 222320990092816, completeMsid: null, lastUpdated: null,
lastPolled: null, created: Fri Jan 10 12:26:04 CET 2025, removed: null}, job
origin: 69848 com.cloud.exception.InsufficientServerCapacityException: Unable
to create a deployment for VM instance
{"id":557,"instanceName":"i-39-557-VM","type":"User","uuid":"90f85f42-ed81-42e3-953f-20edba0b1304"}Scope=interface
com.cloud.dc.DataCenter; id=1
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1237)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5467)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.Delegating
MethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:106)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5591)
at
com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:652)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:600)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at
java.base/java.lang.Thread.run(Thread.java:840)`
GitHub link:
https://github.com/apache/cloudstack/discussions/10184#discussioncomment-11844336
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
