maic opened a new issue, #7603:
URL: https://github.com/apache/cloudstack/issues/7603
<!--
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release and main branch are affected too.
Always add information AFTER of these HTML comments, but no need to delete
the comments.
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
* Bug Report
* Improvement Request
* Enhancement Request
* Feature Idea
* Documentation Report
* Other
##### COMPONENT NAME
<!--
Categorize the issue, e.g. API, VR, VPN, UI, etc.
-->
~~~
storage issue
~~~
##### CLOUDSTACK VERSION
<!--
New line separated list of affected versions, commit ID for issues on main
branch.
-->
~~~
4.18
~~~
##### CONFIGURATION
<!--
Information about the configuration if relevant, e.g. basic network,
advanced networking, etc. N/A otherwise
-->
##### OS / ENVIRONMENT
<!--
Information about the environment if relevant, N/A otherwise
-->
##### SUMMARY
<!-- Explain the problem/feature briefly -->
##### STEPS TO REPRODUCE
<!--
For bugs, show exactly how to reproduce the problem, using a minimal
test-case. Use Screenshots if accurate.
For new features, show how the feature would be used.
-->
<!-- Paste example playbooks or commands between quotes below -->
~~~
uses glusterfs10.4 as the primary storage and cannot create VM
~~~
<!-- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS
<!-- What did you expect to happen when running the steps above? -->
~~~
~~~
##### ACTUAL RESULTS
<!-- What actually happened? -->
<!-- Paste verbatim command output between quotes below -->
~~~
~~~
Management error log：
2023-06-06 06:38:29,674 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-1:ctx-5024afa9 job-26/job-28 ctx-b07c23a5) (logid:85ef2375)
release mem from host: 1, old used: (1.00 GB) 1073741824,reserved: (0 bytes) 0,
total: (61.84 GB) 66404003840; new used: (0 bytes) 0,reserved:(0 bytes) 0;
movedfromreserved: false,moveToReserveredfalse
2023-06-06 06:38:29,674 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-2:ctx-9a279884 job-27/job-29 ctx-4d260ac7) (logid:acc6738a)
Rethrow exception com.cloud.exception.InsufficientServerCapacityException:
Unable to create a deployment for VM instance
{"id":4,"instanceName":"s-4-VM","type":"SecondaryStorageVm","uuid":"2de14fc6-04bf-4c99-a207-2e082c2f8e79"}Scope=interface
com.cloud.dc.DataCenter; id=1
2023-06-06 06:38:29,675 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-2:ctx-9a279884 job-27/job-29) (logid:acc6738a) Done with run
of VM work job: com.cloud.vm.VmWorkStart for VM 4, job origin: 27
2023-06-06 06:38:29,676 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-2:ctx-9a279884 job-27/job-29) (logid:acc6738a) Unable to
complete AsyncJobVO: {id:29, userId: 1, accountId: 1, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAABHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 86169082791055, com
pleteMsid: null, lastUpdated: null, lastPolled: null, created: Tue Jun 06
06:38:19 UTC 2023, removed: null}, job origin:27
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM instance
{"id":4,"instanceName":"s-4-VM","type":"SecondaryStorageVm","uuid":"2de14fc6-04bf-4c99-a207-2e082c2f8e79"}Scope=interface
com.cloud.dc.DataCenter; id=1
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1191)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5376)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5500)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-06-06 06:38:29,690 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-1:ctx-5024afa9 job-26/job-28 ctx-b07c23a5) (logid:85ef2375)
Invocation exception, caused by:
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM instance
{"id":3,"instanceName":"v-3-VM","type":"ConsoleProxy","uuid":"566eea1d-894a-4040-9580-13d1974c801d"}Scope=interface
com.cloud.dc.DataCenter; id=1
2023-06-06 06:38:29,690 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-1:ctx-5024afa9 job-26/job-28 ctx-b07c23a5) (logid:85ef2375)
Rethrow exception com.cloud.exception.InsufficientServerCapacityException:
Unable to create a deployment for VM instance
{"id":3,"instanceName":"v-3-VM","type":"ConsoleProxy","uuid":"566eea1d-894a-4040-9580-13d1974c801d"}Scope=interface
com.cloud.dc.DataCenter; id=1
agent error log：
2023-06-06 04:26:53,401 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:192e1783) Attempting to remove storage
pool 8778b476-66d3-3848-8f0e-718d85c322d4 from libvirt
2023-06-06 04:26:53,423 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:192e1783) Storage pool
8778b476-66d3-3848-8f0e-718d85c322d4 has no corresponding secret. Not removing
any secret.
2023-06-06 04:26:53,764 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:192e1783) Storage pool
8778b476-66d3-3848-8f0e-718d85c322d4 was successfully removed from libvirt.
2023-06-06 04:26:54,063 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-3:null) (logid:192e1783) Trying to fetch storage pool
3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 04:26:54,098 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-3:null) (logid:192e1783) Trying to fetch storage pool
3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 04:26:54,166 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-3:null) (logid:192e1783) Creating volume
70169d30-1815-4d64-9e6f-16b6dece69ea from template
642b7a0f-5c87-4881-a27c-4fbaa2db34be in pool
3b8ce946-c4a6-38d0-9484-1495a452e9d2 (Gluster) with size (0 bytes) 0
2023-06-06 04:26:54,167 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-3:null) (logid:192e1783) Attempting to create volume
70169d30-1815-4d64-9e6f-16b6dece69ea (Gluster) in pool
3b8ce946-c4a6-38d0-9484-1495a452e9d2 with size (4.88 GB) 5242880000
2023-06-06 04:26:58,368 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:192e1783) Trying to fetch storage pool
3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 04:26:58,489 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:192e1783) Trying to fetch storage pool
3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 04:26:58,715 WARN [kvm.resource.LibvirtKvmAgentHook]
(agentRequest-Handler-4:null) (logid:192e1783) Groovy script
'/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is not
available. Transformations will not be applied.
2023-06-06 04:26:58,716 WARN [kvm.resource.LibvirtKvmAgentHook]
(agentRequest-Handler-4:null) (logid:192e1783) Groovy scripting engine is not
initialized. Data transformation skipped.
2023-06-06 04:26:58,859 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:2b045fd6) Trying to fetch storage pool
3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 04:26:58,886 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:2b045fd6) Trying to fetch storage pool
3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 04:26:58,924 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:2b045fd6) Creating volume
019b0e20-8bcd-4852-bebe-681489271e2b from template
642b7a0f-5c87-4881-a27c-4fbaa2db34be in pool
3b8ce946-c4a6-38d0-9484-1495a452e9d2 (Gluster) with size (0 bytes) 0
2023-06-06 04:26:58,924 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:2b045fd6) Attempting to create volume
019b0e20-8bcd-4852-bebe-681489271e2b (Gluster) in pool
3b8ce946-c4a6-38d0-9484-1495a452e9d2 with size (4.88 GB) 5242880000
2023-06-06 04:26:59,805 WARN [resource.wrapper.LibvirtStartCommandWrapper]
(agentRequest-Handler-4:null) (logid:192e1783) LibvirtException
org.libvirt.LibvirtException: internal error: process exited while
connecting to monitor: [2023-06-06 04:26:59.440660 +0000] I
[io-stats.c:3711:ios_sample_buf_size_configure] 0-primarys: Configure
ios_sample_buf size is 1024 because ios_sample_interval is 0
2023-06-06T04:26:59.452732Z qemu-system-x86_64: -blockdev
{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage"}:
Could not read L1 table: Input/output error
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.domainCreateXML(Unknown Source)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1815)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:89)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:49)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1847)
at com.cloud.agent.Agent.processRequest(Agent.java:662)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1082)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-06-06 04:26:59,965 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:192e1783) Trying to fetch storage pool
3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 04:27:00,151 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:2b045fd6) Trying to fetch storage pool
3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 04:27:00,229 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:2b045fd6) Trying to fetch storage pool
3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 04:27:00,298 WARN [kvm.resource.LibvirtKvmAgentHook]
(agentRequest-Handler-2:null) (logid:2b045fd6) Groovy script
'/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is not
available. Transformations will not be applied.
2023-06-06 04:27:00,298 WARN [kvm.resource.LibvirtKvmAgentHook]
(agentRequest-Handler-2:null) (logid:2b045fd6) Groovy scripting engine is not
initialized. Data transformation skipped.
2023-06-06 04:27:01,065 WARN [resource.wrapper.LibvirtStartCommandWrapper]
(agentRequest-Handler-2:null) (logid:2b045fd6) LibvirtException
org.libvirt.LibvirtException: internal error: qemu unexpectedly closed the
monitor: [2023-06-06 04:27:00.642128 +0000] I
[io-stats.c:3711:ios_sample_buf_size_configure] 0-primarys: Configure
ios_sample_buf size is 1024 because ios_sample_interval is 0
2023-06-06T04:27:00.653774Z qemu-system-x86_64: -blockdev
{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage"}:
Could not read L1 table: Input/output error
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.domainCreateXML(Unknown Source)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1815)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:89)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:49)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1847)
at com.cloud.agent.Agent.processRequest(Agent.java:662)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1082)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-06-06 04:27:01,232 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:2b045fd6) Trying to fetch storage pool
3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 04:27:05,501 INFO [kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:null) (logid:2b045fd6) No existing libvirtd connection
found. Opening a new one
2023-06-06 04:27:05,505 WARN [kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:null) (logid:2b045fd6) Can not find a connection for
Instance s-1-VM. Assuming the default connection.
2023-06-06 04:27:05,629 WARN [kvm.resource.LibvirtKvmAgentHook]
(agentRequest-Handler-3:null) (logid:2b045fd6) Groovy script
'/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not available.
Transformations will not be applied.
2023-06-06 04:27:05,629 WARN [kvm.resource.LibvirtKvmAgentHook]
(agentRequest-Handler-3:null) (logid:2b045fd6) Groovy scripting engine is not
initialized. Data transformation skipped.
2023-06-06 04:27:06,292 INFO [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:192e1783) No existing libvirtd connection
found. Opening a new one
2023-06-06 04:27:06,295 WARN [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:192e1783) Can not find a connection for
Instance v-2-VM. Assuming the default connection.
2023-06-06 04:27:06,415 WARN [kvm.resource.LibvirtKvmAgentHook]
(agentRequest-Handler-5:null) (logid:192e1783) Groovy script
'/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not available.
Transformations will not be applied.
2023-06-06 04:27:06,416 WARN [kvm.resource.LibvirtKvmAgentHook]
(agentRequest-Handler-5:null) (logid:192e1783) Groovy scripting engine is not
initialized. Data transformation skipped.
2023-06-06 04:27:16,651 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:2b045fd6) Trying to fetch storage pool
3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 04:27:16,735 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:2b045fd6) Trying to fetch storage pool
3b8ce946-c4a6-38d0-9484-1495a452e9d2 from libvirt
2023-06-06 04:27:16,788 WARN [kvm.resource.LibvirtKvmAgentHook]
(agentRequest-Handler-1:null) (logid:2b045fd6) Groovy script
'/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is not
available. Transformations will not be applied.
2023-06-06 04:27:16,789 WARN [kvm.resource.LibvirtKvmAgentHook]
(agentRequest-Handler-1:null) (logid:2b045fd6) Groovy scripting engine is not
initialized. Data transformation skipped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
