Re: Rebooted and now I see unable to find storage pool
Thank you. Looks like a good document. -jeremy > On Jun 14, 2021, at 3:33 PM, Andrija Panic wrote: > > https://www.shapeblue.com/ceph-and-cloudstack-part-1/ (there are 3 parts...) > > You should also read on the client-side caching, etc (on KVM hosts). > > Best, > > On Fri, 11 Jun 2021 at 14:16, Jeremy Hansen wrote: > >> Turns out my other two vm hosts didn’t have rbd/ceph libs installed. This >> fixed my issue. >> >> -jeremy >> >>> On Jun 10, 2021, at 6:50 PM, Jeremy Hansen wrote: >>> >>> 2021-06-10 18:46:19,665 ERROR [kvm.storage.LibvirtStorageAdaptor] >> (agentRequest-Handler-5:null) (logid:34e88890) Failed to create RBD storage >> pool: org.libvirt.LibvirtException: failed to create the RBD IoCTX. Does >> the pool 'rbd' exist?: No such file or directory >>> 2021-06-10 18:46:19,666 ERROR [kvm.storage.LibvirtStorageAdaptor] >> (agentRequest-Handler-5:null) (logid:34e88890) Failed to create the RBD >> storage pool, cleaning up the libvirt secret >>> >>> I should mention that I’ve defined a Ceph RBD primary storage volume. >> Disabling RBD allowed the vm hosts to rejoin the cluster, but I’d like to >> understand what happened here as I plan on using Ceph RBD as my primary >> storage. >>> >>> Thanks >>> -jeremy >>> >>> On Jun 10, 2021, at 6:45 PM, Jeremy Hansen wrote: I removed all of my VMs and all volumes. I rebooted all the servers >> involved in my cluster and now I see this: 2021-06-10 18:41:38,824 WARN [cloud.agent.Agent] >> (agentRequest-Handler-2:null) (logid:4f4da278) Caught: com.cloud.utils.exception.CloudRuntimeException: Failed to create >> storage pool: a6768f2e-3e3c-3aad-938e-83a9efb6deab at >> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645) at >> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329) at >> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323) at >> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42) at >> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35) at >> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) at >> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1643) at com.cloud.agent.Agent.processRequest(Agent.java:661) at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079) at com.cloud.utils.nio.Task.call(Task.java:83) at com.cloud.utils.nio.Task.call(Task.java:29) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at >> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at >> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) and too of my VM hosts are unable to connect. How do I resolve this situation? How did I lose a storage pool? Thanks -jeremy >>> >> >> > > -- > > Andrija Panić signature.asc Description: Message signed with OpenPGP
Re: Rebooted and now I see unable to find storage pool
https://www.shapeblue.com/ceph-and-cloudstack-part-1/ (there are 3 parts...) You should also read on the client-side caching, etc (on KVM hosts). Best, On Fri, 11 Jun 2021 at 14:16, Jeremy Hansen wrote: > Turns out my other two vm hosts didn’t have rbd/ceph libs installed. This > fixed my issue. > > -jeremy > > > On Jun 10, 2021, at 6:50 PM, Jeremy Hansen wrote: > > > > 2021-06-10 18:46:19,665 ERROR [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-5:null) (logid:34e88890) Failed to create RBD storage > pool: org.libvirt.LibvirtException: failed to create the RBD IoCTX. Does > the pool 'rbd' exist?: No such file or directory > > 2021-06-10 18:46:19,666 ERROR [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-5:null) (logid:34e88890) Failed to create the RBD > storage pool, cleaning up the libvirt secret > > > > I should mention that I’ve defined a Ceph RBD primary storage volume. > Disabling RBD allowed the vm hosts to rejoin the cluster, but I’d like to > understand what happened here as I plan on using Ceph RBD as my primary > storage. > > > > Thanks > > -jeremy > > > > > >> On Jun 10, 2021, at 6:45 PM, Jeremy Hansen wrote: > >> > >> > >> I removed all of my VMs and all volumes. I rebooted all the servers > involved in my cluster and now I see this: > >> > >> 2021-06-10 18:41:38,824 WARN [cloud.agent.Agent] > (agentRequest-Handler-2:null) (logid:4f4da278) Caught: > >> com.cloud.utils.exception.CloudRuntimeException: Failed to create > storage pool: a6768f2e-3e3c-3aad-938e-83a9efb6deab > >>at > com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645) > >>at > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329) > >>at > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323) > >>at > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42) > >>at > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35) > >>at > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) > >>at > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1643) > >>at com.cloud.agent.Agent.processRequest(Agent.java:661) > >>at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079) > >>at com.cloud.utils.nio.Task.call(Task.java:83) > >>at com.cloud.utils.nio.Task.call(Task.java:29) > >>at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > >>at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > >>at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > >>at java.base/java.lang.Thread.run(Thread.java:829) > >> > >> and too of my VM hosts are unable to connect. > >> > >> How do I resolve this situation? How did I lose a storage pool? > >> > >> Thanks > >> -jeremy > >> > >> > > > > -- Andrija Panić
Re: Rebooted and now I see unable to find storage pool
Turns out my other two vm hosts didn’t have rbd/ceph libs installed. This fixed my issue. -jeremy > On Jun 10, 2021, at 6:50 PM, Jeremy Hansen wrote: > > 2021-06-10 18:46:19,665 ERROR [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-5:null) (logid:34e88890) Failed to create RBD storage > pool: org.libvirt.LibvirtException: failed to create the RBD IoCTX. Does the > pool 'rbd' exist?: No such file or directory > 2021-06-10 18:46:19,666 ERROR [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-5:null) (logid:34e88890) Failed to create the RBD > storage pool, cleaning up the libvirt secret > > I should mention that I’ve defined a Ceph RBD primary storage volume. > Disabling RBD allowed the vm hosts to rejoin the cluster, but I’d like to > understand what happened here as I plan on using Ceph RBD as my primary > storage. > > Thanks > -jeremy > > >> On Jun 10, 2021, at 6:45 PM, Jeremy Hansen wrote: >> >> >> I removed all of my VMs and all volumes. I rebooted all the servers >> involved in my cluster and now I see this: >> >> 2021-06-10 18:41:38,824 WARN [cloud.agent.Agent] >> (agentRequest-Handler-2:null) (logid:4f4da278) Caught: >> com.cloud.utils.exception.CloudRuntimeException: Failed to create storage >> pool: a6768f2e-3e3c-3aad-938e-83a9efb6deab >>at >> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645) >>at >> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329) >>at >> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323) >>at >> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42) >>at >> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35) >>at >> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) >>at >> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1643) >>at com.cloud.agent.Agent.processRequest(Agent.java:661) >>at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079) >>at com.cloud.utils.nio.Task.call(Task.java:83) >>at com.cloud.utils.nio.Task.call(Task.java:29) >>at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) >>at >> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) >>at >> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) >>at java.base/java.lang.Thread.run(Thread.java:829) >> >> and too of my VM hosts are unable to connect. >> >> How do I resolve this situation? How did I lose a storage pool? >> >> Thanks >> -jeremy >> >> >
Re: Rebooted and now I see unable to find storage pool
2021-06-10 18:46:19,665 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:34e88890) Failed to create RBD storage pool: org.libvirt.LibvirtException: failed to create the RBD IoCTX. Does the pool 'rbd' exist?: No such file or directory 2021-06-10 18:46:19,666 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:34e88890) Failed to create the RBD storage pool, cleaning up the libvirt secret I should mention that I’ve defined a Ceph RBD primary storage volume. Disabling RBD allowed the vm hosts to rejoin the cluster, but I’d like to understand what happened here as I plan on using Ceph RBD as my primary storage. Thanks -jeremy > On Jun 10, 2021, at 6:45 PM, Jeremy Hansen wrote: > > > I removed all of my VMs and all volumes. I rebooted all the servers involved > in my cluster and now I see this: > > 2021-06-10 18:41:38,824 WARN [cloud.agent.Agent] > (agentRequest-Handler-2:null) (logid:4f4da278) Caught: > com.cloud.utils.exception.CloudRuntimeException: Failed to create storage > pool: a6768f2e-3e3c-3aad-938e-83a9efb6deab > at > com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645) > at > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329) > at > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323) > at > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42) > at > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35) > at > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) > at > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1643) > at com.cloud.agent.Agent.processRequest(Agent.java:661) > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079) > at com.cloud.utils.nio.Task.call(Task.java:83) > at com.cloud.utils.nio.Task.call(Task.java:29) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > > and too of my VM hosts are unable to connect. > > How do I resolve this situation? How did I lose a storage pool? > > Thanks > -jeremy > > signature.asc Description: Message signed with OpenPGP
Rebooted and now I see unable to find storage pool
I removed all of my VMs and all volumes. I rebooted all the servers involved in my cluster and now I see this: 2021-06-10 18:41:38,824 WARN [cloud.agent.Agent] (agentRequest-Handler-2:null) (logid:4f4da278) Caught: com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: a6768f2e-3e3c-3aad-938e-83a9efb6deab at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645) at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329) at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1643) at com.cloud.agent.Agent.processRequest(Agent.java:661) at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079) at com.cloud.utils.nio.Task.call(Task.java:83) at com.cloud.utils.nio.Task.call(Task.java:29) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) and too of my VM hosts are unable to connect. How do I resolve this situation? How did I lose a storage pool? Thanks -jeremy signature.asc Description: Message signed with OpenPGP
