Turns out my other two vm hosts didn’t have rbd/ceph libs installed. This fixed my issue.
-jeremy > On Jun 10, 2021, at 6:50 PM, Jeremy Hansen <jer...@skidrow.la> wrote: > > 2021-06-10 18:46:19,665 ERROR [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-5:null) (logid:34e88890) Failed to create RBD storage > pool: org.libvirt.LibvirtException: failed to create the RBD IoCTX. Does the > pool 'rbd' exist?: No such file or directory > 2021-06-10 18:46:19,666 ERROR [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-5:null) (logid:34e88890) Failed to create the RBD > storage pool, cleaning up the libvirt secret > > I should mention that I’ve defined a Ceph RBD primary storage volume. > Disabling RBD allowed the vm hosts to rejoin the cluster, but I’d like to > understand what happened here as I plan on using Ceph RBD as my primary > storage. > > Thanks > -jeremy > > >> On Jun 10, 2021, at 6:45 PM, Jeremy Hansen <jer...@skidrow.la> wrote: >> >> >> I removed all of my VMs and all volumes. I rebooted all the servers >> involved in my cluster and now I see this: >> >> 2021-06-10 18:41:38,824 WARN [cloud.agent.Agent] >> (agentRequest-Handler-2:null) (logid:4f4da278) Caught: >> com.cloud.utils.exception.CloudRuntimeException: Failed to create storage >> pool: a6768f2e-3e3c-3aad-938e-83a9efb6deab >> at >> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645) >> at >> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329) >> at >> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323) >> at >> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42) >> at >> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35) >> at >> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) >> at >> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1643) >> at com.cloud.agent.Agent.processRequest(Agent.java:661) >> at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079) >> at com.cloud.utils.nio.Task.call(Task.java:83) >> at com.cloud.utils.nio.Task.call(Task.java:29) >> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) >> at >> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) >> at >> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) >> at java.base/java.lang.Thread.run(Thread.java:829) >> >> and too of my VM hosts are unable to connect. >> >> How do I resolve this situation? How did I lose a storage pool? >> >> Thanks >> -jeremy >> >> >