kiranchavala opened a new issue, #13740: URL: https://github.com/apache/cloudstack/issues/13740
### problem Kvm host go into alert and connecting state if the ceph storage pool is down ### versions ACS 4.22.1 ### The steps to reproduce the bug 1. Have a cloudstack env with nfs storage and a zone wide ceph storage pool 2. Disable the ceph storage pool from cloudstack 3. Shutdown down the ceph storage servers 4. Reconnect the kvm host The kvm host go into alert and connecting state <img width="1469" height="306" alt="Image" src="https://github.com/user-attachments/assets/7b8c3d46-59bd-4652-b424-d174e996d84b"; /> <img width="1548" height="305" alt="Image" src="https://github.com/user-attachments/assets/93f78878-2914-487c-8a8c-d2f976d80077"; /> Logs MGMT ``` 2026-07-29 07:35:03,055 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-13:[ctx-923461b9]) (logid:9763922f) Notifying other nodes of to disconnect 2026-07-29 07:35:03,055 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-13:[ctx-923461b9]) (logid:9763922f) Failed to handle host connection: com.cloud.exception.ConnectionException: Unable to connect host Host {"id":4,"name":"Cloudstack-Kvm-Host2","type":"Routing","uuid":"d11d3454-5d60-4a2b-a53c-9f34c6cd4cd6"} to storage pool StoragePool {"id":10,"name":"ceph-lab-ec","poolType":"RBD","uuid":"15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f"} due to com.cloud.utils.exception.CloudRuntimeException: Unable to establish connection from storage head to storage pool StoragePool {"id":10,"name":"ceph-lab-ec","poolType":"RBD","uuid":"15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f"} due to Failed to create storage pool: Failed to create storage pool: 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f at com.cloud.storage.listener.StoragePoolMonitor.processConnect(StoragePoolMonitor.java:179) at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:770) at com.cloud.agent.manager.AgentManagerImpl.connectHostAgent(AgentManagerImpl.java:1426) at com.cloud.agent.manager.AgentManagerImpl.sendReadyAndGetAttache(AgentManagerImpl.java:1388) at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1442) at com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1538) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to establish connection from storage head to storage pool StoragePool {"id":10,"name":"ceph-lab-ec","poolType":"RBD","uuid":"15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f"} due to Failed to create storage pool: Failed to create storage pool: 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f at org.apache.cloudstack.storage.datastore.provider.DefaultHostListener.hostConnect(DefaultHostListener.java:155) at org.apache.cloudstack.engine.subsystem.api.storage.HypervisorHostListener.hostConnect(HypervisorHostListener.java:29) at com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(StorageManagerImpl.java:1920) at com.cloud.storage.listener.StoragePoolMonitor.processConnect(StoragePoolMonitor.java:176) ... 13 more 2026-07-29 07:35:03,055 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-13:[ctx-923461b9]) (logid:9763922f) Failed to send ready command: java.nio.channels.ClosedChannelException at com.cloud.utils.nio.Link.send(Link.java:319) at com.cloud.utils.nio.Link.send(Link.java:294) at com.cloud.utils.nio.Link.send(Link.java:290) at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1457) at com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1538) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) ``` Kvm host agent logs ``` 2026-07-29 07:28:38,112 WARN [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:84ffb064) Storage pool 803e29b5-b789-30cd-990c-7bb2fa499b21 was not found running in libvirt. Need to create it. 2026-07-29 07:28:38,112 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:84ffb064) Didn't find an existing storage pool 803e29b5-b789-30cd-990c-7bb2fa499b21 by UUID, checking for pools with duplicate paths 2026-07-29 07:29:08,596 ERROR [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:84ffb064) Failed to create RBD storage pool: org.libvirt.LibvirtException: failed to connect to the RADOS monitor on: 192.168.55.245,: No such file or directory 2026-07-29 07:29:08,596 ERROR [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:84ffb064) Failed to create the RBD storage pool, cleaning up the libvirt secret 2026-07-29 07:29:08,653 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:84ffb064) Trying to fetch storage pool 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f from libvirt 2026-07-29 07:29:08,654 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:84ffb064) Asking libvirt to refresh storage pool 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f 2026-07-29 07:29:38,657 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:84ffb064) Attempting to create storage pool 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f (RBD) in libvirt 2026-07-29 07:29:38,657 WARN [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:84ffb064) Storage pool 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f was not found running in libvirt. Need to create it. 2026-07-29 07:29:38,657 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:84ffb064) Didn't find an existing storage pool 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f by UUID, checking for pools with duplicate paths 2026-07-29 07:29:44,092 INFO [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Detected agent disconnect event, running through 0 disconnect hooks 2026-07-29 07:30:50,799 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:[]) (logid:) Found existing defined storage pool b4b91040-eea6-4c03-ab8a-8bc21ab1d154, using it. 2026-07-29 07:30:50,799 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:[]) (logid:) Trying to fetch storage pool b4b91040-eea6-4c03-ab8a-8bc21ab1d154 from libvirt 2026-07-29 07:30:51,811 INFO [cloud.agent.Agent] (Agent-Handler-2:[]) (logid:) Process agent startup answer, agent [id: 0, uuid: , name: ] connected to the server 2026-07-29 07:30:51,823 INFO [cloud.agent.Agent] (Agent-Handler-2:[]) (logid:) Startup Response Received: agent [id: 0, uuid: , name: ] 2026-07-29 07:30:52,107 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:7ee7ba43) Attempting to create storage pool e78e94d1-b759-3c4c-ab13-70e231006e0c (NetworkFilesystem) in libvirt 2026-07-29 07:30:52,108 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:7ee7ba43) Found existing defined storage pool e78e94d1-b759-3c4c-ab13-70e231006e0c, using it. 2026-07-29 07:30:52,110 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:7ee7ba43) Trying to fetch storage pool e78e94d1-b759-3c4c-ab13-70e231006e0c from libvirt 2026-07-29 07:30:52,165 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:7ee7ba43) Attempting to create storage pool e7409462-25e4-3184-9bee-33aa50363ccd (NetworkFilesystem) in libvirt 2026-07-29 07:30:52,165 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:7ee7ba43) Found existing defined storage pool e7409462-25e4-3184-9bee-33aa50363ccd, using it. 2026-07-29 07:30:52,167 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:7ee7ba43) Trying to fetch storage pool e7409462-25e4-3184-9bee-33aa50363ccd from libvirt 2026-07-29 07:30:52,261 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:7ee7ba43) Attempting to create storage pool 803e29b5-b789-30cd-990c-7bb2fa499b21 (RBD) in libvirt 2026-07-29 07:30:52,261 WARN [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:7ee7ba43) Storage pool 803e29b5-b789-30cd-990c-7bb2fa499b21 was not found running in libvirt. Need to create it. 2026-07-29 07:30:52,261 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:7ee7ba43) Didn't find an existing storage pool 803e29b5-b789-30cd-990c-7bb2fa499b21 by UUID, checking for pools with duplicate paths 2026-07-29 07:30:55,440 INFO [cloud.agent.Agent] (Agent-Handler-1:[]) (logid:) Connected to the host: 192.168.55.217 ``` Issue is present even if the cloudstack-agent is restarted ### What to do about it? The kvm host should be in up state even if the ceph storage pool is down as there is nfs storage available -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
