GitHub user GerorgeEG created a discussion: KVM host declared down and then getting stuck in connecting state in ACS
### problem We experienced issue atleast 2-3 times where KVM host declared down by ACS and then it gets stuck in "Connecting" state in ACS. We have seen this issue in our 2 environments and it is production impacting issue. KVM cloudstack agent logs ========= `2026-05-18 22:10:49,716 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-3:[]) (logid:) Trying to fetch storage pool 15cf3201-666a-3b51-8c73-233589579245 from libvirt 2026-05-18 22:10:50,133 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:) Trying to fetch storage pool 15cf3201-666a-3b51-8c73-233589579245 from libvirt 2026-05-18 22:10:51,699 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:) Asking libvirt to refresh storage pool 15cf3201-666a-3b51-8c73-233589579245 2026-05-18 22:12:58,808 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:) Trying to fetch storage pool 59139077-6f99-32a5-886b-852149d9bf36 from libvirt 2026-05-18 22:12:58,821 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:) Asking libvirt to refresh storage pool 59139077-6f99-32a5-886b-852149d9bf36 2026-05-18 22:13:06,637 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-1:[]) (logid:) Trying to fetch storage pool 15cf3201-666a-3b51-8c73-233589579245 from libvirt 2026-05-18 22:17:13,430 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:) Trying to fetch storage pool e0709dcf-1389-36b6-a5ed-4a1de4af2da7 from libvirt 2026-05-18 22:17:16,806 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:) Trying to fetch storage pool e0709dcf-1389-36b6-a5ed-4a1de4af2da7 from libvirt 2026-05-18 22:17:20,213 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:) Trying to fetch storage pool e0709dcf-1389-36b6-a5ed-4a1de4af2da7 from libvirt 2026-05-18 22:17:23,401 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:) Trying to fetch storage pool e0709dcf-1389-36b6-a5ed-4a1de4af2da7 from libvirt 2026-05-18 22:17:30,474 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:) Trying to fetch storage pool a9c72063-59b8-3381-b091-33efea922438 from libvirt 2026-05-18 22:17:30,513 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:) Trying to fetch storage pool a9c72063-59b8-3381-b091-33efea922438 from libvirt 2026-05-18 22:17:30,540 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:) Trying to fetch storage pool a9c72063-59b8-3381-b091-33efea922438 from libvirt 2026-05-18 22:17:30,566 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:) Trying to fetch storage pool a9c72063-59b8-3381-b091-33efea922438 from libvirt 2026-05-18 22:17:30,591 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:) Trying to fetch storage pool a9c72063-59b8-3381-b091-33efea922438 from libvirt 2026-05-18 22:17:30,617 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:) Trying to fetch storage pool a9c72063-59b8-3381-b091-33efea922438 from libvirt 2026-05-18 22:17:30,653 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:) Trying to fetch storage pool a9c72063-59b8-3381-b091-33efea922438 from libvirt 2026-05-18 22:17:30,688 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:) Trying to fetch storage pool a9c72063-59b8-3381-b091-33efea922438 from libvirt 2026-05-18 22:17:39,513 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:) Trying to fetch storage pool 15cf3201-666a-3b51-8c73-233589579245 from libvirt 2026-05-18 22:22:33,008 INFO [cloud.agent.Agent] (Agent-Handler-4:[]) (logid:8dfd4ff2) Lost connection to host: x.x.x.x. Attempting reconnection while we still have 5 commands in progress. 2026-05-18 22:22:33,009 INFO [utils.nio.NioClient] (Agent-Handler-4:[]) (logid:8dfd4ff2) NioClient connection closed 2026-05-18 22:22:38,009 INFO [cloud.agent.Agent] (Agent-Handler-4:[]) (logid:8dfd4ff2) Reconnecting to host: x.x.x.x 2026-05-18 22:22:38,009 INFO [utils.nio.NioClient] (Agent-Handler-4:[]) (logid:8dfd4ff2) Connecting to x.x.x.x:8250 2026-05-18 22:22:38,010 INFO [utils.nio.NioClient] (Agent-Handler-4:[]) (logid:8dfd4ff2) Connected to x.x.x.x:8250 2026-05-18 22:22:38,010 INFO [utils.nio.Link] (Agent-Handler-4:[]) (logid:8dfd4ff2) Conf file found: /etc/cloudstack/agent/agent.properties 2026-05-18 22:22:38,081 INFO [utils.nio.NioClient] (Agent-Handler-4:[]) (logid:8dfd4ff2) SSL: Handshake done 2026-05-18 22:22:39,384 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-1:[]) (logid:) Asking libvirt to refresh storage pool 15cf3201-666a-3b51-8c73-233589579245 2026-05-18 22:22:39,384 WARN [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:) Unable to send response: null 2026-05-18 22:22:39,395 WARN [cloud.agent.Agent] (AgentOutRequest-Handler-5:[]) (logid:) Unable to send request to /x.x.x.x:8250 due to 'null', request: null 2026-05-18 22:22:39,395 WARN [cloud.agent.Agent] (AgentOutRequest-Handler-1:[]) (logid:) Unable to send request to /x.x.x.x:8250 due to 'null', request: null 2026-05-18 22:22:39,405 WARN [cloud.agent.Agent] (AgentOutRequest-Handler-2:[]) (logid:) Unable to send request to /x.x.x.x:8250 due to 'null', request: null 2026-05-18 22:22:39,417 WARN [cloud.agent.Agent] (AgentOutRequest-Handler-3:[]) (logid:) Unable to send request to /x.x.x.x:8250 due to 'null', request: null 2026-05-18 22:22:39,444 WARN [cloud.agent.Agent] (AgentOutRequest-Handler-4:[]) (logid:) Unable to send request to /x.x.x.x:8250 due to 'null', request: null` ### versions ACS version post-upgrade : 4.20.2.0 (upgraded from 4.19.1.2 few weeks before) Management and DB OS : RHEL 8 KVM : RHEL 8 ### The steps to reproduce the bug 1. It is generated by ACS on its own not able to regenerate 2. 3. ... ### What to do about it? KVM host gets reconnected on its own after 1 hour. GitHub link: https://github.com/apache/cloudstack/discussions/13258 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
