Hello Andrija Presently, CloudStack's host lists exhibited stability prior to the disaster, but their statuses are currently fluctuating continuously. Some hosts are initially marked as disconnected, but after a period, they transition to a connected state."
[image: image.png] *Using virsh we are getting VM status on cshost1 as below* root@cshost1:~# virsh list Id Name State ------------------------------- 10 i-14-597-VM running 61 r-757-VM running 69 i-24-767-VM running 76 r-71-VM running 82 i-24-797-VM running 113 r-335-VM running 128 r-577-VM running 148 i-14-1151-VM running 164 i-2-1253-VM running Regards Mosharaf Hossain Manager, Product Development IT Division Bangladesh Export Import Company Ltd. Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757 Cell: +8801787680828, Email: mosharaf.hoss...@bol-online.com, Web: www.bol-online.com <https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts&ust=1557908951423000&usg=AFQjCNGMxIuHSHsD3qO6y5JddpEZ0S592A> On Mon, Sep 18, 2023 at 12:43 PM Andrija Panic <andrija.pa...@gmail.com> wrote: > Hi, > > the message " Agent-Handler-1:null) (logid:) Connection with libvirtd is > broken: invalid connection pointer in virConnectGetVersion " - is a false > alarm and does NOT means any errors actually. > > I can see that ACS agent sees different storage pools - namely > "daab90ad-42d3-3c48-a9e4-b4c3c7fcdc84" and > "a2d455c6-68cb-303f-a7fa-287e62a5be9c" - and I don't see any explicit error > message about these 2 pools (both RBD/Ceph) pools. > > Also I can see that the cloudstack agent says it's connected to the mgmt > host - which means that all pools are in place (otherwise the agent would > not connect) > > 1. Are you KVM hosts all green when checking in CloudStack UI > (Connected/Up)? > 2. You can always use virsh to list pools and see if they are there > > Best, > > On Wed, 13 Sept 2023 at 13:54, Mosharaf Hossain < > mosharaf.hoss...@bol-online.com> wrote: > >> Hello Folks >> We've recently performed an upgrade on our Cephadm cluster, transitioning >> from Ceph Quiency to Reef. However, following the manual implementation of >> a read balancer in the Reef cluster, we've experienced a significant >> slowdown in client I/O operations within the Ceph cluster, affecting both >> client bandwidth and overall cluster performance. >> >> This slowdown has resulted in unresponsiveness across all virtual machines >> within the cluster, despite the fact that the cluster exclusively utilizes >> SSD storage." >> >> In the CloudStack agent, we are getting libvirrt can't connect to CEPH >> pool >> and generating an error message. >> >> 2023-09-13 16:57:51,660 INFO [cloud.agent.Agent] (Agent-Handler-4:null) >> (logid:) Lost connection to host: 10.10.11.61. Attempting reconnection >> while we still have 1 command in progress. >> 2023-09-13 16:57:51,661 INFO [utils.nio.NioClient] (Agent-Handler-4:null) >> (logid:) NioClient connection closed >> 2023-09-13 16:57:51,662 INFO [cloud.agent.Agent] (Agent-Handler-4:null) >> (logid:) Reconnecting to host:10.10.11.62 >> 2023-09-13 16:57:51,662 INFO [utils.nio.NioClient] (Agent-Handler-4:null) >> (logid:) Connecting to 10.10.11.62:8250 >> 2023-09-13 16:57:51,663 INFO [utils.nio.Link] (Agent-Handler-4:null) >> (logid:) Conf file found: /etc/cloudstack/agent/agent.properties >> 2023-09-13 16:57:51,779 INFO [utils.nio.NioClient] (Agent-Handler-4:null) >> (logid:) SSL: Handshake done >> 2023-09-13 16:57:51,779 INFO [utils.nio.NioClient] (Agent-Handler-4:null) >> (logid:) Connected to 10.10.11.62:8250 >> 2023-09-13 16:57:51,815 INFO [utils.linux.KVMHostInfo] >> (Agent-Handler-1:null) (logid:) Fetching CPU speed from command "lscpu". >> 2023-09-13 16:57:51,836 INFO [utils.linux.KVMHostInfo] >> (Agent-Handler-1:null) (logid:) Command [lscpu | grep -i 'Model name' | >> head -n 1 | egrep -o '[[:digit:]].[[:digit:]]+GHz' | sed 's/GHz//g'] >> resulted in the value [2100] for CPU speed. >> 2023-09-13 16:57:51,900 INFO [kvm.storage.LibvirtStorageAdaptor] >> (Agent-Handler-1:null) (logid:) Attempting to create storage pool >> e205cf5f-ea32-46c7-ba18-d18f62772b80 (Filesystem) in libvirt >> 2023-09-13 16:57:51,901 ERROR [kvm.resource.LibvirtConnection] >> (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: >> invalid >> connection pointer in virConnectGetVersion >> 2023-09-13 16:57:51,903 INFO [kvm.storage.LibvirtStorageAdaptor] >> (Agent-Handler-1:null) (logid:) Found existing defined storage pool >> e205cf5f-ea32-46c7-ba18-d18f62772b80, using it. >> 2023-09-13 16:57:51,904 INFO [kvm.storage.LibvirtStorageAdaptor] >> (Agent-Handler-1:null) (logid:) Trying to fetch storage pool >> e205cf5f-ea32-46c7-ba18-d18f62772b80 from libvirt >> 2023-09-13 16:57:51,924 INFO [cloud.agent.Agent] (Agent-Handler-2:null) >> (logid:) Process agent startup answer, agent id = 0 >> 2023-09-13 16:57:51,924 INFO [cloud.agent.Agent] (Agent-Handler-2:null) >> (logid:) Set agent id 0 >> 2023-09-13 16:57:51,955 INFO [cloud.agent.Agent] (Agent-Handler-2:null) >> (logid:) Startup Response Received: agent id = 0 >> 2023-09-13 16:57:52,047 INFO [kvm.storage.LibvirtStorageAdaptor] >> (agentRequest-Handler-5:null) (logid:e396a97c) Attempting to create >> storage >> pool daab90ad-42d3-3c48-a9e4-b4c3c7fcdc84 (RBD) in libvirt >> 2023-09-13 16:57:52,050 INFO [kvm.storage.LibvirtStorageAdaptor] >> (agentRequest-Handler-5:null) (logid:e396a97c) Found existing defined >> storage pool daab90ad-42d3-3c48-a9e4-b4c3c7fcdc84, using it. >> 2023-09-13 16:57:52,050 INFO [kvm.storage.LibvirtStorageAdaptor] >> (agentRequest-Handler-5:null) (logid:e396a97c) Trying to fetch storage >> pool >> daab90ad-42d3-3c48-a9e4-b4c3c7fcdc84 from libvirt >> 2023-09-13 16:57:52,161 INFO [kvm.storage.LibvirtStorageAdaptor] >> (agentRequest-Handler-1:null) (logid:e396a97c) Attempting to create >> storage >> pool a2d455c6-68cb-303f-a7fa-287e62a5be9c (RBD) in libvirt >> 2023-09-13 16:57:52,163 WARN [kvm.storage.LibvirtStorageAdaptor] >> (agentRequest-Handler-1:null) (logid:e396a97c) Storage pool >> a2d455c6-68cb-303f-a7fa-287e62a5be9c was not found running in libvirt. >> Need >> to create it. >> 2023-09-13 16:57:52,164 INFO [kvm.storage.LibvirtStorageAdaptor] >> (agentRequest-Handler-1:null) (logid:e396a97c) Didn't find an existing >> storage pool a2d455c6-68cb-303f-a7fa-287e62a5be9c by UUID, checking for >> pools with duplicate paths >> 2023-09-13 16:57:56,780 INFO [cloud.agent.Agent] (Agent-Handler-4:null) >> (logid:) Connected to the host: 10.10.11.62 >> ^C >> >> >> Kindly guide us to move forward. >> >> >> >> Regards >> Mosharaf Hossain >> Manager, Product Development >> IT Division >> >> Bangladesh Export Import Company Ltd. >> >> Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh >> >> Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757 >> >> Cell: +8801787680828, Email: mosharaf.hoss...@bol-online.com, Web: >> www.bol-online.com >> < >> https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts&ust=1557908951423000&usg=AFQjCNGMxIuHSHsD3qO6y5JddpEZ0S592A >> > >> > > > -- > > Andrija Panić >
