Hi, ok, thx for the info.
I meant to use virsh to list pools (storage pools), not VMs - to see if the storage pools are created inside libvirt. Best, On Tue, 19 Sept 2023 at 08:25, Mosharaf Hossain < mosharaf.hoss...@bol-online.com> wrote: > Hello Andrija > > Presently, CloudStack's host lists exhibited stability prior to the > disaster, but their statuses are currently fluctuating continuously. Some > hosts are initially marked as disconnected, but after a period, they > transition to a connected state." > > > > > [image: image.png] > > *Using virsh we are getting VM status on cshost1 as below* > root@cshost1:~# virsh list > Id Name State > ------------------------------- > 10 i-14-597-VM running > 61 r-757-VM running > 69 i-24-767-VM running > 76 r-71-VM running > 82 i-24-797-VM running > 113 r-335-VM running > 128 r-577-VM running > 148 i-14-1151-VM running > 164 i-2-1253-VM running > > > Regards > Mosharaf Hossain > Manager, Product Development > IT Division > > Bangladesh Export Import Company Ltd. > > Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh > > Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757 > > Cell: +8801787680828, Email: mosharaf.hoss...@bol-online.com, Web: > www.bol-online.com > > <https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts&ust=1557908951423000&usg=AFQjCNGMxIuHSHsD3qO6y5JddpEZ0S592A> > > > > On Mon, Sep 18, 2023 at 12:43 PM Andrija Panic <andrija.pa...@gmail.com> > wrote: > >> Hi, >> >> the message " Agent-Handler-1:null) (logid:) Connection with libvirtd is >> broken: invalid connection pointer in virConnectGetVersion " - is a false >> alarm and does NOT means any errors actually. >> >> I can see that ACS agent sees different storage pools - namely >> "daab90ad-42d3-3c48-a9e4-b4c3c7fcdc84" and >> "a2d455c6-68cb-303f-a7fa-287e62a5be9c" - and I don't see any explicit error >> message about these 2 pools (both RBD/Ceph) pools. >> >> Also I can see that the cloudstack agent says it's connected to the mgmt >> host - which means that all pools are in place (otherwise the agent would >> not connect) >> >> 1. Are you KVM hosts all green when checking in CloudStack UI >> (Connected/Up)? >> 2. You can always use virsh to list pools and see if they are there >> >> Best, >> >> On Wed, 13 Sept 2023 at 13:54, Mosharaf Hossain < >> mosharaf.hoss...@bol-online.com> wrote: >> >>> Hello Folks >>> We've recently performed an upgrade on our Cephadm cluster, transitioning >>> from Ceph Quiency to Reef. However, following the manual implementation >>> of >>> a read balancer in the Reef cluster, we've experienced a significant >>> slowdown in client I/O operations within the Ceph cluster, affecting both >>> client bandwidth and overall cluster performance. >>> >>> This slowdown has resulted in unresponsiveness across all virtual >>> machines >>> within the cluster, despite the fact that the cluster exclusively >>> utilizes >>> SSD storage." >>> >>> In the CloudStack agent, we are getting libvirrt can't connect to CEPH >>> pool >>> and generating an error message. >>> >>> 2023-09-13 16:57:51,660 INFO [cloud.agent.Agent] (Agent-Handler-4:null) >>> (logid:) Lost connection to host: 10.10.11.61. Attempting reconnection >>> while we still have 1 command in progress. >>> 2023-09-13 16:57:51,661 INFO [utils.nio.NioClient] >>> (Agent-Handler-4:null) >>> (logid:) NioClient connection closed >>> 2023-09-13 16:57:51,662 INFO [cloud.agent.Agent] (Agent-Handler-4:null) >>> (logid:) Reconnecting to host:10.10.11.62 >>> 2023-09-13 16:57:51,662 INFO [utils.nio.NioClient] >>> (Agent-Handler-4:null) >>> (logid:) Connecting to 10.10.11.62:8250 >>> 2023-09-13 16:57:51,663 INFO [utils.nio.Link] (Agent-Handler-4:null) >>> (logid:) Conf file found: /etc/cloudstack/agent/agent.properties >>> 2023-09-13 16:57:51,779 INFO [utils.nio.NioClient] >>> (Agent-Handler-4:null) >>> (logid:) SSL: Handshake done >>> 2023-09-13 16:57:51,779 INFO [utils.nio.NioClient] >>> (Agent-Handler-4:null) >>> (logid:) Connected to 10.10.11.62:8250 >>> 2023-09-13 16:57:51,815 INFO [utils.linux.KVMHostInfo] >>> (Agent-Handler-1:null) (logid:) Fetching CPU speed from command "lscpu". >>> 2023-09-13 16:57:51,836 INFO [utils.linux.KVMHostInfo] >>> (Agent-Handler-1:null) (logid:) Command [lscpu | grep -i 'Model name' | >>> head -n 1 | egrep -o '[[:digit:]].[[:digit:]]+GHz' | sed 's/GHz//g'] >>> resulted in the value [2100] for CPU speed. >>> 2023-09-13 16:57:51,900 INFO [kvm.storage.LibvirtStorageAdaptor] >>> (Agent-Handler-1:null) (logid:) Attempting to create storage pool >>> e205cf5f-ea32-46c7-ba18-d18f62772b80 (Filesystem) in libvirt >>> 2023-09-13 16:57:51,901 ERROR [kvm.resource.LibvirtConnection] >>> (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: >>> invalid >>> connection pointer in virConnectGetVersion >>> 2023-09-13 16:57:51,903 INFO [kvm.storage.LibvirtStorageAdaptor] >>> (Agent-Handler-1:null) (logid:) Found existing defined storage pool >>> e205cf5f-ea32-46c7-ba18-d18f62772b80, using it. >>> 2023-09-13 16:57:51,904 INFO [kvm.storage.LibvirtStorageAdaptor] >>> (Agent-Handler-1:null) (logid:) Trying to fetch storage pool >>> e205cf5f-ea32-46c7-ba18-d18f62772b80 from libvirt >>> 2023-09-13 16:57:51,924 INFO [cloud.agent.Agent] (Agent-Handler-2:null) >>> (logid:) Process agent startup answer, agent id = 0 >>> 2023-09-13 16:57:51,924 INFO [cloud.agent.Agent] (Agent-Handler-2:null) >>> (logid:) Set agent id 0 >>> 2023-09-13 16:57:51,955 INFO [cloud.agent.Agent] (Agent-Handler-2:null) >>> (logid:) Startup Response Received: agent id = 0 >>> 2023-09-13 16:57:52,047 INFO [kvm.storage.LibvirtStorageAdaptor] >>> (agentRequest-Handler-5:null) (logid:e396a97c) Attempting to create >>> storage >>> pool daab90ad-42d3-3c48-a9e4-b4c3c7fcdc84 (RBD) in libvirt >>> 2023-09-13 16:57:52,050 INFO [kvm.storage.LibvirtStorageAdaptor] >>> (agentRequest-Handler-5:null) (logid:e396a97c) Found existing defined >>> storage pool daab90ad-42d3-3c48-a9e4-b4c3c7fcdc84, using it. >>> 2023-09-13 16:57:52,050 INFO [kvm.storage.LibvirtStorageAdaptor] >>> (agentRequest-Handler-5:null) (logid:e396a97c) Trying to fetch storage >>> pool >>> daab90ad-42d3-3c48-a9e4-b4c3c7fcdc84 from libvirt >>> 2023-09-13 16:57:52,161 INFO [kvm.storage.LibvirtStorageAdaptor] >>> (agentRequest-Handler-1:null) (logid:e396a97c) Attempting to create >>> storage >>> pool a2d455c6-68cb-303f-a7fa-287e62a5be9c (RBD) in libvirt >>> 2023-09-13 16:57:52,163 WARN [kvm.storage.LibvirtStorageAdaptor] >>> (agentRequest-Handler-1:null) (logid:e396a97c) Storage pool >>> a2d455c6-68cb-303f-a7fa-287e62a5be9c was not found running in libvirt. >>> Need >>> to create it. >>> 2023-09-13 16:57:52,164 INFO [kvm.storage.LibvirtStorageAdaptor] >>> (agentRequest-Handler-1:null) (logid:e396a97c) Didn't find an existing >>> storage pool a2d455c6-68cb-303f-a7fa-287e62a5be9c by UUID, checking for >>> pools with duplicate paths >>> 2023-09-13 16:57:56,780 INFO [cloud.agent.Agent] (Agent-Handler-4:null) >>> (logid:) Connected to the host: 10.10.11.62 >>> ^C >>> >>> >>> Kindly guide us to move forward. >>> >>> >>> >>> Regards >>> Mosharaf Hossain >>> Manager, Product Development >>> IT Division >>> >>> Bangladesh Export Import Company Ltd. >>> >>> Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh >>> >>> Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757 >>> >>> Cell: +8801787680828, Email: mosharaf.hoss...@bol-online.com, Web: >>> www.bol-online.com >>> < >>> https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts&ust=1557908951423000&usg=AFQjCNGMxIuHSHsD3qO6y5JddpEZ0S592A >>> > >>> >> >> >> -- >> >> Andrija Panić >> > -- Andrija Panić
