Re: Ceph RBD Connection
Ok, so I figured out my problem after digging deeper into troubleshooting. My issue was just that ceph monitor is not listening to port 6789 on all interfaces. Its just listening to it on the network address 10.0.0.5. Changing the pool configuration to account for that fixed my issue. I wrote a simple c program based off documentation at https://docs.ceph.com/en/latest/rados/api/librados/ which is where it hit me after trying to use `rbd -m localhost:6789 list libvirt-pool`
Re: Ceph RBD Connection
On 5/23/21 8:48 PM, Han Han wrote: You can check the following: 1. Make sure the ceph mon node are running on the localhost:6789 root@server ~# netstat -pan | grep 6789 tcp 0 0 10.0.0.5:6789 0.0.0.0:* LISTEN 2087/ceph-mon 2. Check if the ceph authorization is enabled on the ceph cluster. Check if authorization is used in rbd pool xml. You can provide more details on the rbd pool xml, ceph conf, and the version of librbd1(client side), ceph(server side). root@server ~# ceph auth get client.libvirt exported keyring for client.libvirt [client.libvirt] key = AQDtM6lgpebpIoIKaFXQSTAg== caps mon = "profile rbd" caps osd = "profile rbd pool=libvirt-pool" root@server ~# virsh secret-list UUID Usage f290cb3a-a9de-48d7-89ce-09311eadff4a ceph client.libvirt secret root@server ~# virsh secret-dumpxml f290cb3a-a9de-48d7-89ce-09311eadff4a f290cb3a-a9de-48d7-89ce-09311eadff4a client.libvirt secret root@server ~# virsh pool-dumpxml libvirt-pool libvirt-pool 5ae8abb9-31ab-44c0-a708-072b95182018 0 0 0 libvirt-pool root@server ~# ceph versions { "mon": { "ceph version 15.2.12 (ce065eabfa5ce81323b009786bdf5bb03127cbe1) octopus (stable)": 2 }, "mgr": { "ceph version 15.2.12 (ce065eabfa5ce81323b009786bdf5bb03127cbe1) octopus (stable)": 2 }, "osd": { "ceph version 15.2.12 (ce065eabfa5ce81323b009786bdf5bb03127cbe1) octopus (stable)": 2 }, "mds": {}, "rgw": { "ceph version 15.2.12 (ce065eabfa5ce81323b009786bdf5bb03127cbe1) octopus (stable)": 1 }, "overall": { "ceph version 15.2.12 (ce065eabfa5ce81323b009786bdf5bb03127cbe1) octopus (stable)": 7 } } root@server ~# pkgfile /usr/lib/librbd.so.1.12.0 community/ceph-libs root@server ~# pacsearch ceph-libs community/ceph-libs 15.2.12-1 [installed] Distributed, fault-tolerant storage platform delivering object, block, and file system Sorry I should have included this information in my original email. I followed this guide (https://blog.modest-destiny.com/posts/kvm-libvirt-add-ceph-rbd-pool/) to setup the above configuration and I also tried following the guide at https://docs.ceph.com/en/latest/rbd/libvirt/
Re: Ceph RBD Connection
On Sun, May 23, 2021 at 4:33 AM Mr. Gecko wrote: > Hello, > > I setup a Ceph Cluster on my system in hopes of using it with libvirtd, > however I'm finding myself unable to have libvirtd make the connection. > > [root@server ~]# virsh pool-start "${CEPH_POOL}" > error: Failed to start pool libvirt-pool > error: failed to connect to the RADOS monitor on: localhost:6789,: No > such file or directory > > I know that rbd is working as I am able to use it with the rbd utility. > > [root@server ~]# rbd list libvirt-pool > Kolab > [root@server ~]# rbd device map libvirt-pool/Kolab > /dev/rbd0 > [root@server ~]# lsblk /dev/rbd0 > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > rbd0 253:00 20G 0 disk > ├─rbd0p1 253:10 1G 0 part > └─rbd0p2 253:20 19G 0 part > > The libvirtd logs are not providing any useful information on the issue, > I tried changing log-level to 1 and enabling the log_filters/adding > "1:rbd" to the log filters. > > I have tried both the libvirt version 7.3.0 in my repository and the > bleeding edge with a compile from git. Both returns the same error. > > You can check the following: 1. Make sure the ceph mon node are running on the localhost:6789 2. Check if the ceph authorization is enabled on the ceph cluster. Check if authorization is used in rbd pool xml. You can provide more details on the rbd pool xml, ceph conf, and the version of librbd1(client side), ceph(server side). > Any help in troubleshooting this issue? I'm somewhat new when it comes > to all of this. > > Thank you, > > James Coleman > >
Ceph RBD Connection
Hello, I setup a Ceph Cluster on my system in hopes of using it with libvirtd, however I'm finding myself unable to have libvirtd make the connection. [root@server ~]# virsh pool-start "${CEPH_POOL}" error: Failed to start pool libvirt-pool error: failed to connect to the RADOS monitor on: localhost:6789,: No such file or directory I know that rbd is working as I am able to use it with the rbd utility. [root@server ~]# rbd list libvirt-pool Kolab [root@server ~]# rbd device map libvirt-pool/Kolab /dev/rbd0 [root@server ~]# lsblk /dev/rbd0 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT rbd0 253:0 0 20G 0 disk ├─rbd0p1 253:1 0 1G 0 part └─rbd0p2 253:2 0 19G 0 part The libvirtd logs are not providing any useful information on the issue, I tried changing log-level to 1 and enabling the log_filters/adding "1:rbd" to the log filters. I have tried both the libvirt version 7.3.0 in my repository and the bleeding edge with a compile from git. Both returns the same error. Any help in troubleshooting this issue? I'm somewhat new when it comes to all of this. Thank you, James Coleman