Re: Ceph RBD Connection
Ok, so I figured out my problem after digging deeper into troubleshooting. My issue was just that ceph monitor is not listening to port 6789 on all interfaces. Its just listening to it on the network address 10.0.0.5. Changing the pool configuration to account for that fixed my issue. I wrote a simple c program based off documentation at https://docs.ceph.com/en/latest/rados/api/librados/ which is where it hit me after trying to use `rbd -m localhost:6789 list libvirt-pool`
Re: Ceph RBD Connection
On 5/23/21 8:48 PM, Han Han wrote:
You can check the following:
1. Make sure the ceph mon node are running on the localhost:6789
root@server ~# netstat -pan | grep 6789
tcp 0 0 10.0.0.5:6789 0.0.0.0:* LISTEN
2087/ceph-mon
2. Check if the ceph authorization is enabled on the ceph cluster.
Check if authorization is
used in rbd pool xml.
You can provide more details on the rbd pool xml, ceph conf, and the
version of librbd1(client side), ceph(server side).
root@server ~# ceph auth get client.libvirt
exported keyring for client.libvirt
[client.libvirt]
key = AQDtM6lgpebpIoIKaFXQSTAg==
caps mon = "profile rbd"
caps osd = "profile rbd pool=libvirt-pool"
root@server ~# virsh secret-list
UUID Usage
f290cb3a-a9de-48d7-89ce-09311eadff4a ceph client.libvirt secret
root@server ~# virsh secret-dumpxml f290cb3a-a9de-48d7-89ce-09311eadff4a
f290cb3a-a9de-48d7-89ce-09311eadff4a
client.libvirt secret
root@server ~# virsh pool-dumpxml libvirt-pool
libvirt-pool
5ae8abb9-31ab-44c0-a708-072b95182018
0
0
0
libvirt-pool
root@server ~# ceph versions
{
"mon": {
"ceph version 15.2.12
(ce065eabfa5ce81323b009786bdf5bb03127cbe1) octopus (stable)": 2
},
"mgr": {
"ceph version 15.2.12
(ce065eabfa5ce81323b009786bdf5bb03127cbe1) octopus (stable)": 2
},
"osd": {
"ceph version 15.2.12
(ce065eabfa5ce81323b009786bdf5bb03127cbe1) octopus (stable)": 2
},
"mds": {},
"rgw": {
"ceph version 15.2.12
(ce065eabfa5ce81323b009786bdf5bb03127cbe1) octopus (stable)": 1
},
"overall": {
"ceph version 15.2.12
(ce065eabfa5ce81323b009786bdf5bb03127cbe1) octopus (stable)": 7
}
}
root@server ~# pkgfile /usr/lib/librbd.so.1.12.0
community/ceph-libs
root@server ~# pacsearch ceph-libs
community/ceph-libs 15.2.12-1 [installed]
Distributed, fault-tolerant storage platform delivering object,
block, and file system
Sorry I should have included this information in my original email. I
followed this guide
(https://blog.modest-destiny.com/posts/kvm-libvirt-add-ceph-rbd-pool/)
to setup the above configuration and I also tried following the guide at
https://docs.ceph.com/en/latest/rbd/libvirt/
Re: Ceph RBD Connection
On Sun, May 23, 2021 at 4:33 AM Mr. Gecko wrote:
> Hello,
>
> I setup a Ceph Cluster on my system in hopes of using it with libvirtd,
> however I'm finding myself unable to have libvirtd make the connection.
>
> [root@server ~]# virsh pool-start "${CEPH_POOL}"
> error: Failed to start pool libvirt-pool
> error: failed to connect to the RADOS monitor on: localhost:6789,: No
> such file or directory
>
> I know that rbd is working as I am able to use it with the rbd utility.
>
> [root@server ~]# rbd list libvirt-pool
> Kolab
> [root@server ~]# rbd device map libvirt-pool/Kolab
> /dev/rbd0
> [root@server ~]# lsblk /dev/rbd0
> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> rbd0 253:00 20G 0 disk
> ├─rbd0p1 253:10 1G 0 part
> └─rbd0p2 253:20 19G 0 part
>
> The libvirtd logs are not providing any useful information on the issue,
> I tried changing log-level to 1 and enabling the log_filters/adding
> "1:rbd" to the log filters.
>
> I have tried both the libvirt version 7.3.0 in my repository and the
> bleeding edge with a compile from git. Both returns the same error.
>
>
You can check the following:
1. Make sure the ceph mon node are running on the localhost:6789
2. Check if the ceph authorization is enabled on the ceph cluster. Check if
authorization is
used in rbd pool xml.
You can provide more details on the rbd pool xml, ceph conf, and the
version of librbd1(client side), ceph(server side).
> Any help in troubleshooting this issue? I'm somewhat new when it comes
> to all of this.
>
> Thank you,
>
> James Coleman
>
>
