Hi,
I am facing an issue in adding ceph RBD storage to CloudStack. It is failing
with "Failed to add datasource" error.
I have followed all the available instructions related to CloudStack, KVM and
ceph storage integration.
CenOS 6.5 KVM is used as KVM node here. I have read in some blog that we need
to compile LibVirt in CenOS KVM nodes to make Ceph storage to work with in
CloudStack.
Hence I have got git cloned LibVirt package from its source and upgraded
LibVirt and Qemu versions.
(Commands used --> git cone #####, ./autogen.sh, make, make install).
It seems CentOS 6.5 KVM needs to be enabled for RBD (driver) support which
needs to be specified as a parameter while compiling LibVirt.
Can anyone throw some pointers on how to rectify this problem?
Management Server Exception:
2014-06-20 09:58:03,757 DEBUG [agent.transport.Request] (catalina-exec-6:null)
Seq 1-1602164611: Received: { Ans: , MgmtId: 52234925782, via: 1, Ver: v1,
Flags: 10, { Answer } }
2014-06-20 09:58:03,757 DEBUG [agent.manager.AgentManagerImpl]
(catalina-exec-6:null) Details from executing class
com.cloud.agent.api.ModifyStoragePoolCommand: java.lang.NullPointerException
at
com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:531)
at
com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:185)
at
com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:177)
I even tried defining a pool using virsh command even that is failing with
"internal error missing backend for pool type 8".
This indicates my KVM LibVirt is not supported for RBD.
Virsh exception on manual pool definition
<pool type='rbd'>
<name>c574980a-19fc-37e9-b6e3-788a7439575d</name>
<uuid>c574980a-19fc-37e9-b6e3-788a7439575d</uuid>
<source>
<host name='192.168.153.25' port='6789'/>
<name>cloudstack</name>
<auth username='cloudstack' type='ceph'>
<secret uuid='c574980a-19fc-37e9-b6e3-788a7439575d'/>
</auth>
</source>
</pool>
[root@kvm-ovs-002 agent]# virsh pool-define /tmp/rbd.xml
error: Failed to define pool from /tmp/rbd.xml
error: internal error missing backend for pool type 8
The Ceph storage is working fine and confirmed with following statistics of it.
Ceph output
[root@kvm-ovs-002 ~]# ceph auth list
installed auth entries:
osd.0
key: AQCwTKFTSOudGhAAsWAMRFuCqHjvTQKEV0zjvw==
caps: [mon] allow profile osd
caps: [osd] allow *
client.admin
key: AQBRQqFTWOjBKhAA2s7KnL1z3h7PuKeqXMd7SA==
caps: [mds] allow
caps: [mon] allow *
caps: [osd] allow *
client.bootstrap-mds
key: AQBSQqFTYKm6CRAAjjZotpN68yJaOjS2QTKzKg==
caps: [mon] allow profile bootstrap-mds
client.bootstrap-osd
key: AQBRQqFT6GzXNxAA4ZTmVX6LIu0k4Sk7bh2Ifg==
caps: [mon] allow profile bootstrap-osd
client.cloudstack
key: AQBNTaFTeCuwFRAA0NE7CCm9rwuq3ngLcGEysQ==
caps: [mon] allow r
caps: [osd] allow rwx pool=cloudstack
[root@ceph ~]# ceph status
cluster 9c1be0b6-f600-45d7-ae0f-df7bcd3a82cd
health HEALTH_WARN 292 pgs degraded; 292 pgs stale; 292 pgs stuck stale;
292 pgs stuck unclean; 1/1 in osds are down; clock skew detected on
mon.kvm-ovs-00
2
monmap e1: 2 mons at
{ceph=192.168.153.25:6789/0,kvm-ovs-002=192.168.160.3: 6789/0}, election epoch
10, quorum 0,1 ceph,kvm-ovs-002
osdmap e8: 1 osds: 0 up, 1 in
pgmap v577: 292 pgs, 4 pools, 0 bytes data, 0 objects
26036 MB used, 824 GB / 895 GB avail
292 stale+active+degraded
[root@kvm-ovs-002 agent]# cat /etc/redhat-release
CentOS release 6.5 (Final)
The compiled LibVirt is showing upgraded version in virsh but still finding old
rpm packages in KVM.
Give me some hint on whether to clean-up these old RPMs?
Virsh version
[root@kvm-ovs-002 agent]# virsh version
Compiled against library: libvirt 1.2.6
Using library: libvirt 1.2.6
Using API: QEMU 1.2.6
Running hypervisor: QEMU 0.12.1
[root@kvm-ovs-002 agent]# rpm -qa | grep qemu
qemu-kvm-tools-0.12.1.2-2.415.el6_5.10.x86_64
qemu-kvm-0.12.1.2-2.415.el6.3ceph.x86_64
gpxe-roms-qemu-0.9.7-6.10.el6.noarch
qemu-kvm-0.12.1.2-2.415.el6_5.10.x86_64
qemu-img-0.12.1.2-2.415.el6.3ceph.x86_64
qemu-img-0.12.1.2-2.415.el6_5.10.x86_64
qemu-kvm-tools-0.12.1.2-2.415.el6.3ceph.x86_64
qemu-guest-agent-0.12.1.2-2.415.el6_5.10.x86_64
You have mail in /var/spool/mail/root
[root@kvm-ovs-002 agent]# rpm -qa | grep libvirt
libvirt-python-0.10.2-29.el6_5.9.x86_64
libvirt-java-0.4.9-1.el6.noarch
libvirt-cim-0.6.1-9.el6_5.1.x86_64
libvirt-client-0.10.2-29.el6_5.9.x86_64
libvirt-devel-0.10.2-29.el6_5.9.x86_64
fence-virtd-libvirt-0.2.3-15.el6.x86_64
libvirt-0.10.2-29.el6_5.9.x86_64
libvirt-snmp-0.0.2-4.el6.x86_64
*Attached all the log files from management and kvm servers.
Thanks,
Praveen Kumar Buravilli
