Thanks Andrija for your detailed instructions. Here a question, can I execute all the steps mentioned at http://pastebin.com/HwCZEASR in the centos kvm node which had LibVirt compiled from its git source already?
Praveen Kumar Buravilli Cloud Platform Implementation Engineer, APAC Cloud Services M +91-9885456905 praveen.buravi...@citrix.com -----Original Message----- From: Andrija Panic [mailto:andrija.pa...@gmail.com] Sent: 20 June 2014 14:35 To: users@cloudstack.apache.org Subject: Re: Adding ceph RBD storage failed in CentOS 6.5 KVM Been there, done that....: This libvirt error "error: internal error missing backend for pool type 8" means that libvirt was not compiled with RBD backend support. Here are my steps to compile libvirt 1.2.3 few months ago - change configure options if you want, I tried to use as much options as possible. http://pastebin.com/HwCZEASR (note the "ceph-devel" must be installed as in instructions in order to be able to compile with RBD support) Also note, that if you are using CentOS 6.5, the "-s" flag was removed from qemu-img packages, meaning you will not be able to use snapshot functionality on cloudstack (not related to libbvirt), this is ANY snapshoting will be broken - there is workarround down :) Also, beside making sure libvirt can talk to RBD/CEPH, you MUST be sure your qemu-img and qemu-kvm was compiled with RBD support, check like this: qemu-img | grep "Supported formats" should get somwething like this - note the "rbd" on the end of output: *Supported formats: raw cow qcow vdi vmdk cloop dmg bochs vpc vvfat qcow2 qed parallels nbd blkdebug host_cdrom host_floppy host_device file rbd* If both qemu and libvirt are fine, you will be able to add CEPH to ACS 4.2 or newer. If you have stock (CentOS) versions of qemu-img and qemu-kvm - they are NOT compiled with RBD support, so you will not be able to use CEPH. You need to install Intank's version of RPM packages, that are based on official RedHat stock code of those packages, but are pathced for RBD/CEPH support. Refer to http://admintweets.com/centos-kvm-and-ceph-client-side-setup/ in order to download Intanks's RPMs - note that the latest RPMs you will find are probably also based on RHEL 6.5 version, that is missing "-s" flag, so you will still NOT be able to use disk snapshotting in ACS... I solved this by installing a little bit older RPMs from Intanks (qemu-img and qemu-kvm that are based on RHEL 6.2 that still has that famous "-s" flag preseent), please let me know if you need this provided, since they are NOT present on the Intanks download page at the moment exact versions of packages I installed and which are working fine for me: qemu-kvm-tools-0.12.1.2-2.415.el6.3ceph.x86_64 qemu-img-0.12.1.2-2.355.el6.2.cuttlefish.x86_64 qemu-kvm-0.12.1.2-2.415.el6.3ceph.x86_64 This whole setup is working fine for me... Hope that helps, was going through the same pain, as you do now... :) Best, Andrija Panic On 20 June 2014 04:38, Praveen Buravilli <praveen.buravi...@citrix.com> wrote: > Hi, > > > > I am facing an issue in adding ceph RBD storage to CloudStack. It is > failing with “Failed to add datasource” error. > > I have followed all the available instructions related to CloudStack, > KVM and ceph storage integration. > > > > CenOS 6.5 KVM is used as KVM node here. I have read in some blog that > we need to compile LibVirt in CenOS KVM nodes to make Ceph storage to > work with in CloudStack. > > Hence I have got git cloned LibVirt package from its source and > upgraded LibVirt and Qemu versions. > > (Commands used à git cone #####, ./autogen.sh, make, make install). > > > > It seems CentOS 6.5 KVM needs to be enabled for RBD (driver) support > which needs to be specified as a parameter while compiling LibVirt. > > > > Can anyone throw some pointers on how to rectify this problem? > > > > *Management Server Exception:* > > 2014-06-20 09:58:03,757 DEBUG [agent.transport.Request] > (catalina-exec-6:null) Seq 1-1602164611: Received: { Ans: , MgmtId: > 52234925782, via: 1, Ver: v1, Flags: 10, { Answer } } > > 2014-06-20 09:58:03,757 DEBUG [agent.manager.AgentManagerImpl] > (catalina-exec-6:null) Details from executing class > com.cloud.agent.api.ModifyStoragePoolCommand: > java.lang.NullPointerException > > at > com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePo > ol(LibvirtStorageAdaptor.java:531) > > at > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePo > ol(KVMStoragePoolManager.java:185) > > at > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePo > ol(KVMStoragePoolManager.java:177) > > > > I even tried defining a pool using virsh command even that is failing > with “internal error missing backend for pool type 8”. > > This indicates my KVM LibVirt is not supported for RBD. > > > > *Virsh exception on manual pool definition* > > <pool type='rbd'> > > <name>c574980a-19fc-37e9-b6e3-788a7439575d</name> > > <uuid>c574980a-19fc-37e9-b6e3-788a7439575d</uuid> > > <source> > > <host name='192.168.153.25' port='6789'/> > > <name>cloudstack</name> > > <auth username='cloudstack' type='ceph'> > > <secret uuid='c574980a-19fc-37e9-b6e3-788a7439575d'/> > > </auth> > > </source> > > </pool> > > > > [root@kvm-ovs-002 agent]# virsh pool-define /tmp/rbd.xml > > error: Failed to define pool from /tmp/rbd.xml > > error: internal error missing backend for pool type 8 > > > > The Ceph storage is working fine and confirmed with following > statistics of it. > > > > *Ceph output* > > [root@kvm-ovs-002 ~]# ceph auth list > > installed auth entries: > > > > osd.0 > > key: AQCwTKFTSOudGhAAsWAMRFuCqHjvTQKEV0zjvw== > > caps: [mon] allow profile osd > > caps: [osd] allow * > > client.admin > > key: AQBRQqFTWOjBKhAA2s7KnL1z3h7PuKeqXMd7SA== > > caps: [mds] allow > > caps: [mon] allow * > > caps: [osd] allow * > > client.bootstrap-mds > > key: AQBSQqFTYKm6CRAAjjZotpN68yJaOjS2QTKzKg== > > caps: [mon] allow profile bootstrap-mds > > client.bootstrap-osd > > key: AQBRQqFT6GzXNxAA4ZTmVX6LIu0k4Sk7bh2Ifg== > > caps: [mon] allow profile bootstrap-osd > > client.cloudstack > > key: AQBNTaFTeCuwFRAA0NE7CCm9rwuq3ngLcGEysQ== > > caps: [mon] allow r > > caps: [osd] allow rwx pool=cloudstack > > > > [root@ceph ~]# ceph status > > cluster 9c1be0b6-f600-45d7-ae0f-df7bcd3a82cd > > health HEALTH_WARN 292 pgs degraded; 292 pgs stale; 292 pgs stuck > stale; 292 pgs stuck unclean; 1/1 in osds are down; clock skew > detected on > mon.kvm-ovs-00 > 2 > > monmap e1: 2 mons at {ceph= > 192.168.153.25:6789/0,kvm-ovs-002=192.168.160.3: 6789/0}, election > epoch 10, quorum 0,1 ceph,kvm-ovs-002 > > osdmap e8: 1 osds: 0 up, 1 in > > pgmap v577: 292 pgs, 4 pools, 0 bytes data, 0 objects > > 26036 MB used, 824 GB / 895 GB avail > > 292 stale+active+degraded > > > > [root@kvm-ovs-002 agent]# cat /etc/redhat-release > > CentOS release 6.5 (Final) > > > > The compiled LibVirt is showing upgraded version in virsh but still > finding old rpm packages in KVM. > > Give me some hint on whether to clean-up these old RPMs? > > > > *Virsh version* > > [root@kvm-ovs-002 agent]# virsh version > > Compiled against library: libvirt 1.2.6 > > Using library: libvirt 1.2.6 > > Using API: QEMU 1.2.6 > > Running hypervisor: QEMU 0.12.1 > > > > [root@kvm-ovs-002 agent]# rpm -qa | grep qemu > > qemu-kvm-tools-0.12.1.2-2.415.el6_5.10.x86_64 > > qemu-kvm-0.12.1.2-2.415.el6.3ceph.x86_64 > > gpxe-roms-qemu-0.9.7-6.10.el6.noarch > > qemu-kvm-0.12.1.2-2.415.el6_5.10.x86_64 > > qemu-img-0.12.1.2-2.415.el6.3ceph.x86_64 > > qemu-img-0.12.1.2-2.415.el6_5.10.x86_64 > > qemu-kvm-tools-0.12.1.2-2.415.el6.3ceph.x86_64 > > qemu-guest-agent-0.12.1.2-2.415.el6_5.10.x86_64 > > You have mail in /var/spool/mail/root > > [root@kvm-ovs-002 agent]# rpm -qa | grep libvirt > > libvirt-python-0.10.2-29.el6_5.9.x86_64 > > libvirt-java-0.4.9-1.el6.noarch > > libvirt-cim-0.6.1-9.el6_5.1.x86_64 > > libvirt-client-0.10.2-29.el6_5.9.x86_64 > > libvirt-devel-0.10.2-29.el6_5.9.x86_64 > > fence-virtd-libvirt-0.2.3-15.el6.x86_64 > > libvirt-0.10.2-29.el6_5.9.x86_64 > > libvirt-snmp-0.0.2-4.el6.x86_64 > > > > *Attached all the log files from management and kvm servers. > > > > Thanks, > > *Praveen Kumar Buravilli* > > > -- Andrija Panić -------------------------------------- http://admintweets.com --------------------------------------
