Hey Bill, So let me ask you this. How do you create rbd images? I'm attempting to create a blank datastore, but qemu-img convert keeps segfaulting. Any suggestions are appreciated.
root@red6:~# qemu-img convert -O rbd /var/tmp/61e14679af7dd1e0e1e09e230c89f82a rbd:one/one-17 Segmentation fault (core dumped) The oned log indicates the same: Wed Jul 10 17:52:30 2013 [ImM][I]: Creating disk at of 5120Mb (type: ext4) Wed Jul 10 17:52:31 2013 [ImM][I]: Command execution fail: /var/lib/one/remotes/datastore/ceph/mkfs Wed Jul 10 17:52:31 2013 [ImM][E]: mkfs: Command " set -e Wed Jul 10 17:52:31 2013 [ImM][I]: Wed Jul 10 17:52:31 2013 [ImM][I]: # create and format Wed Jul 10 17:52:31 2013 [ImM][I]: dd if=/dev/zero of=/var/tmp/61e14679af7dd1e0e1e09e230c89f82a bs=1 count=1 seek=5120M Wed Jul 10 17:52:31 2013 [ImM][I]: mkfs -t ext4 -F /var/tmp/61e14679af7dd1e0e1e09e230c89f82a Wed Jul 10 17:52:31 2013 [ImM][I]: Wed Jul 10 17:52:31 2013 [ImM][I]: # create rbd Wed Jul 10 17:52:31 2013 [ImM][I]: qemu-img convert -O rbd /var/tmp/61e14679af7dd1e0e1e09e230c89f82a rbd:one/one-17 Wed Jul 10 17:52:31 2013 [ImM][I]: Wed Jul 10 17:52:31 2013 [ImM][I]: # remove original Wed Jul 10 17:52:31 2013 [ImM][I]: rm -f /var/tmp/61e14679af7dd1e0e1e09e230c89f82a" failed: 1+0 records in Wed Jul 10 17:52:31 2013 [ImM][I]: 1+0 records out Wed Jul 10 17:52:31 2013 [ImM][I]: 1 byte (1 B) copied, 0.000232576 s, 4.3 kB/s Wed Jul 10 17:52:31 2013 [ImM][I]: mke2fs 1.42.5 (29-Jul-2012) Wed Jul 10 17:52:31 2013 [ImM][I]: Segmentation fault (core dumped) Wed Jul 10 17:52:31 2013 [ImM][E]: Error registering one/one-17 in localhost Wed Jul 10 17:52:31 2013 [ImM][I]: ExitCode: 139 Wed Jul 10 17:52:31 2013 [ImM][E]: Error creating datablock: Error registering one/one-17 in localhost My image definition looks like this: IMAGE 17 INFORMATION ID : 17 NAME : ubuntu-server-13.04-x86_64 USER : oneadmin GROUP : oneadmin DATASTORE : rbd1 TYPE : DATABLOCK REGISTER TIME : 07/10 17:52:30 PERSISTENT : Yes SOURCE : FSTYPE : ext4 SIZE : 5G STATE : err RUNNING_VMS : 0 PERMISSIONS OWNER : um- GROUP : --- OTHER : --- IMAGE TEMPLATE DESCRIPTION="ubuntu-server-13.04-x86_64" DEV_PREFIX="hd" ERROR="Wed Jul 10 17:52:31 2013 : Error creating datablock: Error registering one/one-17 in localhost" Thanks, Jon A On Wed, Jul 10, 2013 at 5:14 PM, Jon <three1...@gmail.com> wrote: > Hey Bill, > > Thanks for this. This works perfectly! > > Thanks, > Jon A > > > On Wed, Jul 10, 2013 at 6:44 AM, Campbell, Bill < > bcampb...@axcess-financial.com> wrote: > >> Not entirely. You shouldn’t need to manually create/mount an RBD for the >> system datastore. Since the system datastore holds the running VM >> deployment files (and not necessarily an RBD image, just a reference to it >> in the deployment file) then this directory does not necessarily need to be >> shared.**** >> >> ** ** >> >> Here’s what we do:**** >> >> ** ** >> >> **· **OpenNebula system configured with no special >> exports/shares.**** >> >> **· **The System datastore is modified to use the SSH transfer >> manager**** >> >> **· **We modify the SSH transfer manager pre/post migrate >> scripts (by default located in /var/lib/one/remotes/tm/ssh/) to copy files >> from the source host to the destination host prior to migration/delete >> files on source after successful migration.**** >> >> ** ** >> >> Don’t worry about mapping/unmapping RBD volumes. When creating/importing >> images into the Ceph datastore the RBDs should be created at this point. >> So long as the Hypervisor nodes can see/interact with the Ceph cluster, >> when you deploy the VM it will use the RBD in the cluster for storage (no >> files copied/mapped locally, all handled by QEMU).**** >> >> ** ** >> >> ** ** >> >> Here is the pre-migrate script we use (very simple):**** >> >> ** ** >> >> *#!/bin/bash* >> >> * * >> >> *SRC=$1* >> >> *DST=$2* >> >> *REMDIR=$3* >> >> *VMID=$4* >> >> *DSID=$5* >> >> *TEMPLATE=$6* >> >> * * >> >> *ssh $DST mkdir -p /var/lib/one/datastores/0/$VMID* >> >> * * >> >> *ssh $SRC scp /var/lib/one/datastores/0/$VMID/* >> $DST:/var/lib/one/datastores/0/$VMID/* >> >> * * >> >> *exit 0* >> >> ** ** >> >> And the post-migrate script:**** >> >> ** ** >> >> *#!/bin/bash* >> >> ** ** >> >> *SRC=$1* >> >> *DST=$2* >> >> *REMDIR=$3* >> >> *VMID=$4* >> >> *DSID=$5* >> >> *TEMPLATE=$6* >> >> * * >> >> *ssh $SRC rm -rf /var/lib/one/datastores/0/$VMID* >> >> * * >> >> *exit 0* >> >> ** ** >> >> ** ** >> >> Hope this helps!**** >> >> ** ** >> >> *From:* Jon [mailto:three1...@gmail.com] >> *Sent:* Wednesday, July 10, 2013 12:01 AM >> *To:* Campbell, Bill >> *Cc:* Users OpenNebula >> *Subject:* Re: [one-users] How to use Ceph/RBD for System Datastore**** >> >> ** ** >> >> Hey Bill,**** >> >> ** ** >> >> Thanks for getting back to me. **** >> >> ** ** >> >> If I'm understanding you correctly, you're basically using the ssh >> transfer manager to perform live migrations?**** >> >> Do you then create/mount one rbd per host?**** >> >> ** ** >> >> E.g., **** >> >> ** ** >> >> host1:**** >> >> mount /dev/rbd/rbd/host1-one-system /var/lib/one/datastores/0**** >> >> ** ** >> >> host2:**** >> >> mount /dev/rbd/rbd/host2-one-system /var/lib/one/datastores/0**** >> >> ** ** >> >> then use the modified ssh drivers to perform the migrations?**** >> >> ** ** >> >> I would definitely be interested in learning how you accomplished that.** >> ** >> >> ** ** >> >> My other thought was to use CephFS for shared storage. This would >> eliminate the need for a NFS/GlusterFS/CLVM, which is an extra layer of >> complexity I would like to avoid. As I understand it though, CephFS isn't >> "ready for prime-time" which gives me pause... **** >> >> ** ** >> >> Thanks again,**** >> >> Jon A**** >> >> On Tue, Jul 9, 2013 at 7:55 PM, Campbell, Bill < >> bcampb...@axcess-financial.com> wrote:**** >> >> Jon,**** >> >> I think I understand what you are trying to do, but I think it doesn't >> quite work that way. Let me try to explain (and please let me know if I >> don't explain it well enough ;-))**** >> >> ** ** >> >> I don't think that you can use Ceph directly as a system datastore. The >> way the Ceph datastore driver works for migrations is leveraging whatever >> transfer method you have for the system datastore to perform the migration. >> For example, if you use the 'shared' system datastore, then it will use >> that transfer manager's pre and post migration drivers. For 'ssh', the ssh >> drivers, and so on. The way the Ceph datastore is implemented is as Ceph >> Block Devices, so unfortunately there is not a way to use it as a simple >> shared volume.**** >> >> ** ** >> >> There are 2 potential solutions for getting live migrations working for >> your Ceph datastore VMs:**** >> >> - Create a shared NFS volume (or other 'sharable' filesystem, like >> GFS2, OCFS2, etc., however these are much more complicated to configure >> and >> usually not worth the hassle) and have the shared volume mounted to the >> same location on each hypervisor node. In a previous test deployment, we >> just exported out the /var/lib/one/vms directory to the hypervisors. At >> this point, all of the hypervisors should be able to see the deployment >> files in the same location and you should be able to perform a migration. >> **** >> - Use SSH as the transfer manager for your system datastore, and >> modify the pre and post-migrate scripts to copy the deployment files from >> the current VM host to the target VM host. This is the method we use >> currently in our deployment, as it is one less configuration step that we >> have to worry about maintaining on each node, and makes expanding our >> cluster much quicker and easier. I can share with you the pre and >> post-migrate scripts we use if you like.**** >> >> Let me know if the above makes sense, and of course if you need any >> additional help please don't hesitate to bug me. I'm very familiar with >> the Ceph drivers ;-)**** >> >> ** ** >> ------------------------------ >> >> *From: *"Jon" <three1...@gmail.com> >> *To: *"Users OpenNebula" <users@lists.opennebula.org> >> *Sent: *Tuesday, July 9, 2013 8:05:51 PM >> *Subject: *[one-users] How to use Ceph/RBD for System Datastore**** >> >> ** ** >> >> ** ** >> >> Hello All,**** >> >> ** ** >> >> I am using Ceph as my storage back end and would like to know how to >> configure the system datastore, such that I can live migrate vms.**** >> >> ** ** >> >> Following the directions, I thought I could create a datastore, format >> it, and mount it at /var/lib/one/datastores/0 , however, I discovered, that >> isn't quite how things work.**** >> >> ** ** >> >> >> >> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-May/001913.html* >> *** >> >> You can read more about that at the above link, but long story short, to >> mount a shared filesystem it has to be a "clustered" filesystem (I think >> CephFS is the "clustered filesystem", in this case).**** >> >> ** ** >> >> I attempted to modify my system datastore config, however, I was unable >> to change the DS_MAD parameter, and vm creation errors out telling me >> there's no /var/lib/one/remotes/tm/ceph/mkswap driver (there isn't)**** >> >> ** ** >> >> >> oneadmin@red6:~$ onedatastore show 0 **** >> >> >> DATASTORE 0 INFORMATION >> **** >> >> >> ID : 0 **** >> >> >> NAME : system **** >> >> >> USER : oneadmin **** >> >> >> GROUP : oneadmin **** >> >> >> CLUSTER : - **** >> >> >> TYPE : SYSTEM **** >> >> >> DS_MAD : - **** >> >> >> TM_MAD : ceph **** >> >> >> BASE PATH : /var/lib/one/datastores/0**** >> >> >> DISK_TYPE : FILE **** >> >> >> **** >> >> >> PERMISSIONS >> **** >> >> >> OWNER : um- **** >> >> >> GROUP : u-- **** >> >> >> OTHER : --- **** >> >> >> **** >> >> >> DATASTORE TEMPLATE >> **** >> >> >> DISK_TYPE="rbd"**** >> >> >> DS_MAD="-"**** >> >> >> TM_MAD="ceph"**** >> >> >> TYPE="SYSTEM_DS"**** >> >> >> **** >> >> >> IMAGES **** >> >> ** ** >> >> Maybe I'm just confused. Can anyone provide some guidance on setting >> ceph up as the system datastore?**** >> >> ** ** >> >> Thanks,**** >> >> Jon A**** >> >> ** ** >> >> _______________________________________________ >> Users mailing list >> Users@lists.opennebula.org >> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org**** >> >> ** ** >> >> ** ** >> >> *NOTICE: Protect the information in this message in accordance with the >> company's security policies. If you received this message in error, >> immediately notify the sender and destroy all copies.***** >> >> ** ** >> >> ** ** >> >> *NOTICE: Protect the information in this message in accordance with the >> company's security policies. If you received this message in error, >> immediately notify the sender and destroy all copies.* >> >> >
_______________________________________________ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org