Hey Bill, Thanks for this. This works perfectly!
Thanks, Jon A On Wed, Jul 10, 2013 at 6:44 AM, Campbell, Bill < bcampb...@axcess-financial.com> wrote: > Not entirely. You shouldn’t need to manually create/mount an RBD for the > system datastore. Since the system datastore holds the running VM > deployment files (and not necessarily an RBD image, just a reference to it > in the deployment file) then this directory does not necessarily need to be > shared.**** > > ** ** > > Here’s what we do:**** > > ** ** > > **· **OpenNebula system configured with no special exports/shares. > **** > > **· **The System datastore is modified to use the SSH transfer > manager**** > > **· **We modify the SSH transfer manager pre/post migrate scripts > (by default located in /var/lib/one/remotes/tm/ssh/) to copy files from the > source host to the destination host prior to migration/delete files on > source after successful migration.**** > > ** ** > > Don’t worry about mapping/unmapping RBD volumes. When creating/importing > images into the Ceph datastore the RBDs should be created at this point. > So long as the Hypervisor nodes can see/interact with the Ceph cluster, > when you deploy the VM it will use the RBD in the cluster for storage (no > files copied/mapped locally, all handled by QEMU).**** > > ** ** > > ** ** > > Here is the pre-migrate script we use (very simple):**** > > ** ** > > *#!/bin/bash* > > * * > > *SRC=$1* > > *DST=$2* > > *REMDIR=$3* > > *VMID=$4* > > *DSID=$5* > > *TEMPLATE=$6* > > * * > > *ssh $DST mkdir -p /var/lib/one/datastores/0/$VMID* > > * * > > *ssh $SRC scp /var/lib/one/datastores/0/$VMID/* > $DST:/var/lib/one/datastores/0/$VMID/* > > * * > > *exit 0* > > ** ** > > And the post-migrate script:**** > > ** ** > > *#!/bin/bash* > > ** ** > > *SRC=$1* > > *DST=$2* > > *REMDIR=$3* > > *VMID=$4* > > *DSID=$5* > > *TEMPLATE=$6* > > * * > > *ssh $SRC rm -rf /var/lib/one/datastores/0/$VMID* > > * * > > *exit 0* > > ** ** > > ** ** > > Hope this helps!**** > > ** ** > > *From:* Jon [mailto:three1...@gmail.com] > *Sent:* Wednesday, July 10, 2013 12:01 AM > *To:* Campbell, Bill > *Cc:* Users OpenNebula > *Subject:* Re: [one-users] How to use Ceph/RBD for System Datastore**** > > ** ** > > Hey Bill,**** > > ** ** > > Thanks for getting back to me. **** > > ** ** > > If I'm understanding you correctly, you're basically using the ssh > transfer manager to perform live migrations?**** > > Do you then create/mount one rbd per host?**** > > ** ** > > E.g., **** > > ** ** > > host1:**** > > mount /dev/rbd/rbd/host1-one-system /var/lib/one/datastores/0**** > > ** ** > > host2:**** > > mount /dev/rbd/rbd/host2-one-system /var/lib/one/datastores/0**** > > ** ** > > then use the modified ssh drivers to perform the migrations?**** > > ** ** > > I would definitely be interested in learning how you accomplished that.*** > * > > ** ** > > My other thought was to use CephFS for shared storage. This would > eliminate the need for a NFS/GlusterFS/CLVM, which is an extra layer of > complexity I would like to avoid. As I understand it though, CephFS isn't > "ready for prime-time" which gives me pause... **** > > ** ** > > Thanks again,**** > > Jon A**** > > On Tue, Jul 9, 2013 at 7:55 PM, Campbell, Bill < > bcampb...@axcess-financial.com> wrote:**** > > Jon,**** > > I think I understand what you are trying to do, but I think it doesn't > quite work that way. Let me try to explain (and please let me know if I > don't explain it well enough ;-))**** > > ** ** > > I don't think that you can use Ceph directly as a system datastore. The > way the Ceph datastore driver works for migrations is leveraging whatever > transfer method you have for the system datastore to perform the migration. > For example, if you use the 'shared' system datastore, then it will use > that transfer manager's pre and post migration drivers. For 'ssh', the ssh > drivers, and so on. The way the Ceph datastore is implemented is as Ceph > Block Devices, so unfortunately there is not a way to use it as a simple > shared volume.**** > > ** ** > > There are 2 potential solutions for getting live migrations working for > your Ceph datastore VMs:**** > > - Create a shared NFS volume (or other 'sharable' filesystem, like > GFS2, OCFS2, etc., however these are much more complicated to configure and > usually not worth the hassle) and have the shared volume mounted to the > same location on each hypervisor node. In a previous test deployment, we > just exported out the /var/lib/one/vms directory to the hypervisors. At > this point, all of the hypervisors should be able to see the deployment > files in the same location and you should be able to perform a migration. > **** > - Use SSH as the transfer manager for your system datastore, and > modify the pre and post-migrate scripts to copy the deployment files from > the current VM host to the target VM host. This is the method we use > currently in our deployment, as it is one less configuration step that we > have to worry about maintaining on each node, and makes expanding our > cluster much quicker and easier. I can share with you the pre and > post-migrate scripts we use if you like.**** > > Let me know if the above makes sense, and of course if you need any > additional help please don't hesitate to bug me. I'm very familiar with > the Ceph drivers ;-)**** > > ** ** > ------------------------------ > > *From: *"Jon" <three1...@gmail.com> > *To: *"Users OpenNebula" <users@lists.opennebula.org> > *Sent: *Tuesday, July 9, 2013 8:05:51 PM > *Subject: *[one-users] How to use Ceph/RBD for System Datastore**** > > ** ** > > ** ** > > Hello All,**** > > ** ** > > I am using Ceph as my storage back end and would like to know how to > configure the system datastore, such that I can live migrate vms.**** > > ** ** > > Following the directions, I thought I could create a datastore, format it, > and mount it at /var/lib/one/datastores/0 , however, I discovered, that > isn't quite how things work.**** > > ** ** > > >> > http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-May/001913.html** > ** > > You can read more about that at the above link, but long story short, to > mount a shared filesystem it has to be a "clustered" filesystem (I think > CephFS is the "clustered filesystem", in this case).**** > > ** ** > > I attempted to modify my system datastore config, however, I was unable to > change the DS_MAD parameter, and vm creation errors out telling me there's > no /var/lib/one/remotes/tm/ceph/mkswap driver (there isn't)**** > > ** ** > > >> oneadmin@red6:~$ onedatastore show 0 **** > > >> DATASTORE 0 INFORMATION > **** > > >> ID : 0 **** > > >> NAME : system **** > > >> USER : oneadmin **** > > >> GROUP : oneadmin **** > > >> CLUSTER : - **** > > >> TYPE : SYSTEM **** > > >> DS_MAD : - **** > > >> TM_MAD : ceph **** > > >> BASE PATH : /var/lib/one/datastores/0**** > > >> DISK_TYPE : FILE **** > > >> **** > > >> PERMISSIONS > **** > > >> OWNER : um- **** > > >> GROUP : u-- **** > > >> OTHER : --- **** > > >> **** > > >> DATASTORE TEMPLATE > **** > > >> DISK_TYPE="rbd"**** > > >> DS_MAD="-"**** > > >> TM_MAD="ceph"**** > > >> TYPE="SYSTEM_DS"**** > > >> **** > > >> IMAGES **** > > ** ** > > Maybe I'm just confused. Can anyone provide some guidance on setting ceph > up as the system datastore?**** > > ** ** > > Thanks,**** > > Jon A**** > > ** ** > > _______________________________________________ > Users mailing list > Users@lists.opennebula.org > http://lists.opennebula.org/listinfo.cgi/users-opennebula.org**** > > ** ** > > ** ** > > *NOTICE: Protect the information in this message in accordance with the > company's security policies. If you received this message in error, > immediately notify the sender and destroy all copies.***** > > ** ** > > ** ** > > *NOTICE: Protect the information in this message in accordance with the > company's security policies. If you received this message in error, > immediately notify the sender and destroy all copies.* > >
_______________________________________________ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
