Hi all, I am currently doing my first steps with KVM as hypervisor for CloudStack. I was shocked to realize that currently live volume migration between different shared storages is not supported with KVM. This is a feature I use intensively with XenServer.
How do you get along with this limitation? I do really expect you to use some workarounds, or do you all only accept vm downtimes for a storage migration? With my first investigation I found three techniques mentioned and would like to ask for suggestions which to investigate deeper: x Eric describes a technique using snapshosts and pauses to do a live storage migration in this mailing list tread. x Dag suggests using virsh with --copy-storage-all --xml. x I found articles about using virsh blockcopy for storage live migration. Greetings, Melanie Am Freitag, den 02.02.2018, 15:55 +0100 schrieb Andrija Panic: > @Dag, you might want to check with Mike Tutkowski, how he implemented > this > for the "online storage migration" from other storages (CEPH and NFS > implemented so far as sources) to SolidFire. > > We are doing exactly the same demo/manual way (this is what Mike has > sent > me back in the days), so perhaps you want to see how to translate > this into > general things (so ANY to ANY storage migration) inside CloudStack. > > Cheers > > On 2 February 2018 at 10:28, Dag Sonstebo <dag.sonst...@shapeblue.com > > > wrote: > > > All > > > > I am doing a bit of R&D around this for a client at the moment. I > > am > > semi-successful in getting live migrations to different storage > > pools to > > work. The method I’m using is as follows – this does not take into > > account > > any efficiency optimisation around the disk transfer (which is next > > on my > > list). The below should answer your question Eric about moving to a > > different location – and I am also working with your steps to see > > where I > > can improve the following. Keep in mind all of this is external to > > CloudStack – although CloudStack picks up the destination KVM host > > automatically it does not update the volume tables etc., neither > > does it do > > any housekeeping. > > > > 1) Ensure the same network bridges are up on source and destination > > – > > these are found with: > > > > [root@kvm1 ~]# virsh dumpxml 9 | grep source > > <source file='/mnt/00e88a7b-985f-3be8-b717- > > 0a59d8197640/d0ab5dd5- > > e3dd-47ac-a326-5ce3d47d194d'/> > > <source bridge='breth1-725'/> > > <source path='/dev/pts/3'/> > > <source path='/dev/pts/3'/> > > > > So from this make sure breth1-725 is up on the destionation host > > (do it > > the hard way or cheat and spin up a VM from same account and > > network on > > that host) > > > > 2) Find size of source disk and create stub disk in destination > > (this part > > can be made more efficient to speed up disk transfer – by doing > > similar > > things to what Eric is doing): > > > > [root@kvm1 ~]# qemu-img info /mnt/00e88a7b-985f-3be8-b717- > > 0a59d8197640/d0ab5dd5-e3dd-47ac-a326-5ce3d47d194d > > image: /mnt/00e88a7b-985f-3be8-b717-0a59d8197640/d0ab5dd5-e3dd- > > 47ac-a326-5ce3d47d194d > > file format: qcow2 > > virtual size: 8.0G (8589934592 bytes) > > disk size: 32M > > cluster_size: 65536 > > backing file: /mnt/00e88a7b-985f-3be8-b717-0a59d8197640/3caaf4c9- > > eaec- > > 11e7-800b-06b4a401075c > > > > ###################### > > > > [root@kvm3 50848ff7-c6aa-3fdd-b487-27899bf2129c]# qemu-img create > > -f > > qcow2 d0ab5dd5-e3dd-47ac-a326-5ce3d47d194d 8G > > Formatting 'd0ab5dd5-e3dd-47ac-a326-5ce3d47d194d', fmt=qcow2 > > size=8589934592 encryption=off cluster_size=65536 > > [root@kvm3 50848ff7-c6aa-3fdd-b487-27899bf2129c]# qemu-img info > > d0ab5dd5-e3dd-47ac-a326-5ce3d47d194d > > image: d0ab5dd5-e3dd-47ac-a326-5ce3d47d194d > > file format: qcow2 > > virtual size: 8.0G (8589934592 bytes) > > disk size: 448K > > cluster_size: 65536 > > > > 3) Rewrite the new VM XML file for the destination with: > > a) New disk location, in this case this is just a new path (Eric – > > this > > answers your question) > > b) Different IP addresses for VNC – in this case 10.0.0.1 to > > 10.0.0.2 > > and carry out migration. > > > > [root@kvm1 ~]# virsh dumpxml 9 | sed -e 's/00e88a7b-985f-3be8-b717- > > 0a59d8197640/50848ff7-c6aa-3fdd-b487-27899bf2129c/g' | sed -e 's/ > > 10.0.0.1/10.0.0.2/g' > /root/i-2-14-VM.xml > > > > [root@kvm1 ~]# virsh migrate --live --persistent --copy-storage-all > > --xml > > /root/i-2-14-VM.xml i-2-14-VM qemu+tcp://10.0.0.2/system --verbose > > --abort-on-error > > Migration: [ 25 %] > > > > 4) Once complete delete the source file. This can be done with > > extra > > switches on the virsh migrate command if need be. > > = = = > > > > In the simplest tests this works – destination VM remains online > > and has > > storage in new location – but it’s not persistent – sometimes the > > destination VM ends up in a paused state, and I’m working on how to > > get > > around this. I also noted virsh migrate has a migrate- > > setmaxdowntime which > > I think can be useful here. > > > > Regards, > > Dag Sonstebo > > Cloud Architect > > ShapeBlue > > > > On 01/02/2018, 20:30, "Andrija Panic" <andrija.pa...@gmail.com> > > wrote: > > > > Actually, we have this feature (we call this internally > > online-storage-migration) to migrate volume from CEPH/NFS to > > SolidFire > > (thanks to Mike Tutkowski) > > > > There is libvirt mechanism, where basically you start another > > PAUSED > > VM on > > another host (same name and same XML file, except the storage > > volumes > > are > > pointing to new storage, different paths, etc and maybe VNC > > listening > > address needs to be changed or so) and then you issue on > > original > > host/VM > > the live migrate command with few parameters... the libvirt > > will > > transaprently handle the copy data process from Soruce to New > > volumes, > > and > > after migration the VM will be alive (with new XML since have > > new > > volumes) > > on new host, while the original VM on original host is > > destroyed.... > > > > (I can send you manual for this, that is realted to SF, but > > idea is the > > same and you can exercies this on i.e. 2 NFS volumes on 2 > > different > > storages) > > > > This mechanism doesn't exist in ACS in general (AFAIK), except > > for when > > migrating to SolidFire. > > > > Perhaps community/DEV can help extend Mike's code to do same > > work on > > different storage types... > > > > Cheers > > > > > > dag.sonst...@shapeblue.com > > www.shapeblue.com > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > @shapeblue > > > > > > > > On 19 January 2018 at 18:45, Eric Green <eric.lee.gr...@gmail.com> > > wrote: > > > > > KVM is able to live migrate entire virtual machines complete > > with > > local > > > volumes (see 'man virsh') but does require nbd (Network Block > > Device) to be > > > installed on the destination host to do so. It may need > > installation > > of > > > later libvirt / qemu packages from OpenStack repositories on > > Centos > > 6, I'm > > > not sure, but just works on Centos 7. In any event, I have > > used this > > > functionality to move virtual machines between virtualization > > hosts > > on my > > > home network. It works. > > > > > > What is missing is the ability to live-migrate a disk from > > one shared > > > storage to another. The functionality built into virsh live- > > migrates > > the > > > volume ***to the exact same location on the new host***, so > > obviously is > > > useless for migrating the disk to a new location on shared > > storage. I > > > looked everywhere for the ability of KVM to live migrate a > > disk from > > point > > > A to point B all by itself, and found no such thing. > > libvirt/qemu > > has the > > > raw capabilities needed to do this, but it is not currently > > exposed > > as a > > > single API via the qemu console or virsh. It can be emulated > > via > > scripting > > > however: > > > > > > 1. Pause virtual machine > > > 2. Do qcow2 snapshot. > > > 3. Detach base disk, attach qcow2 snapshot > > > 4. unpause virtual machine > > > 5. copy qcow2 base file to new location > > > 6. pause virtual machine > > > 7. detach snapshot > > > 8. unsnapshot qcow2 snapshot at its new location. > > > 9. attach new base at new location. > > > 10. unpause virtual machine. > > > > > > Thing is, if that entire process is not built into the > > underlying > > > kvm/qemu/libvirt infrastructure as tested functionality with > > a > > defined API, > > > there's no guarantee that it will work seamlessly and will > > continue > > working > > > with the next release of the underlying infrastructure. This > > is using > > > multiple different tools to manipulate the qcow2 file and > > attach/detach > > > base disks to the running (but paused) kvm domain, and would > > have to > > be > > > tested against all variations of those tools on all supported > > Cloudstack > > > KVM host platforms. The test matrix looks pretty grim. > > > > > > By contrast, the migrate-with-local-storage process is built > > into > > virsh > > > and is tested by the distribution vendor and the set of tools > > provided with > > > the distribution is guaranteed to work with the virsh / > > libvirt/ qemu > > > distributed by the distribution vendor. That makes the test > > matrix > > for > > > move-with-local-storage look a lot simpler -- "is this > > functionality > > > supported by that version of virsh on that distribution? Yes? > > Enable > > it. > > > No? Don't enable it." > > > > > > I'd love to have live migration of disks on shared storage > > with > > Cloudstack > > > KVM, but not at the expense of reliability. Shutting down a > > virtual > > machine > > > in order to migrate one of its disks from one shared > > datastore to > > another > > > is not ideal, but at least it's guaranteed reliable. > > > > > > > > > > On Jan 19, 2018, at 04:54, Rafael Weingärtner < > > > rafaelweingart...@gmail.com> wrote: > > > > > > > > Hey Marc, > > > > It is very interesting that you are going to pick this up > > for KVM. > > I am > > > > working in a related issue for XenServer [1]. > > > > If you can confirm that KVM is able to live migrate local > > volumes > > to > > > other > > > > local storage or shared storage I could make the feature I > > am > > working on > > > > available to KVM as well. > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/CLOUDSTACK-10240 > > > > > > > > On Thu, Jan 18, 2018 at 11:35 AM, Marc-Aurèle Brothier < > > > ma...@exoscale.ch> > > > > wrote: > > > > > > > >> There's a PR waiting to be fixed about live migration with > > local > > volume > > > for > > > >> KVM. So it will come at some point. I'm the one who made > > this PR > > but I'm > > > >> not using the upstream release so it's hard for me to > > debug the > > problem. > > > >> You can add yourself to the PR to get notify when things > > are > > moving on > > > it. > > > >> > > > >> https://github.com/apache/cloudstack/pull/1709 > > > >> > > > >> On Wed, Jan 17, 2018 at 10:56 AM, Eric Green < > > eric.lee.gr...@gmail.com> > > > >> wrote: > > > >> > > > >>> Theoretically on Centos 7 as the host KVM OS it could be > > done > > with a > > > >>> couple of pauses and the snapshotting mechanism built > > into > > qcow2, but > > > >> there > > > >>> is no simple way to do it directly via virsh, the > > libvirtd/qemu > > control > > > >>> program that is used to manage virtualization. It's not > > as with > > > issuing a > > > >>> simple vmotion 'migrate volume' call in Vmware. > > > >>> > > > >>> I scripted out how it would work without that direct > > support in > > > >>> libvirt/virsh and after looking at all the points where > > things > > could go > > > >>> wrong, honestly, I think we need to wait until there is > > support > > in > > > >>> libvirt/virsh to do this. virsh clearly has the > > capability > > internally > > > to > > > >> do > > > >>> live migration of storage, since it does this for live > > domain > > migration > > > >> of > > > >>> local storage between machines when migrating KVM domains > > from > > one host > > > >> to > > > >>> another, but that capability is not currently exposed in > > a way > > > Cloudstack > > > >>> could use, at least not on Centos 7. > > > >>> > > > >>> > > > >>>> On Jan 17, 2018, at 01:05, Piotr Pisz <pp...@pulab.pl> > > wrote: > > > >>>> > > > >>>> Hello, > > > >>>> > > > >>>> Is there a chance that one day it will be possible to > > migrate > > volume > > > >>> (root disk) of a live VM in KVM between storage pools (in > > CloudStack)? > > > >>>> Like a storage vMotion in Vmware. > > > >>>> > > > >>>> Best regards, > > > >>>> Piotr > > > >>>> > > > >>> > > > >>> > > > >> > > > > > > > > > > > > > > > > -- > > > > Rafael Weingärtner > > > > > > > > > > > > -- > > > > Andrija Panić > > > > > > > > -- -- Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 40 50 51 - 62 Fax: 030 / 40 50 51 - 19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
signature.asc
Description: This is a digitally signed message part
