Hi Andrija, thank you so much for your detailled explanation! Looks like the my problem can be solved. :)
To summarize the information you provided: As long as CloudStack does not support volume live migration I could be using virsh with --copy-storage --xml. BUT: CentOS7 is lacking necessary features! Bad luck. I started out with CentOS7 as Disto. You suggest, that it could be worth trying the qemu/libvirt packages from the oVirt repository. I will look into this now. But if that gets complicated: Cloudstack documentation lists CentOS7 and Ubuntu 14.04 as supported Distros. Are there other not officially supported Distros/Version I could be using? I wanted to avoid the quite outdated Ubuntu 14.04 and did for that reason decide towards CentOS7. And another general question: How is CloudStack getting along with the Volumes of its VMs changing the storage repository without beeing informed about it. Does it get this information through polling, or do I have to manipulate the database? And to make things clearer: At the moment I am using storage attached through Gibrechannel using clustered LVM logic. Could also be changing to GFS2 on cLVM. Never heard anyone mentioning such a setup by now. Am I the only one running KVM on a proprietary storage system over Fibrechannel, are there limitation/problems to be expected from such a setup? Greetings, Melanie Am Mittwoch, den 25.09.2019, 11:46 +0200 schrieb Andrija Panic: > So, let me explain. > > Doing "online storage migration" aka live storage migration is > working for > CEPH/NFS --> SolidFire, starting from 4.11+ > Internally it is done in the same way as "virsh with --copy-storage- > all > --xml" in short > > Longer explanation: > Steps: > You create new volumes on the destination storage (SolidFire in this > case), > set QoS etc - simply prepare the destination volumes (empty volumes > atm). > On source host/VM, dump VM XML, edit XML, change disk section to > point to > new volume path, protocol, etc - and also the IP address for the VNC > (cloudstack requirement), save XMLT > Then you do "virsh with --copy-storage-all --xml. myEditedVM.xml ..." > stuff > that does the job. > Then NBD driver will be used to copy blocks from the source volumes > to the > destination volumes while that virsh command is working... (here's my > demo, > in details.. > https://www.youtube.com/watch?v=Eo8BuHBnVgg&list=PLEr0fbgkyLKyiPnNzPz7XDjxnmQNxjJWT&index=5&t=2s > ) > > This is yet to be extended/coded to support NFS-->NFS or CEPH-->CEPH > or > CEPH/NFS-->CEPH/NFS... should not be that much work, the logic is > there > (bit part of the code) > Also, starting from 4.12, you can actually (I believe using > identical > logic) migrate only ROOT volume that are on the LOCAL storage (local > disks) > to another host/local storage - but DATA disks are not supported. > > Now...imagine the feature is there - if using CentOS7, our friends at > RedHat have removed support for actually using live storage migration > (unless you are paying for RHEV - but it does work fine on CentOS6, > and > Ubuntu 14.04+ > > I recall "we" had to use qemu/libvirt from the "oVirt" repo which > DOES > (DID) support storage live migration (normal EV packages from the > Special > Interest Group (2.12 tested) - did NOT include this...) > > I can send you step-by-step .docx guide for manually mimicking what > is done > (in SolidFire, but identical logic for other storages) - but not sure > if > that still helps you... > > > Andrija > > On Wed, 25 Sep 2019 at 10:51, Melanie Desaive < > m.desa...@heinlein-support.de> > wrote: > > > Hi all, > > > > I am currently doing my first steps with KVM as hypervisor for > > CloudStack. I was shocked to realize that currently live volume > > migration between different shared storages is not supported with > > KVM. > > This is a feature I use intensively with XenServer. > > > > How do you get along with this limitation? I do really expect you > > to > > use some workarounds, or do you all only accept vm downtimes for a > > storage migration? > > > > With my first investigation I found three techniques mentioned and > > would like to ask for suggestions which to investigate deeper: > > > > x Eric describes a technique using snapshosts and pauses to do a > > live > > storage migration in this mailing list tread. > > x Dag suggests using virsh with --copy-storage-all --xml. > > x I found articles about using virsh blockcopy for storage live > > migration. > > > > Greetings, > > > > Melanie > > > > Am Freitag, den 02.02.2018, 15:55 +0100 schrieb Andrija Panic: > > > @Dag, you might want to check with Mike Tutkowski, how he > > > implemented > > > this > > > for the "online storage migration" from other storages (CEPH and > > > NFS > > > implemented so far as sources) to SolidFire. > > > > > > We are doing exactly the same demo/manual way (this is what Mike > > > has > > > sent > > > me back in the days), so perhaps you want to see how to translate > > > this into > > > general things (so ANY to ANY storage migration) inside > > > CloudStack. > > > > > > Cheers > > > > > > On 2 February 2018 at 10:28, Dag Sonstebo < > > > dag.sonst...@shapeblue.com > > > wrote: > > > > > > > All > > > > > > > > I am doing a bit of R&D around this for a client at the moment. > > > > I > > > > am > > > > semi-successful in getting live migrations to different storage > > > > pools to > > > > work. The method I’m using is as follows – this does not take > > > > into > > > > account > > > > any efficiency optimisation around the disk transfer (which is > > > > next > > > > on my > > > > list). The below should answer your question Eric about moving > > > > to a > > > > different location – and I am also working with your steps to > > > > see > > > > where I > > > > can improve the following. Keep in mind all of this is external > > > > to > > > > CloudStack – although CloudStack picks up the destination KVM > > > > host > > > > automatically it does not update the volume tables etc., > > > > neither > > > > does it do > > > > any housekeeping. > > > > > > > > 1) Ensure the same network bridges are up on source and > > > > destination > > > > – > > > > these are found with: > > > > > > > > [root@kvm1 ~]# virsh dumpxml 9 | grep source > > > > <source file='/mnt/00e88a7b-985f-3be8-b717- > > > > 0a59d8197640/d0ab5dd5- > > > > e3dd-47ac-a326-5ce3d47d194d'/> > > > > <source bridge='breth1-725'/> > > > > <source path='/dev/pts/3'/> > > > > <source path='/dev/pts/3'/> > > > > > > > > So from this make sure breth1-725 is up on the destionation > > > > host > > > > (do it > > > > the hard way or cheat and spin up a VM from same account and > > > > network on > > > > that host) > > > > > > > > 2) Find size of source disk and create stub disk in destination > > > > (this part > > > > can be made more efficient to speed up disk transfer – by doing > > > > similar > > > > things to what Eric is doing): > > > > > > > > [root@kvm1 ~]# qemu-img info /mnt/00e88a7b-985f-3be8-b717- > > > > 0a59d8197640/d0ab5dd5-e3dd-47ac-a326-5ce3d47d194d > > > > image: /mnt/00e88a7b-985f-3be8-b717-0a59d8197640/d0ab5dd5-e3dd- > > > > 47ac-a326-5ce3d47d194d > > > > file format: qcow2 > > > > virtual size: 8.0G (8589934592 bytes) > > > > disk size: 32M > > > > cluster_size: 65536 > > > > backing file: /mnt/00e88a7b-985f-3be8-b717- > > > > 0a59d8197640/3caaf4c9- > > > > eaec- > > > > 11e7-800b-06b4a401075c > > > > > > > > ###################### > > > > > > > > [root@kvm3 50848ff7-c6aa-3fdd-b487-27899bf2129c]# qemu-img > > > > create > > > > -f > > > > qcow2 d0ab5dd5-e3dd-47ac-a326-5ce3d47d194d 8G > > > > Formatting 'd0ab5dd5-e3dd-47ac-a326-5ce3d47d194d', fmt=qcow2 > > > > size=8589934592 encryption=off cluster_size=65536 > > > > [root@kvm3 50848ff7-c6aa-3fdd-b487-27899bf2129c]# qemu-img info > > > > d0ab5dd5-e3dd-47ac-a326-5ce3d47d194d > > > > image: d0ab5dd5-e3dd-47ac-a326-5ce3d47d194d > > > > file format: qcow2 > > > > virtual size: 8.0G (8589934592 bytes) > > > > disk size: 448K > > > > cluster_size: 65536 > > > > > > > > 3) Rewrite the new VM XML file for the destination with: > > > > a) New disk location, in this case this is just a new path > > > > (Eric – > > > > this > > > > answers your question) > > > > b) Different IP addresses for VNC – in this case 10.0.0.1 to > > > > 10.0.0.2 > > > > and carry out migration. > > > > > > > > [root@kvm1 ~]# virsh dumpxml 9 | sed -e 's/00e88a7b-985f-3be8- > > > > b717- > > > > 0a59d8197640/50848ff7-c6aa-3fdd-b487-27899bf2129c/g' | sed -e > > > > 's/ > > > > 10.0.0.1/10.0.0.2/g' > /root/i-2-14-VM.xml > > > > > > > > [root@kvm1 ~]# virsh migrate --live --persistent --copy- > > > > storage-all > > > > --xml > > > > /root/i-2-14-VM.xml i-2-14-VM qemu+tcp://10.0.0.2/system -- > > > > verbose > > > > --abort-on-error > > > > Migration: [ 25 %] > > > > > > > > 4) Once complete delete the source file. This can be done with > > > > extra > > > > switches on the virsh migrate command if need be. > > > > = = = > > > > > > > > In the simplest tests this works – destination VM remains > > > > online > > > > and has > > > > storage in new location – but it’s not persistent – sometimes > > > > the > > > > destination VM ends up in a paused state, and I’m working on > > > > how to > > > > get > > > > around this. I also noted virsh migrate has a migrate- > > > > setmaxdowntime which > > > > I think can be useful here. > > > > > > > > Regards, > > > > Dag Sonstebo > > > > Cloud Architect > > > > ShapeBlue > > > > > > > > On 01/02/2018, 20:30, "Andrija Panic" <andrija.pa...@gmail.com> > > > > wrote: > > > > > > > > Actually, we have this feature (we call this internally > > > > online-storage-migration) to migrate volume from CEPH/NFS > > > > to > > > > SolidFire > > > > (thanks to Mike Tutkowski) > > > > > > > > There is libvirt mechanism, where basically you start > > > > another > > > > PAUSED > > > > VM on > > > > another host (same name and same XML file, except the > > > > storage > > > > volumes > > > > are > > > > pointing to new storage, different paths, etc and maybe VNC > > > > listening > > > > address needs to be changed or so) and then you issue on > > > > original > > > > host/VM > > > > the live migrate command with few parameters... the libvirt > > > > will > > > > transaprently handle the copy data process from Soruce to > > > > New > > > > volumes, > > > > and > > > > after migration the VM will be alive (with new XML since > > > > have > > > > new > > > > volumes) > > > > on new host, while the original VM on original host is > > > > destroyed.... > > > > > > > > (I can send you manual for this, that is realted to SF, but > > > > idea is the > > > > same and you can exercies this on i.e. 2 NFS volumes on 2 > > > > different > > > > storages) > > > > > > > > This mechanism doesn't exist in ACS in general (AFAIK), > > > > except > > > > for when > > > > migrating to SolidFire. > > > > > > > > Perhaps community/DEV can help extend Mike's code to do > > > > same > > > > work on > > > > different storage types... > > > > > > > > Cheers > > > > > > > > > > > > dag.sonst...@shapeblue.com > > > > www.shapeblue.com > > > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > > > @shapeblue > > > > > > > > > > > > > > > > On 19 January 2018 at 18:45, Eric Green < > > > > eric.lee.gr...@gmail.com> > > > > wrote: > > > > > > > > > KVM is able to live migrate entire virtual machines > > > > complete > > > > with > > > > local > > > > > volumes (see 'man virsh') but does require nbd (Network > > > > Block > > > > Device) to be > > > > > installed on the destination host to do so. It may need > > > > installation > > > > of > > > > > later libvirt / qemu packages from OpenStack repositories > > > > on > > > > Centos > > > > 6, I'm > > > > > not sure, but just works on Centos 7. In any event, I > > > > have > > > > used this > > > > > functionality to move virtual machines between > > > > virtualization > > > > hosts > > > > on my > > > > > home network. It works. > > > > > > > > > > What is missing is the ability to live-migrate a disk > > > > from > > > > one shared > > > > > storage to another. The functionality built into virsh > > > > live- > > > > migrates > > > > the > > > > > volume ***to the exact same location on the new host***, > > > > so > > > > obviously is > > > > > useless for migrating the disk to a new location on > > > > shared > > > > storage. I > > > > > looked everywhere for the ability of KVM to live migrate > > > > a > > > > disk from > > > > point > > > > > A to point B all by itself, and found no such thing. > > > > libvirt/qemu > > > > has the > > > > > raw capabilities needed to do this, but it is not > > > > currently > > > > exposed > > > > as a > > > > > single API via the qemu console or virsh. It can be > > > > emulated > > > > via > > > > scripting > > > > > however: > > > > > > > > > > 1. Pause virtual machine > > > > > 2. Do qcow2 snapshot. > > > > > 3. Detach base disk, attach qcow2 snapshot > > > > > 4. unpause virtual machine > > > > > 5. copy qcow2 base file to new location > > > > > 6. pause virtual machine > > > > > 7. detach snapshot > > > > > 8. unsnapshot qcow2 snapshot at its new location. > > > > > 9. attach new base at new location. > > > > > 10. unpause virtual machine. > > > > > > > > > > Thing is, if that entire process is not built into the > > > > underlying > > > > > kvm/qemu/libvirt infrastructure as tested functionality > > > > with > > > > a > > > > defined API, > > > > > there's no guarantee that it will work seamlessly and > > > > will > > > > continue > > > > working > > > > > with the next release of the underlying infrastructure. > > > > This > > > > is using > > > > > multiple different tools to manipulate the qcow2 file and > > > > attach/detach > > > > > base disks to the running (but paused) kvm domain, and > > > > would > > > > have to > > > > be > > > > > tested against all variations of those tools on all > > > > supported > > > > Cloudstack > > > > > KVM host platforms. The test matrix looks pretty grim. > > > > > > > > > > By contrast, the migrate-with-local-storage process is > > > > built > > > > into > > > > virsh > > > > > and is tested by the distribution vendor and the set of > > > > tools > > > > provided with > > > > > the distribution is guaranteed to work with the virsh / > > > > libvirt/ qemu > > > > > distributed by the distribution vendor. That makes the > > > > test > > > > matrix > > > > for > > > > > move-with-local-storage look a lot simpler -- "is this > > > > functionality > > > > > supported by that version of virsh on that distribution? > > > > Yes? > > > > Enable > > > > it. > > > > > No? Don't enable it." > > > > > > > > > > I'd love to have live migration of disks on shared > > > > storage > > > > with > > > > Cloudstack > > > > > KVM, but not at the expense of reliability. Shutting down > > > > a > > > > virtual > > > > machine > > > > > in order to migrate one of its disks from one shared > > > > datastore to > > > > another > > > > > is not ideal, but at least it's guaranteed reliable. > > > > > > > > > > > > > > > > On Jan 19, 2018, at 04:54, Rafael Weingärtner < > > > > > rafaelweingart...@gmail.com> wrote: > > > > > > > > > > > > Hey Marc, > > > > > > It is very interesting that you are going to pick this > > > > up > > > > for KVM. > > > > I am > > > > > > working in a related issue for XenServer [1]. > > > > > > If you can confirm that KVM is able to live migrate > > > > local > > > > volumes > > > > to > > > > > other > > > > > > local storage or shared storage I could make the > > > > feature I > > > > am > > > > working on > > > > > > available to KVM as well. > > > > > > > > > > > > > > > > > > [1] > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-10240 > > > > > > > > > > > > On Thu, Jan 18, 2018 at 11:35 AM, Marc-Aurèle Brothier > > > > < > > > > > ma...@exoscale.ch> > > > > > > wrote: > > > > > > > > > > > >> There's a PR waiting to be fixed about live migration > > > > with > > > > local > > > > volume > > > > > for > > > > > >> KVM. So it will come at some point. I'm the one who > > > > made > > > > this PR > > > > but I'm > > > > > >> not using the upstream release so it's hard for me to > > > > debug the > > > > problem. > > > > > >> You can add yourself to the PR to get notify when > > > > things > > > > are > > > > moving on > > > > > it. > > > > > >> > > > > > >> https://github.com/apache/cloudstack/pull/1709 > > > > > >> > > > > > >> On Wed, Jan 17, 2018 at 10:56 AM, Eric Green < > > > > eric.lee.gr...@gmail.com> > > > > > >> wrote: > > > > > >> > > > > > >>> Theoretically on Centos 7 as the host KVM OS it could > > > > be > > > > done > > > > with a > > > > > >>> couple of pauses and the snapshotting mechanism built > > > > into > > > > qcow2, but > > > > > >> there > > > > > >>> is no simple way to do it directly via virsh, the > > > > libvirtd/qemu > > > > control > > > > > >>> program that is used to manage virtualization. It's > > > > not > > > > as with > > > > > issuing a > > > > > >>> simple vmotion 'migrate volume' call in Vmware. > > > > > >>> > > > > > >>> I scripted out how it would work without that direct > > > > support in > > > > > >>> libvirt/virsh and after looking at all the points > > > > where > > > > things > > > > could go > > > > > >>> wrong, honestly, I think we need to wait until there > > > > is > > > > support > > > > in > > > > > >>> libvirt/virsh to do this. virsh clearly has the > > > > capability > > > > internally > > > > > to > > > > > >> do > > > > > >>> live migration of storage, since it does this for > > > > live > > > > domain > > > > migration > > > > > >> of > > > > > >>> local storage between machines when migrating KVM > > > > domains > > > > from > > > > one host > > > > > >> to > > > > > >>> another, but that capability is not currently exposed > > > > in > > > > a way > > > > > Cloudstack > > > > > >>> could use, at least not on Centos 7. > > > > > >>> > > > > > >>> > > > > > >>>> On Jan 17, 2018, at 01:05, Piotr Pisz < > > > > pp...@pulab.pl> > > > > wrote: > > > > > >>>> > > > > > >>>> Hello, > > > > > >>>> > > > > > >>>> Is there a chance that one day it will be possible > > > > to > > > > migrate > > > > volume > > > > > >>> (root disk) of a live VM in KVM between storage pools > > > > (in > > > > CloudStack)? > > > > > >>>> Like a storage vMotion in Vmware. > > > > > >>>> > > > > > >>>> Best regards, > > > > > >>>> Piotr > > > > > >>>> > > > > > >>> > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Rafael Weingärtner > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Andrija Panić > > > > > > > > > > > > > > -- > > -- > > Heinlein Support GmbH > > Schwedter Str. 8/9b, 10119 Berlin > > > > https://www.heinlein-support.de > > > > Tel: 030 / 40 50 51 - 62 > > Fax: 030 / 40 50 51 - 19 > > > > Amtsgericht Berlin-Charlottenburg - HRB 93818 B > > Geschäftsführer: Peer Heinlein - Sitz: Berlin > > > > -- -- Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 40 50 51 - 62 Fax: 030 / 40 50 51 - 19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
signature.asc
Description: This is a digitally signed message part
