Re: [one-users] slow live migration of VM's
Thanks Javier. The algorithm makes sense and the link pretty much validates what I figured was the process for live migration. Maybe the VM is a lot more active than I thought and it caused a lot of dirty pages. The last VM I have on the server to migrate off is much larger and active, so I guess I should plan for some downtime and do a cold migration where I pause the running system. I think that would take far less time to accomplish. Like I said in my last email, it the VM were usable during the migration, I really would not care how long it took. But having the VM effectively be 'down' makes live migration far less useful. My main concern was that I need to tune something in ONE to make it effective. It sounds like that is really not the case. The VM's that I previously migrated must just have been very low use and did not take long to sync. I will build out a test vm and do a non-live sync to validate the process. Thanks again, gary - Original Message - Here you have the algorithm for live migration: http://www.linux-kvm.org/page/Migration#Algorithm_.28the_short_version.29 In case your VM is doing some work that changes a lot of memory it will keep marking dirty pages and it will continue in the "Iteratively transfer all dirty pages (pages that were written to by the guest)." until the number of dirty pages is small enough so you don't notice the migration. Try stopping services changing memory in the VM before the migration. On Mon, Nov 25, 2013 at 3:32 AM, Gary S. Cuozzo wrote: > Hi all, > I was just moving some VM's around as a prep for some server maintenance, > via live migration. I moved 3 VM's which were all fairly small in size of > 1GB - 3GB RAM and 20GB - 60GB disk. I did the migrations 1 at a time. I > was surprised at how long the migrations took and also that each VM was > pretty much unusable during the migration. SSH sessions timed out and > websites were not able to be accessed. A VM with 3GB RAM allocation took > around 10 minutes to migrate. > > Is this typical? Or do I maybe have some sort of issue on my network or > config? We run a dedicated 1000Mbps management network for this type of > thing, and all data storage access is via separate 4x 1000Mbps bonded > channels. I would have expected 3GB of RAM data to be pretty quick. > Probably < 1 min. > > Are there any settings in ONE that should be tuned? I don't seem to recall > having an issue with migrations when I was just running virt-manager/libvirt > directly. > > The last VM I have to move is a pretty important one and it has 24GB RAM > allocated. So I think it would be easier/quicker to just shut the VM down, > disable the host for maintenance, and redeploy it. I'm hoping to avoid > that. > > Thanks for any advice, > gary > > > > ___ > Users mailing list > Users@lists.opennebula.org > http://lists.opennebula.org/listinfo.cgi/users-opennebula.org > -- Javier Fontán Muiños Developer OpenNebula - The Open Source Toolkit for Data Center Virtualization www.OpenNebula.org | @OpenNebula | github.com/jfontan ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] slow live migration of VM's
Thank you for the reply. The storage is shared storage via NFS, so no disk data should need to be transferred. Just the running VM. I was figuring just the running VM (RAM & state) needs to sync'd. The time to transfer wouldn't be such an issue if the VM's were functional during that time. The main issue is that the guests are completely unusable, so the migration may as well not be 'live'. Thanks, gary - Original Message - From: "Ionut Popovici" To: users@lists.opennebula.org Sent: Monday, November 25, 2013 2:52:51 AM Subject: Re: [one-users] slow live migration of VM's 1st20- 60Gb disk .. is preaty much .. 2nd transfer is done via SCP that si a lil slower than any other transfer but is more safer. 3nd It depends on READ on the host where are the VMS .. if they have other host working .. keep in mind that the machine share the IO of that datastore. On perfect Datastone it sould be around 6ms time to access data on HDD .. but on live datastore it my take a lil more. My test was starting nonpersistent image with size 50GB on a SSD drive it took around 4Mins and was not lunch via network, just copy via scp .. on same drive. On 11/25/2013 4:32 AM, Gary S. Cuozzo wrote: Hi all, I was just moving some VM's around as a prep for some server maintenance, via live migration. I moved 3 VM's which were all fairly small in size of 1GB - 3GB RAM and 20GB - 60GB disk. I did the migrations 1 at a time. I was surprised at how long the migrations took and also that each VM was pretty much unusable during the migration. SSH sessions timed out and websites were not able to be accessed. A VM with 3GB RAM allocation took around 10 minutes to migrate. Is this typical? Or do I maybe have some sort of issue on my network or config? We run a dedicated 1000Mbps management network for this type of thing, and all data storage access is via separate 4x 1000Mbps bonded channels. I would have expected 3GB of RAM data to be pretty quick. Probably < 1 min. Are there any settings in ONE that should be tuned? I don't seem to recall having an issue with migrations when I was just running virt-manager/libvirt directly. The last VM I have to move is a pretty important one and it has 24GB RAM allocated. So I think it would be easier/quicker to just shut the VM down, disable the host for maintenance, and redeploy it. I'm hoping to avoid that. Thanks for any advice, gary ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] slow live migration of VM's
Here you have the algorithm for live migration: http://www.linux-kvm.org/page/Migration#Algorithm_.28the_short_version.29 In case your VM is doing some work that changes a lot of memory it will keep marking dirty pages and it will continue in the "Iteratively transfer all dirty pages (pages that were written to by the guest)." until the number of dirty pages is small enough so you don't notice the migration. Try stopping services changing memory in the VM before the migration. On Mon, Nov 25, 2013 at 3:32 AM, Gary S. Cuozzo wrote: > Hi all, > I was just moving some VM's around as a prep for some server maintenance, > via live migration. I moved 3 VM's which were all fairly small in size of > 1GB - 3GB RAM and 20GB - 60GB disk. I did the migrations 1 at a time. I > was surprised at how long the migrations took and also that each VM was > pretty much unusable during the migration. SSH sessions timed out and > websites were not able to be accessed. A VM with 3GB RAM allocation took > around 10 minutes to migrate. > > Is this typical? Or do I maybe have some sort of issue on my network or > config? We run a dedicated 1000Mbps management network for this type of > thing, and all data storage access is via separate 4x 1000Mbps bonded > channels. I would have expected 3GB of RAM data to be pretty quick. > Probably < 1 min. > > Are there any settings in ONE that should be tuned? I don't seem to recall > having an issue with migrations when I was just running virt-manager/libvirt > directly. > > The last VM I have to move is a pretty important one and it has 24GB RAM > allocated. So I think it would be easier/quicker to just shut the VM down, > disable the host for maintenance, and redeploy it. I'm hoping to avoid > that. > > Thanks for any advice, > gary > > > > ___ > Users mailing list > Users@lists.opennebula.org > http://lists.opennebula.org/listinfo.cgi/users-opennebula.org > -- Javier Fontán Muiños Developer OpenNebula - The Open Source Toolkit for Data Center Virtualization www.OpenNebula.org | @OpenNebula | github.com/jfontan ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] slow live migration of VM's
1st20- 60Gb disk .. is preaty much .. 2nd transfer is done via SCP that si a lil slower than any other transfer but is more safer. 3nd It depends on READ on the host where are the VMS .. if they have other host working .. keep in mind that the machine share the IO of that datastore. On perfect Datastone it sould be around 6ms time to access data on HDD .. but on live datastore it my take a lil more. My test was starting nonpersistent image with size 50GB on a SSD drive it took around 4Mins and was not lunch via network, just copy via scp .. on same drive. On 11/25/2013 4:32 AM, Gary S. Cuozzo wrote: Hi all, I was just moving some VM's around as a prep for some server maintenance, via live migration. I moved 3 VM's which were all fairly small in size of 1GB - 3GB RAM and 20GB - 60GB disk. I did the migrations 1 at a time. I was surprised at how long the migrations took and also that each VM was pretty much unusable during the migration. SSH sessions timed out and websites were not able to be accessed. A VM with 3GB RAM allocation took around 10 minutes to migrate. Is this typical? Or do I maybe have some sort of issue on my network or config? We run a dedicated 1000Mbps management network for this type of thing, and all data storage access is via separate 4x 1000Mbps bonded channels. I would have expected 3GB of RAM data to be pretty quick. Probably < 1 min. Are there any settings in ONE that should be tuned? I don't seem to recall having an issue with migrations when I was just running virt-manager/libvirt directly. The last VM I have to move is a pretty important one and it has 24GB RAM allocated. So I think it would be easier/quicker to just shut the VM down, disable the host for maintenance, and redeploy it. I'm hoping to avoid that. Thanks for any advice, gary ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
[one-users] slow live migration of VM's
Hi all, I was just moving some VM's around as a prep for some server maintenance, via live migration. I moved 3 VM's which were all fairly small in size of 1GB - 3GB RAM and 20GB - 60GB disk. I did the migrations 1 at a time. I was surprised at how long the migrations took and also that each VM was pretty much unusable during the migration. SSH sessions timed out and websites were not able to be accessed. A VM with 3GB RAM allocation took around 10 minutes to migrate. Is this typical? Or do I maybe have some sort of issue on my network or config? We run a dedicated 1000Mbps management network for this type of thing, and all data storage access is via separate 4x 1000Mbps bonded channels. I would have expected 3GB of RAM data to be pretty quick. Probably < 1 min. Are there any settings in ONE that should be tuned? I don't seem to recall having an issue with migrations when I was just running virt-manager/libvirt directly. The last VM I have to move is a pretty important one and it has 24GB RAM allocated. So I think it would be easier/quicker to just shut the VM down, disable the host for maintenance, and redeploy it. I'm hoping to avoid that. Thanks for any advice, gary ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
