Re: [Openstack] Improving Xen support in the libvirt driver
On Wed, May 09, 2012 at 11:08:13PM -0600, Jim Fehlig wrote: Hi, I've been tinkering with improving Xen support in the libvirt driver and wanted to discuss a few issues before submitting patches. Even the latest upstream release of Xen (4.1.x) contains a rather old qemu, version 0.10.2, which rejects qcow2 images with cluster size 64K. The libvirt driver creates the COW image with cluster size of 2M. Is this for performance reasons? Any objections to removing that option and going with 'qemu-img create' default of 64K? In general larger cluster size does improve the performance of qcow2. I'm not sure how much of a delta we get by going from 64k to 2M though. If there's any doubt then I guess it could be made into a configuration parameter. In a setup with both Xen and KVM compute nodes, I've found a few options for controlling scheduling of an instance to the correct node. One option uses availability zones, e.g. # nova.conf on Xen compute nodes node_availability_zone=xen-hosts # launching a Xen PV instance nova boot --image xen-pv-image --availability_zone xen-hosts ... The other involves a recent commit adding additional capabilities for compute nodes [1] and the vm_mode image property [2] used by the XenServer driver to distinguish HVM vs PV images. E.g. # nova.conf on Xen compute nodes additional_compute_capabilities=pv,hvm # Set vm_mode property on Xen image glance update image-uuid vm_mode=pv I prefer that latter approach since vm_mode will be needed in the libvirt driver anyhow to create proper config for PV vs HVM instances. Currently, the driver creates usable config for PV instances, but needs some adjustments for HVM. Yes, tagging the image with details of its required guest ABI does seem like something we need to do to be able to properly support a choice betweeen PV HVM images. It is not very good the way we currently just hardcode PV only for Xen usage in the libvirt driver. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Improving Xen support in the libvirt driver
Il 10/05/2012 07:08, Jim Fehlig ha scritto: Hi, I've been tinkering with improving Xen support in the libvirt driver and wanted to discuss a few issues before submitting patches. Even the latest upstream release of Xen (4.1.x) contains a rather old qemu, version 0.10.2, which rejects qcow2 images with cluster size 64K. The libvirt driver creates the COW image with cluster size of 2M. Is this for performance reasons? Any objections to removing that option and going with 'qemu-img create' default of 64K? If I remember correctly, the qcow images are not the only problem with xen, but I'm far from the code for too long time. In the past (diablo), the method for counting the ram (and cpu perhaps?) did not work with xen and this affected the choices of the scheduler. I have no idea if this happens in essex/folsom. Regards, Muriel ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Improving Xen support in the libvirt driver
On Thu, May 10, 2012 at 09:06:58AM +0100, Daniel P. Berrange wrote: On Wed, May 09, 2012 at 11:08:13PM -0600, Jim Fehlig wrote: Hi, I've been tinkering with improving Xen support in the libvirt driver and wanted to discuss a few issues before submitting patches. Even the latest upstream release of Xen (4.1.x) contains a rather old qemu, version 0.10.2, which rejects qcow2 images with cluster size 64K. The libvirt driver creates the COW image with cluster size of 2M. Is this for performance reasons? Any objections to removing that option and going with 'qemu-img create' default of 64K? In general larger cluster size does improve the performance of qcow2. I'm not sure how much of a delta we get by going from 64k to 2M though. If there's any doubt then I guess it could be made into a configuration parameter. I had a quick chat with Kevin Wolf who's the upstream QEMU qcow2 maintainer and he said that 64k is the current recommended cluster size for qcow2. Above this size, the cost of COW becomes higher causing an overall drop in performance. Looking at GIT history, Nova has used cluster_size=2M since Vish first added qcow2 support, and there's no mention of why in the commit message. So unless further info comes to light, I'd say we ought to just switch to use qemu-img's default setting of 64K for both Xen and KVM. Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Improving Xen support in the libvirt driver
On Thu 10 May 2012 (10:41), Muriel wrote: If I remember correctly, the qcow images are not the only problem with xen, but I'm far from the code for too long time. In the past (diablo), the method for counting the ram (and cpu perhaps?) did not work with xen and this affected the choices of the scheduler. I have no idea if this happens in essex/folsom. I've sent to review some code [1] that tries to fix this issue [2]. [1] https://review.openstack.org/#/c/7296/ [2] https://bugs.launchpad.net/nova/+bug/997014 Regards, -- Álvaro López García al...@ifca.unican.es ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Improving Xen support in the libvirt driver
Il 10/05/2012 11:48, Alvaro Lopez ha scritto: On Thu 10 May 2012 (10:41), Muriel wrote: If I remember correctly, the qcow images are not the only problem with xen, but I'm far from the code for too long time. In the past (diablo), the method for counting the ram (and cpu perhaps?) did not work with xen and this affected the choices of the scheduler. I have no idea if this happens in essex/folsom. I've sent to review some code [1] that tries to fix this issue [2]. [1] https://review.openstack.org/#/c/7296/ [2] https://bugs.launchpad.net/nova/+bug/997014 Regards, Great! But there is a reason if are you using /proc/meminfo instead of getInfo when calculating the memory used? You know if there is a way to get, using libvirt, the reserved memory for dom0? Or the only solution is to read the configuration file of xen? Thanks, Muriel ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Improving Xen support in the libvirt driver
On Thu, May 10, 2012 at 03:17:59PM +0200, Muriel wrote: Il 10/05/2012 11:48, Alvaro Lopez ha scritto: On Thu 10 May 2012 (10:41), Muriel wrote: If I remember correctly, the qcow images are not the only problem with xen, but I'm far from the code for too long time. In the past (diablo), the method for counting the ram (and cpu perhaps?) did not work with xen and this affected the choices of the scheduler. I have no idea if this happens in essex/folsom. I've sent to review some code [1] that tries to fix this issue [2]. [1] https://review.openstack.org/#/c/7296/ [2] https://bugs.launchpad.net/nova/+bug/997014 Regards, Great! But there is a reason if are you using /proc/meminfo instead of getInfo when calculating the memory used? You know if there is a way to get, using libvirt, the reserved memory for dom0? Or the only solution is to read the configuration file of xen? Dom0 appears as just another guest in Xen/libvirt, so you can query its memory allocation using normal libvirt APIs Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Improving Xen support in the libvirt driver
On Thu 10 May 2012 (15:17), Muriel wrote: Great! But there is a reason if are you using /proc/meminfo instead of getInfo when calculating the memory used? You know if there is a way to get, using libvirt, the reserved memory for dom0? Or the only solution is to read the configuration file of xen? I calculated the memory looking into /proc/meminfo because if the memory is not limited (i.e. no dom0_mem option) the dom0 might take all the memory available, that then will be ballooned out. For example, in a machine with 16GB RAM you could have: # xm li NameID Mem VCPUs State Time(s) Domain-0 0 15030 8 r- 1312.8 If you query libvirt for the dom0 mem, the free memory will be around 1GB, but you can create a machine with more RAM (since ballooning is enabled): # xm li NameID Mem VCPUs State Time(s) Domain-0 0 9188 8 r- 1328.6 test 4 7000 4 -b 3.5 If the dom0 memory is fixed and ballooning is disabled, then yes, you can query libvirt directly. Regards, -- Álvaro López García al...@ifca.unican.es ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Improving Xen support in the libvirt driver
Il 10/05/2012 16:08, Alvaro Lopez ha scritto: On Thu 10 May 2012 (15:17), Muriel wrote: Great! But there is a reason if are you using /proc/meminfo instead of getInfo when calculating the memory used? You know if there is a way to get, using libvirt, the reserved memory for dom0? Or the only solution is to read the configuration file of xen? I calculated the memory looking into /proc/meminfo because if the memory is not limited (i.e. no dom0_mem option) the dom0 might take all the memory available, that then will be ballooned out. For example, in a machine with 16GB RAM you could have: # xm li NameID Mem VCPUs State Time(s) Domain-0 0 15030 8 r- 1312.8 If you query libvirt for the dom0 mem, the free memory will be around 1GB, but you can create a machine with more RAM (since ballooning is enabled): # xm li NameID Mem VCPUs State Time(s) Domain-0 0 9188 8 r- 1328.6 test 4 7000 4 -b 3.5 If the dom0 memory is fixed and ballooning is disabled, then yes, you can query libvirt directly. Regards, Thank you for remarking this point, sometimes I forget the balooning system :). I messed up the question: if you consider the case where the memory is limited (dom0_mem) the value of MemTotal in meminfo is wrong. Do you think it makes sense to take the total memory value from libvirt and from meminfo the rest? Thus it should work in both cases. Thanks, M. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Improving Xen support in the libvirt driver
On Thu 10 May 2012 (17:33), Muriel wrote: I messed up the question: if you consider the case where the memory is limited (dom0_mem) the value of MemTotal in meminfo is wrong. Do you think it makes sense to take the total memory value from libvirt and from meminfo the rest? Thus it should work in both cases. Yes, you're totally right, I missed that! I'll fix it ASAP and submit it again. Thank you for pointing it out, -- Álvaro López García al...@ifca.unican.es ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Improving Xen support in the libvirt driver
On May 10, 2012, at 1:56 AM, Daniel P. Berrange wrote: On Thu, May 10, 2012 at 09:06:58AM +0100, Daniel P. Berrange wrote: I had a quick chat with Kevin Wolf who's the upstream QEMU qcow2 maintainer and he said that 64k is the current recommended cluster size for qcow2. Above this size, the cost of COW becomes higher causing an overall drop in performance. Looking at GIT history, Nova has used cluster_size=2M since Vish first added qcow2 support, and there's no mention of why in the commit message. So unless further info comes to light, I'd say we ought to just switch to use qemu-img's default setting of 64K for both Xen and KVM. This is good info. Sounds like we should switch to 64K Vish___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Improving Xen support in the libvirt driver
On May 9, 2012, at 10:08 PM, Jim Fehlig wrote: Hi, I've been tinkering with improving Xen support in the libvirt driver and wanted to discuss a few issues before submitting patches. Awesome! Even the latest upstream release of Xen (4.1.x) contains a rather old qemu, version 0.10.2, which rejects qcow2 images with cluster size 64K. The libvirt driver creates the COW image with cluster size of 2M. Is this for performance reasons? Any objections to removing that option and going with 'qemu-img create' default of 64K? As per other email, 64K seems correct. In a setup with both Xen and KVM compute nodes, I've found a few options for controlling scheduling of an instance to the correct node. One option uses availability zones, e.g. # nova.conf on Xen compute nodes node_availability_zone=xen-hosts # launching a Xen PV instance nova boot --image xen-pv-image --availability_zone xen-hosts ... The other involves a recent commit adding additional capabilities for compute nodes [1] and the vm_mode image property [2] used by the XenServer driver to distinguish HVM vs PV images. E.g. # nova.conf on Xen compute nodes additional_compute_capabilities=pv,hvm # Set vm_mode property on Xen image glance update image-uuid vm_mode=pv I prefer that latter approach since vm_mode will be needed in the libvirt driver anyhow to create proper config for PV vs HVM instances. Currently, the driver creates usable config for PV instances, but needs some adjustments for HVM. Agree that this is best. Once general host aggregates[1] is done, the capabilities and the availability zone will move into aggregate metadata and it will just be making sure that we have reasonable image properties to help the scheduler place the guest correctly. Vish [1] https://blueprints.launchpad.net/nova/+spec/general-host-aggregates ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Improving Xen support in the libvirt driver
Hi, I've been tinkering with improving Xen support in the libvirt driver and wanted to discuss a few issues before submitting patches. Even the latest upstream release of Xen (4.1.x) contains a rather old qemu, version 0.10.2, which rejects qcow2 images with cluster size 64K. The libvirt driver creates the COW image with cluster size of 2M. Is this for performance reasons? Any objections to removing that option and going with 'qemu-img create' default of 64K? In a setup with both Xen and KVM compute nodes, I've found a few options for controlling scheduling of an instance to the correct node. One option uses availability zones, e.g. # nova.conf on Xen compute nodes node_availability_zone=xen-hosts # launching a Xen PV instance nova boot --image xen-pv-image --availability_zone xen-hosts ... The other involves a recent commit adding additional capabilities for compute nodes [1] and the vm_mode image property [2] used by the XenServer driver to distinguish HVM vs PV images. E.g. # nova.conf on Xen compute nodes additional_compute_capabilities=pv,hvm # Set vm_mode property on Xen image glance update image-uuid vm_mode=pv I prefer that latter approach since vm_mode will be needed in the libvirt driver anyhow to create proper config for PV vs HVM instances. Currently, the driver creates usable config for PV instances, but needs some adjustments for HVM. Regards, Jim [1] https://github.com/openstack/nova/commit/bd30eb36bbf2c5164ac47256355c543f6b77e064 [2] https://github.com/openstack/nova/commit/bd30eb36bbf2c5164ac47256355c543f6b77e064 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
