Re: [libvirt] [RFC] block I/O throttling: how to enable in libvirt
On Fri, Sep 02, 2011 at 08:34:19AM -0500, Adam Litke wrote: From: Adam Litke a...@us.ibm.com To: Zhi Yong Wu wu...@linux.vnet.ibm.com Cc: libvir-list@redhat.com Subject: Re: [libvirt] [RFC] block I/O throttling: how to enable in libvirt Message-ID: 20110902133419.gw15...@aglitke.rchland.ibm.com References: 20110901050531.gb17...@f15.cn.ibm.com 20110902021447.gd19...@f15.cn.ibm.com MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: 20110902021447.gd19...@f15.cn.ibm.com User-Agent: Mutt/1.5.21 (2010-09-15) X-Brightmail-Tracker: AA== X-Xagent-From: a...@us.ibm.com X-Xagent-To: wu...@linux.vnet.ibm.com X-Xagent-Gateway: vmsdvma.vnet.ibm.com (XAGENTU8 at VMSDVMA) On Fri, Sep 02, 2011 at 10:14:48AM +0800, Zhi Yong Wu wrote: On Thu, Sep 01, 2011 at 01:05:31PM +0800, Zhi Yong Wu wrote: HI, Adam, Now stefan, Daniel, and Gui all suggest extending blkiotune to keep libivrt unified interface. What do you think of it? It seems like it would be nice to extend the blkiotune API for this use case, but there is one problem. The blkiotune interface operates at the VM level and the QEMU throttling is at the disk level. So if you want per-disk throttling granularity the blkiotune API may not be suitable. I see 3 possible paths forward: 1) Accept the blkiotune limitation of a global setting and add a throttle option that assigns the same throttle value to all disks. blkiotune shares250/shares throttle1024/throttle /blkiotune + Nice for simplicity and ease of implementation - May be too simplistic (can't set different throttling for different disks) 2) Extend the blkiotune API to allow per-disk tunings: blkiotune shares250/shares throttle device='virtio-disk0'1024/throttle /blkiotune + Use an existing API - Unfortunately it doesn't look like this can be done cleanly 3) Use a new API virDomainSetBlkDevParameters() to set device specific tuning parameters: blkiotune shares250/shares /blkiotune ... disk type='file' device='disk' driver name='qemu' type='raw'/ source file='...'/ target dev='vda' bus='virtio'/ blkdevtune throttle1024/throttle Several I/O throttling tags should be supported, such as thrbps, thrbpsrd, thrbpswr, thriops, etc. /blkdevtune /disk + Throttle can be specified with the disk it is effecting - Proliferation of tuning APIs Obviously #1 is the best if you can accept the limitations of the API. #2 would #1 is not accepted by us. If it is adopted, the advantage of QEMU I/O throttling is not used fully. be nice if we could figure out how to extend virTypedParameter to represent a I guess that #2 is less extendable than #3. (disk, param) pair. I see #3 as a last resort. Did I miss any other solutions. To be honest, i prefer to #3. At first, i plan to check how blkiotune works currently. Did I just manage to completely over-complicate this? :) NO, they are nice. thanks. Regards, Zhiyong Wu -- Adam Litke a...@us.ibm.com IBM Linux Technology Center -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] block I/O throttling: how to enable in libvirt
On Thu, Sep 01, 2011 at 01:05:31PM +0800, Zhi Yong Wu wrote: On Wed, Aug 31, 2011 at 08:18:19AM +0100, Stefan Hajnoczi wrote: On Tue, Aug 30, 2011 at 2:46 PM, Adam Litke a...@us.ibm.com wrote: On Tue, Aug 30, 2011 at 09:53:33AM +0100, Stefan Hajnoczi wrote: I/O throttling can be applied independently to each -drive attached to a guest and supports throughput/iops limits. For more information on this QEMU feature and a comparison with blkio-controller, see Ryan Harper's KVM Forum 2011 presentation: http://www.linux-kvm.org/wiki/images/7/72/2011-forum-keep-a-limit-on-it-io-throttling-in-qemu.pdf From the presentation, it seems that both the cgroups method the the qemu method offer comparable control (assuming a block device) so it might possible to apply either method from the same API in a transparent manner. Am I correct or are we suggesting that the Qemu throttling approach should always be used for Qemu domains? QEMU I/O throttling does not provide a proportional share mechanism. So you cannot assign weights to VMs and let them receive a fraction of the available disk time. That is only supported by cgroups blkio-controller because it requires a global view which QEMU does not have. So I think the two are complementary: If proportional share should be used on a host block device, use cgroups blkio-controller. Otherwise use QEMU I/O throttling. Stefan, Do you agree with introducing one new libvirt command blkiothrottle now? If so, i will work on the code draft to make it work. No, I think that the blkiotune command should be extended to support QEMU I/O throttling. This is not new functionality, we already have cgroups blkio-controller support today. Therefore I think it makes sense to keep a unified interface instead of adding a new command. Stefan -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] block I/O throttling: how to enable in libvirt
On Thu, Sep 01, 2011 at 09:11:49AM +0100, Stefan Hajnoczi wrote: On Thu, Sep 01, 2011 at 01:05:31PM +0800, Zhi Yong Wu wrote: On Wed, Aug 31, 2011 at 08:18:19AM +0100, Stefan Hajnoczi wrote: On Tue, Aug 30, 2011 at 2:46 PM, Adam Litke a...@us.ibm.com wrote: On Tue, Aug 30, 2011 at 09:53:33AM +0100, Stefan Hajnoczi wrote: I/O throttling can be applied independently to each -drive attached to a guest and supports throughput/iops limits. For more information on this QEMU feature and a comparison with blkio-controller, see Ryan Harper's KVM Forum 2011 presentation: http://www.linux-kvm.org/wiki/images/7/72/2011-forum-keep-a-limit-on-it-io-throttling-in-qemu.pdf From the presentation, it seems that both the cgroups method the the qemu method offer comparable control (assuming a block device) so it might possible to apply either method from the same API in a transparent manner. Am I correct or are we suggesting that the Qemu throttling approach should always be used for Qemu domains? QEMU I/O throttling does not provide a proportional share mechanism. So you cannot assign weights to VMs and let them receive a fraction of the available disk time. That is only supported by cgroups blkio-controller because it requires a global view which QEMU does not have. So I think the two are complementary: If proportional share should be used on a host block device, use cgroups blkio-controller. Otherwise use QEMU I/O throttling. Stefan, Do you agree with introducing one new libvirt command blkiothrottle now? If so, i will work on the code draft to make it work. No, I think that the blkiotune command should be extended to support QEMU I/O throttling. This is not new functionality, we already have cgroups blkio-controller support today. Therefore I think it makes sense to keep a unified interface instead of adding a new command. Agreed, the virDomainGetBlkioParameters/virDomainSetBlkioParameters APIs, and blkio virsh command are intended to be a generic interface for setting any block related tuning parameters, regardless of what the underling implementation is. So any use of QEMU I/O throttling features should be added to those APIs/commands. Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] block I/O throttling: how to enable in libvirt
On 2011-9-1 16:11, Stefan Hajnoczi wrote: On Thu, Sep 01, 2011 at 01:05:31PM +0800, Zhi Yong Wu wrote: On Wed, Aug 31, 2011 at 08:18:19AM +0100, Stefan Hajnoczi wrote: On Tue, Aug 30, 2011 at 2:46 PM, Adam Litke a...@us.ibm.com wrote: On Tue, Aug 30, 2011 at 09:53:33AM +0100, Stefan Hajnoczi wrote: I/O throttling can be applied independently to each -drive attached to a guest and supports throughput/iops limits. For more information on this QEMU feature and a comparison with blkio-controller, see Ryan Harper's KVM Forum 2011 presentation: http://www.linux-kvm.org/wiki/images/7/72/2011-forum-keep-a-limit-on-it-io-throttling-in-qemu.pdf From the presentation, it seems that both the cgroups method the the qemu method offer comparable control (assuming a block device) so it might possible to apply either method from the same API in a transparent manner. Am I correct or are we suggesting that the Qemu throttling approach should always be used for Qemu domains? QEMU I/O throttling does not provide a proportional share mechanism. So you cannot assign weights to VMs and let them receive a fraction of the available disk time. That is only supported by cgroups blkio-controller because it requires a global view which QEMU does not have. So I think the two are complementary: If proportional share should be used on a host block device, use cgroups blkio-controller. Otherwise use QEMU I/O throttling. Stefan, Do you agree with introducing one new libvirt command blkiothrottle now? If so, i will work on the code draft to make it work. No, I think that the blkiotune command should be extended to support QEMU I/O throttling. This is not new functionality, we already have cgroups blkio-controller support today. Therefore I think it makes sense to keep a unified interface instead of adding a new command. Agreed. Proportional controlling interfaces and throttling interfaces are all the same cgroup subsystem. So Just extend blkiotune to add new options to support throttling tuning. Thanks, Gui Stefan -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] block I/O throttling: how to enable in libvirt
On Thu, Sep 01, 2011 at 09:49:34AM +0100, Daniel P. Berrange wrote: From: Daniel P. Berrange berra...@redhat.com To: Stefan Hajnoczi stefa...@gmail.com Cc: Zhi Yong Wu wu...@linux.vnet.ibm.com, libvir-list@redhat.com, a...@us.ibm.com, qemu-de...@nongnu.org, guijianf...@cn.fujitsu.com, hu...@cn.fujitsu.com, zwu.ker...@gmail.com Subject: Re: [RFC] block I/O throttling: how to enable in libvirt Message-ID: 20110901084934.ga14...@redhat.com Reply-To: Daniel P. Berrange berra...@redhat.com References: 20110901050531.gb17...@f15.cn.ibm.com 20110901081149.GB14245@stefanha-thinkpad.localdomain MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: 20110901081149.GB14245@stefanha-thinkpad.localdomain User-Agent: Mutt/1.5.21 (2010-09-15) X-Scanned-By: MIMEDefang 2.67 on 10.5.11.12 X-Brightmail-Tracker: AA== X-Xagent-From: berra...@redhat.com X-Xagent-To: wu...@linux.vnet.ibm.com X-Xagent-Gateway: emeavsc.vnet.ibm.com (XAGENTU at EMEAVSC) On Thu, Sep 01, 2011 at 09:11:49AM +0100, Stefan Hajnoczi wrote: On Thu, Sep 01, 2011 at 01:05:31PM +0800, Zhi Yong Wu wrote: On Wed, Aug 31, 2011 at 08:18:19AM +0100, Stefan Hajnoczi wrote: On Tue, Aug 30, 2011 at 2:46 PM, Adam Litke a...@us.ibm.com wrote: On Tue, Aug 30, 2011 at 09:53:33AM +0100, Stefan Hajnoczi wrote: I/O throttling can be applied independently to each -drive attached to a guest and supports throughput/iops limits. For more information on this QEMU feature and a comparison with blkio-controller, see Ryan Harper's KVM Forum 2011 presentation: http://www.linux-kvm.org/wiki/images/7/72/2011-forum-keep-a-limit-on-it-io-throttling-in-qemu.pdf From the presentation, it seems that both the cgroups method the the qemu method offer comparable control (assuming a block device) so it might possible to apply either method from the same API in a transparent manner. Am I correct or are we suggesting that the Qemu throttling approach should always be used for Qemu domains? QEMU I/O throttling does not provide a proportional share mechanism. So you cannot assign weights to VMs and let them receive a fraction of the available disk time. That is only supported by cgroups blkio-controller because it requires a global view which QEMU does not have. So I think the two are complementary: If proportional share should be used on a host block device, use cgroups blkio-controller. Otherwise use QEMU I/O throttling. Stefan, Do you agree with introducing one new libvirt command blkiothrottle now? If so, i will work on the code draft to make it work. No, I think that the blkiotune command should be extended to support QEMU I/O throttling. This is not new functionality, we already have cgroups blkio-controller support today. Therefore I think it makes sense to keep a unified interface instead of adding a new command. Agreed, the virDomainGetBlkioParameters/virDomainSetBlkioParameters APIs, and blkio virsh command are intended to be a generic interface for setting any block related tuning parameters, regardless of what the underling implementation is. So any use of QEMU I/O throttling features should be added to those APIs/commands. thanks for your suggestions. Regards, Zhi Yong Wu Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] block I/O throttling: how to enable in libvirt
On Thu, Sep 01, 2011 at 01:05:31PM +0800, Zhi Yong Wu wrote: - Forwarded message from Zhi Yong Wu wu...@linux.vnet.ibm.com - Date: Thu, 1 Sep 2011 11:55:17 +0800 From: Zhi Yong Wu wu...@linux.vnet.ibm.com To: Stefan Hajnoczi stefa...@gmail.com Cc: Daniel P. Berrange berra...@redhat.com, Stefan Hajnoczi stefa...@gmail.com, Adam Litke a...@us.ibm.com, Zhi Yong Wu wu...@linux.vnet.ibm.com, QEMU Developers qemu-de...@nongnu.org, guijianf...@cn.fujitsu.com, hu...@cn.fujitsu.com Subject: [RFC] block I/O throttling: how to enable in libvirt Message-ID: 20110901035517.gd16...@f15.cn.ibm.com References: CAEH94Li_C=BOe2gV8NyM48njYWMBAo9MTGc1eUOh-Y=cods...@mail.gmail.com CAJSP0QW1CPCokX=F5z7y==vn1s4wh0vtoaq7oj4kc7f7uqm...@mail.gmail.com 20110830134636.gb29...@aglitke.rchland.ibm.com CAJSP0QUHm=y8xjc_kxrg7uffzt3k_xddfqb--sxjc+c0gjo...@mail.gmail.com MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: CAJSP0QUHm=y8xjc_kxrg7uffzt3k_xddfqb--sxjc+c0gjo...@mail.gmail.com User-Agent: Mutt/1.5.21 (2010-09-15) X-Xagent-From: wu...@linux.vnet.ibm.com X-Xagent-To: wu...@linux.vnet.ibm.com X-Xagent-Gateway: vmsdvma.vnet.ibm.com (XAGENTU7 at VMSDVMA) On Wed, Aug 31, 2011 at 08:18:19AM +0100, Stefan Hajnoczi wrote: Subject: Re: The design choice for how to enable block I/O throttling function in libvirt From: Stefan Hajnoczi stefa...@gmail.com To: Adam Litke a...@us.ibm.com Cc: libvir-list@redhat.com, Daniel P. Berrange berra...@redhat.com, Zhi Yong Wu wu...@linux.vnet.ibm.com, Zhi Yong Wu zwu.ker...@gmail.com Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Brightmail-Tracker: AA== X-Xagent-From: stefa...@gmail.com X-Xagent-To: wu...@linux.vnet.ibm.com X-Xagent-Gateway: bldgate.vnet.ibm.com (XAGENTU7 at BLDGATE) On Tue, Aug 30, 2011 at 2:46 PM, Adam Litke a...@us.ibm.com wrote: On Tue, Aug 30, 2011 at 09:53:33AM +0100, Stefan Hajnoczi wrote: On Tue, Aug 30, 2011 at 3:55 AM, Zhi Yong Wu zwu.ker...@gmail.com wrote: I am trying to enable block I/O throttling function in libvirt. But currently i met some design questions, and don't make sure if we should extend blkiotune to support block I/O throttling or introduce one new libvirt command blkiothrottle to cover it or not. If you have some better idea, pls don't hesitate to drop your comments. A little bit of context: this discussion is about adding libvirt support for QEMU disk I/O throttling. Thanks for the additional context Stefan. Today libvirt supports the cgroups blkio-controller, which handles proportional shares and throughput/iops limits on host block devices. blkio-controller does not support network file systems (NFS) or other QEMU remote block drivers (curl, Ceph/rbd, sheepdog) since they are not host block devices. QEMU I/O throttling works with all types of -drive and therefore complements blkio-controller. The first question that pops into my mind is: Should a user need to understand when to use the cgroups blkio-controller vs. the QEMU I/O throttling method? In my opinion, it would be nice if libvirt had a single interface for block I/O throttling and libvirt would decide which mechanism to use based on the type of device and the specific limits that need to be set. Yes, I agree it would be simplest to pick the right mechanism, depending on the type of throttling the user wants. More below. I/O throttling can be applied independently to each -drive attached to a guest and supports throughput/iops limits. For more information on this QEMU feature and a comparison with blkio-controller, see Ryan Harper's KVM Forum 2011 presentation: http://www.linux-kvm.org/wiki/images/7/72/2011-forum-keep-a-limit-on-it-io-throttling-in-qemu.pdf From the presentation, it seems that both the cgroups method the the qemu method offer comparable control (assuming a block device) so it might possible to apply either method from the same API in a transparent manner. Am I correct or are we suggesting that the Qemu throttling approach should always be used for Qemu domains? QEMU I/O throttling does not provide a proportional share mechanism. So you cannot assign weights to VMs and let them receive a fraction of the available disk time. That is only supported by cgroups blkio-controller because it requires a global view which QEMU does not have. So I think the two are complementary: If proportional share should be used on a host block device, use cgroups blkio-controller. Otherwise use QEMU I/O throttling. Stefan, Do you agree with introducing one new libvirt command blkiothrottle now? If so, i will work on the code draft to make it work. Daniel and other maintainers, If you are available, can you make some comments for us?:) HI, Adam, Now stefan, Daniel, and Gui all suggest extending blkiotune to keep libivrt unified interface. What do you think of
