Re: [openstack-dev] Efficiently pin running VMs to physical CPUs automatically
On 2013-06-21 08:58:38, Giorgio Franceschi wrote: > On 21-06-13 07:24, Kashyap Chamarthy wrote: >> On 06/20/2013 10:18 PM, Russell Bryant wrote: >>> On 06/20/2013 10:36 AM, Giorgio Franceschi wrote: Hello, I created a blueprint for the implementation of: A tool for pinning automatically each running virtual CPU to a physical one in the most efficient way, balancing load across sockets/cores and maximizing cache sharing/minimizing cache misses. Ideally able to be run on-demand, as a periodic job, or be triggered by events on the host (vm spawn/destroy). Find it at https://blueprints.launchpad.net/nova/+spec/auto-cpu-pinning Any inputappreciated! >>> I'm actually surprised to see a new tool for this kind of thing. >>> >>> Have you seen numad? >> >> And a related post by Dan Berrange (but on lower layers -- libvirt) which >> explains how to >> do vcpu pinning and control NUMA affinity -- >> http://berrange.com/posts/2010/02/12/controlling-guest-cpu-numa-affinity-in-libvirt-with-qemu-kvm-xen/ >> >> >> >> > Yes Kashyap, thanks for the link, I had read that article while > researching the problem. It addresses the same issues, but the solution > is based on install-time, static configuration. We want something > requiring as little config as possible and able to allocate VMs at > runtime. Basically, what the author suggests in section "Fine tuning CPU > affinity at runtime", but automated for large-scale, hands-off environments. > > Russell, thanks for your suggestion, I did not know of numad. It looks > interesting, but the way I understand it, it is a system-wide > NUMA-binding daemon without any configuration options or fine-tuning > capabilities. We want something that only deals with relevant kvm > processes, not all processes on the system, and also we would like to > make it configurable so that VM can advertise their "pinnability", > because sometimes you might not want all running domains on a host to be > treated the same. This is planned for a future release. Would numad be > suited to this task, in your opinion? I suppose one could use it as a > querying tool, with the -w switch, and apply its suggestions > selectively. Then, it would basically replace the strategy-making part > of pinhead. I will investigate this shortly. > > Thanks everyone for your help, any further input much appreciated! We've used vm-balancer.py in the past, and it seems to work quite well (I don't see numad packaged for ubuntu, nor numad support built into libvirt for ubuntu) http://code.google.com/p/vm-balancer-numa/downloads The following blog post explains it, and the comments contain different versions of a shell script that controls it: http://blog.opennebula.org/?p=3193 Regards, Marcus. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Efficiently pin running VMs to physical CPUs automatically
Very glad to see that, too. We also have some thoughts to look at the usage of CPU resources (like cache) for each VM, and adjust to aggregate and bind VMs to physical CPUs for QoS. Look forward to seeing the initiatives proposed below to move forward. Thanks. -- Shane > -Original Message- > From: Qing He [mailto:qing...@radisys.com] > Sent: Saturday, June 22, 2013 2:11 AM > To: OpenStack Development Mailing List > Subject: Re: [openstack-dev] Efficiently pin running VMs to physical CPUs > automatically > > > Russell, > That's great initiative! I'm wondering if a framework/abstraction layer can > be > built so that different algorithms can be plugged in. I'm sure we can learn > from > the none-vm world: > http://en.wikipedia.org/wiki/Processor_affinity > > Thanks, > Qing > > > > > -Original Message- > > From: Russell Bryant [mailto:rbry...@redhat.com] > > Sent: 20 June 2013 17:48 > > To: openstack-dev@lists.openstack.org > > Subject: Re: [openstack-dev] Efficiently pin running VMs to physical > > CPUs automatically > > > > On 06/20/2013 10:36 AM, Giorgio Franceschi wrote: > >> Hello, I created a blueprint for the implementation of: > >> > >> A tool for pinning automatically each running virtual CPU to a > >> physical one in the most efficient way, balancing load across > >> sockets/cores and maximizing cache sharing/minimizing cache misses. > >> Ideally able to be run on-demand, as a periodic job, or be triggered > >> by events on the host (vm spawn/destroy). > >> > >> Find it at > >> https://blueprints.launchpad.net/nova/+spec/auto-cpu-pinning > >> > >> Any inputappreciated! > > > > I'm actually surprised to see a new tool for this kind of thing. > > > > Have you seen numad? > > > > -- > > Russell Bryant > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Efficiently pin running VMs to physical CPUs automatically
Russell, That's great initiative! I'm wondering if a framework/abstraction layer can be built so that different algorithms can be plugged in. I'm sure we can learn from the none-vm world: http://en.wikipedia.org/wiki/Processor_affinity Thanks, Qing > > -Original Message- > From: Russell Bryant [mailto:rbry...@redhat.com] > Sent: 20 June 2013 17:48 > To: openstack-dev@lists.openstack.org > Subject: Re: [openstack-dev] Efficiently pin running VMs to physical > CPUs automatically > > On 06/20/2013 10:36 AM, Giorgio Franceschi wrote: >> Hello, I created a blueprint for the implementation of: >> >> A tool for pinning automatically each running virtual CPU to a >> physical one in the most efficient way, balancing load across >> sockets/cores and maximizing cache sharing/minimizing cache misses. >> Ideally able to be run on-demand, as a periodic job, or be triggered >> by events on the host (vm spawn/destroy). >> >> Find it at >> https://blueprints.launchpad.net/nova/+spec/auto-cpu-pinning >> >> Any inputappreciated! > > I'm actually surprised to see a new tool for this kind of thing. > > Have you seen numad? > > -- > Russell Bryant > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Efficiently pin running VMs to physical CPUs automatically
> It seems that numad is libvirt specific - is that the case? Hi, Hyper-V 2012 supports NUMA as well. It'd be great to plan an hypervisor independent solution from the start. On 21.06.2013, at 11:13, "Bob Ball" wrote: > It seems that numad is libvirt specific - is that the case? > > I'm not sure if there is a daemon for other hypervisors but would it make > sense to have this functionality in OpenStack so we can extend it to work for > each hypervisor allowing it to control the affinity in their own way? I > guess this would need the Pinhead tool to either support multiple hypervisors > or provide the pinning strategy to Nova which could then invoke the > individual drivers. > > Outside numa optimisations I think there are good reasons for Nova to support > modifying the affinity / pinning rules - for example I can imagine that some > flavours might be permitted dedicated or isolated vCPUs? Integrating this > tool would allow us to provide it further hints/rules defined by the flavour > or administrator. > > Bob > > -Original Message- > From: Russell Bryant [mailto:rbry...@redhat.com] > Sent: 20 June 2013 17:48 > To: openstack-dev@lists.openstack.org > Subject: Re: [openstack-dev] Efficiently pin running VMs to physical CPUs > automatically > > On 06/20/2013 10:36 AM, Giorgio Franceschi wrote: >> Hello, I created a blueprint for the implementation of: >> >> A tool for pinning automatically each running virtual CPU to a physical >> one in the most efficient way, balancing load across sockets/cores and >> maximizing cache sharing/minimizing cache misses. Ideally able to be run >> on-demand, as a periodic job, or be triggered by events on the host (vm >> spawn/destroy). >> >> Find it at https://blueprints.launchpad.net/nova/+spec/auto-cpu-pinning >> >> Any inputappreciated! > > I'm actually surprised to see a new tool for this kind of thing. > > Have you seen numad? > > -- > Russell Bryant > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Efficiently pin running VMs to physical CPUs automatically
On Fri, Jun 21, 2013 at 05:09:24PM +, Bob Ball wrote: > Sorry, my point about numad being libvirt specific was that I > couldn't find references to other hypervisors using numad for > their placement. I recognise that it's not _tied_ to libvirt > but the reality seems to be that only libvirt uses it. > > Xen, for example, can't use numad because dom0 might only know > about a subset of the system - it'd make sense for dom0 to only > be placed on a single numa node. Xen does of course have its own > automatic placement to take account of the numa nodes - I assume > this is also true of other hypervisors. That is merely a limitation of the current impl, not a technology roadblock. numad could easily be made to ask the Xen hypervisor what the topology of the entire host was if that was desired for Xen. Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Efficiently pin running VMs to physical CPUs automatically
Sorry, my point about numad being libvirt specific was that I couldn't find references to other hypervisors using numad for their placement. I recognise that it's not _tied_ to libvirt but the reality seems to be that only libvirt uses it. Xen, for example, can't use numad because dom0 might only know about a subset of the system - it'd make sense for dom0 to only be placed on a single numa node. Xen does of course have its own automatic placement to take account of the numa nodes - I assume this is also true of other hypervisors. Perhaps my question is a broader one about whether we want Nova to have some influence in the pinning rules, or if we just want to ignore numa placement and let each hypervisor to do it in its own way? Bob -Original Message- From: Daniel P. Berrange [mailto:berra...@redhat.com] Sent: 21 June 2013 10:55 To: OpenStack Development Mailing List Subject: Re: [openstack-dev] Efficiently pin running VMs to physical CPUs automatically On Fri, Jun 21, 2013 at 09:10:32AM +, Bob Ball wrote: > It seems that numad is libvirt specific - is that the case? No, it is a completely independant project https://git.fedorahosted.org/git/numad.git It existed before libvirt started using it for automatic placement. Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Efficiently pin running VMs to physical CPUs automatically
On Thu, Jun 20, 2013 at 12:48:16PM -0400, Russell Bryant wrote: > On 06/20/2013 10:36 AM, Giorgio Franceschi wrote: > > Hello, I created a blueprint for the implementation of: > > > > A tool for pinning automatically each running virtual CPU to a physical > > one in the most efficient way, balancing load across sockets/cores and > > maximizing cache sharing/minimizing cache misses. Ideally able to be run > > on-demand, as a periodic job, or be triggered by events on the host (vm > > spawn/destroy). > > > > Find it at https://blueprints.launchpad.net/nova/+spec/auto-cpu-pinning > > > > Any inputappreciated! > > I'm actually surprised to see a new tool for this kind of thing. > > Have you seen numad? The approach used by 'pinhead' tool dscribed in the blueprint seems to be pretty much equivalent to what 'numad' is already providing for Libvirt KVM and LXC guests. NB, numad is actually a standalone program for optimizing NUMA placement of any processes on a server. Libvirt talks to it when starting a guest to request info on where best to place the guest. Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Efficiently pin running VMs to physical CPUs automatically
On Fri, Jun 21, 2013 at 09:10:32AM +, Bob Ball wrote: > It seems that numad is libvirt specific - is that the case? No, it is a completely independant project https://git.fedorahosted.org/git/numad.git It existed before libvirt started using it for automatic placement. Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Efficiently pin running VMs to physical CPUs automatically
It seems that numad is libvirt specific - is that the case? I'm not sure if there is a daemon for other hypervisors but would it make sense to have this functionality in OpenStack so we can extend it to work for each hypervisor allowing it to control the affinity in their own way? I guess this would need the Pinhead tool to either support multiple hypervisors or provide the pinning strategy to Nova which could then invoke the individual drivers. Outside numa optimisations I think there are good reasons for Nova to support modifying the affinity / pinning rules - for example I can imagine that some flavours might be permitted dedicated or isolated vCPUs? Integrating this tool would allow us to provide it further hints/rules defined by the flavour or administrator. Bob -Original Message- From: Russell Bryant [mailto:rbry...@redhat.com] Sent: 20 June 2013 17:48 To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] Efficiently pin running VMs to physical CPUs automatically On 06/20/2013 10:36 AM, Giorgio Franceschi wrote: > Hello, I created a blueprint for the implementation of: > > A tool for pinning automatically each running virtual CPU to a physical > one in the most efficient way, balancing load across sockets/cores and > maximizing cache sharing/minimizing cache misses. Ideally able to be run > on-demand, as a periodic job, or be triggered by events on the host (vm > spawn/destroy). > > Find it at https://blueprints.launchpad.net/nova/+spec/auto-cpu-pinning > > Any inputappreciated! I'm actually surprised to see a new tool for this kind of thing. Have you seen numad? -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Efficiently pin running VMs to physical CPUs automatically
On 21-06-13 07:24, Kashyap Chamarthy wrote: On 06/20/2013 10:18 PM, Russell Bryant wrote: On 06/20/2013 10:36 AM, Giorgio Franceschi wrote: Hello, I created a blueprint for the implementation of: A tool for pinning automatically each running virtual CPU to a physical one in the most efficient way, balancing load across sockets/cores and maximizing cache sharing/minimizing cache misses. Ideally able to be run on-demand, as a periodic job, or be triggered by events on the host (vm spawn/destroy). Find it at https://blueprints.launchpad.net/nova/+spec/auto-cpu-pinning Any inputappreciated! I'm actually surprised to see a new tool for this kind of thing. Have you seen numad? And a related post by Dan Berrange (but on lower layers -- libvirt) which explains how to do vcpu pinning and control NUMA affinity -- http://berrange.com/posts/2010/02/12/controlling-guest-cpu-numa-affinity-in-libvirt-with-qemu-kvm-xen/ Yes Kashyap, thanks for the link, I had read that article while researching the problem. It addresses the same issues, but the solution is based on install-time, static configuration. We want something requiring as little config as possible and able to allocate VMs at runtime. Basically, what the author suggests in section "Fine tuning CPU affinity at runtime", but automated for large-scale, hands-off environments. Russell, thanks for your suggestion, I did not know of numad. It looks interesting, but the way I understand it, it is a system-wide NUMA-binding daemon without any configuration options or fine-tuning capabilities. We want something that only deals with relevant kvm processes, not all processes on the system, and also we would like to make it configurable so that VM can advertise their "pinnability", because sometimes you might not want all running domains on a host to be treated the same. This is planned for a future release. Would numad be suited to this task, in your opinion? I suppose one could use it as a querying tool, with the -w switch, and apply its suggestions selectively. Then, it would basically replace the strategy-making part of pinhead. I will investigate this shortly. Thanks everyone for your help, any further input much appreciated! De informatie in dit e-mailbericht en eventuele bijlagen is vertrouwelijk en is alleen bestemd voor de beoogde ontvanger(s). Indien u dit bericht ten onrechte heeft ontvangen, wordt u verzocht de verzender daarvan in kennis te stellen en het bericht te vernietigen. Het is niet toegestaan de hierin opgenomen informatie op welke wijze dan ook te gebruiken of openbaar te maken. The information contained in this e-mail, including possible attachments, is confidential and is solely for the use of the intended recipient(s). Should you have received this e-mail unintentionally you are then requested to inform the sender and to destroy the message.It is prohibited to use or disclose the information this message contains in whatsoever way. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Efficiently pin running VMs to physical CPUs automatically
On 06/20/2013 10:18 PM, Russell Bryant wrote: > On 06/20/2013 10:36 AM, Giorgio Franceschi wrote: >> Hello, I created a blueprint for the implementation of: >> >> A tool for pinning automatically each running virtual CPU to a physical >> one in the most efficient way, balancing load across sockets/cores and >> maximizing cache sharing/minimizing cache misses. Ideally able to be run >> on-demand, as a periodic job, or be triggered by events on the host (vm >> spawn/destroy). >> >> Find it at https://blueprints.launchpad.net/nova/+spec/auto-cpu-pinning >> >> Any inputappreciated! > > I'm actually surprised to see a new tool for this kind of thing. > > Have you seen numad? And a related post by Dan Berrange (but on lower layers -- libvirt) which explains how to do vcpu pinning and control NUMA affinity -- http://berrange.com/posts/2010/02/12/controlling-guest-cpu-numa-affinity-in-libvirt-with-qemu-kvm-xen/ -- /kashyap ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Efficiently pin running VMs to physical CPUs automatically
On 06/20/2013 10:36 AM, Giorgio Franceschi wrote: > Hello, I created a blueprint for the implementation of: > > A tool for pinning automatically each running virtual CPU to a physical > one in the most efficient way, balancing load across sockets/cores and > maximizing cache sharing/minimizing cache misses. Ideally able to be run > on-demand, as a periodic job, or be triggered by events on the host (vm > spawn/destroy). > > Find it at https://blueprints.launchpad.net/nova/+spec/auto-cpu-pinning > > Any inputappreciated! I'm actually surprised to see a new tool for this kind of thing. Have you seen numad? -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Efficiently pin running VMs to physical CPUs automatically
Hello, I created a blueprint for the implementation of: A tool for pinning automatically each running virtual CPU to a physical one in the most efficient way, balancing load across sockets/cores and maximizing cache sharing/minimizing cache misses. Ideally able to be run on-demand, as a periodic job, or be triggered by events on the host (vm spawn/destroy). Find it at https://blueprints.launchpad.net/nova/+spec/auto-cpu-pinning Any inputappreciated! Thanks -- *Giorgio Franceschi* System Engineer *TMG Online Media/Hyves* Frederiksplein, 42. 1017 XN Amsterdam De informatie in dit e-mailbericht en eventuele bijlagen is vertrouwelijk en is alleen bestemd voor de beoogde ontvanger(s). Indien u dit bericht ten onrechte heeft ontvangen, wordt u verzocht de verzender daarvan in kennis te stellen en het bericht te vernietigen. Het is niet toegestaan de hierin opgenomen informatie op welke wijze dan ook te gebruiken of openbaar te maken. The information contained in this e-mail, including possible attachments, is confidential and is solely for the use of the intended recipient(s). Should you have received this e-mail unintentionally you are then requested to inform the sender and to destroy the message.It is prohibited to use or disclose the information this message contains in whatsoever way. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
