Re: [Openstack-operators] over commit ratios
In addition to these factors, collocation happens to be another key source of noise. By collocation I mean VMs doing the same/similar work running on the same hypervisor. This happens under low capacity situations when the scheduler could not enforce anti-affinity. Subbu On Apr 22, 2015, at 5:09 AM, George Shuklin george.shuk...@gmail.com wrote: Yes, it really depends on the used backing technique. We using SSDs and raw images, so IO is not an issue. But memory is more important: if you lack IO capability you left with slow guests. If you lack memory you left with dead guests (hello, OOM killer). BTW: Swap is needed not to swapin/swapout, but to relief memory pressure. With properly configured memory swin/swout should be less than 2-3. On 04/22/2015 09:49 AM, Tim Bell wrote: I'd also keep an eye on local I/O... we've found this to be the resource which can cause the worst noisy neighbours. Swapping makes this worse. Tim -Original Message- From: George Shuklin [mailto:george.shuk...@gmail.com] Sent: 21 April 2015 23:55 To: openstack-operators@lists.openstack.org Subject: Re: [Openstack-operators] over commit ratios It's very depend on production type. If you can control guests and predict their memory consumption, use it as base for ratio. If you can't (typical for public clouds) - use 1 or smaller with reserved_host_memory_mb in nova.conf. And one more: some swap sapce is really necessary. Add at least twice of reserved_host_memory_mb - it really improves performance and prevents strange OOMs in the situation of very large host with very small dom0 footprint. On 04/21/2015 10:59 PM, Caius Howcroft wrote: Just a general question: what kind of over commit ratios do people normally run in production with? We currently run 2 for cpu and 1 for memory (with some held back for OS/ceph) i.e.: default['bcpc']['nova']['ram_allocation_ratio'] = 1.0 default['bcpc']['nova']['reserved_host_memory_mb'] = 1024 # often larger default['bcpc']['nova']['cpu_allocation_ratio'] = 2.0 Caius ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] over commit ratios
A juno feature may help with this, Utilization based scheduling: https://blueprints.launchpad.net/nova/+spec/utilization-aware-scheduling That helps when landing the instance, but doesn't help if utilization changes /after/ instances have landed, but could help with a resize action to relocate the instance. - jlk On Wed, Apr 22, 2015 at 8:44 AM, Allamaraju, Subbu su...@subbu.org wrote: In addition to these factors, collocation happens to be another key source of noise. By collocation I mean VMs doing the same/similar work running on the same hypervisor. This happens under low capacity situations when the scheduler could not enforce anti-affinity. Subbu On Apr 22, 2015, at 5:09 AM, George Shuklin george.shuk...@gmail.com wrote: Yes, it really depends on the used backing technique. We using SSDs and raw images, so IO is not an issue. But memory is more important: if you lack IO capability you left with slow guests. If you lack memory you left with dead guests (hello, OOM killer). BTW: Swap is needed not to swapin/swapout, but to relief memory pressure. With properly configured memory swin/swout should be less than 2-3. On 04/22/2015 09:49 AM, Tim Bell wrote: I'd also keep an eye on local I/O... we've found this to be the resource which can cause the worst noisy neighbours. Swapping makes this worse. Tim -Original Message- From: George Shuklin [mailto:george.shuk...@gmail.com] Sent: 21 April 2015 23:55 To: openstack-operators@lists.openstack.org Subject: Re: [Openstack-operators] over commit ratios It's very depend on production type. If you can control guests and predict their memory consumption, use it as base for ratio. If you can't (typical for public clouds) - use 1 or smaller with reserved_host_memory_mb in nova.conf. And one more: some swap sapce is really necessary. Add at least twice of reserved_host_memory_mb - it really improves performance and prevents strange OOMs in the situation of very large host with very small dom0 footprint. On 04/21/2015 10:59 PM, Caius Howcroft wrote: Just a general question: what kind of over commit ratios do people normally run in production with? We currently run 2 for cpu and 1 for memory (with some held back for OS/ceph) i.e.: default['bcpc']['nova']['ram_allocation_ratio'] = 1.0 default['bcpc']['nova']['reserved_host_memory_mb'] = 1024 # often larger default['bcpc']['nova']['cpu_allocation_ratio'] = 2.0 Caius ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] over commit ratios
I'd also keep an eye on local I/O... we've found this to be the resource which can cause the worst noisy neighbours. Swapping makes this worse. Tim -Original Message- From: George Shuklin [mailto:george.shuk...@gmail.com] Sent: 21 April 2015 23:55 To: openstack-operators@lists.openstack.org Subject: Re: [Openstack-operators] over commit ratios It's very depend on production type. If you can control guests and predict their memory consumption, use it as base for ratio. If you can't (typical for public clouds) - use 1 or smaller with reserved_host_memory_mb in nova.conf. And one more: some swap sapce is really necessary. Add at least twice of reserved_host_memory_mb - it really improves performance and prevents strange OOMs in the situation of very large host with very small dom0 footprint. On 04/21/2015 10:59 PM, Caius Howcroft wrote: Just a general question: what kind of over commit ratios do people normally run in production with? We currently run 2 for cpu and 1 for memory (with some held back for OS/ceph) i.e.: default['bcpc']['nova']['ram_allocation_ratio'] = 1.0 default['bcpc']['nova']['reserved_host_memory_mb'] = 1024 # often larger default['bcpc']['nova']['cpu_allocation_ratio'] = 2.0 Caius ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] over commit ratios
Yes, it really depends on the used backing technique. We using SSDs and raw images, so IO is not an issue. But memory is more important: if you lack IO capability you left with slow guests. If you lack memory you left with dead guests (hello, OOM killer). BTW: Swap is needed not to swapin/swapout, but to relief memory pressure. With properly configured memory swin/swout should be less than 2-3. On 04/22/2015 09:49 AM, Tim Bell wrote: I'd also keep an eye on local I/O... we've found this to be the resource which can cause the worst noisy neighbours. Swapping makes this worse. Tim -Original Message- From: George Shuklin [mailto:george.shuk...@gmail.com] Sent: 21 April 2015 23:55 To: openstack-operators@lists.openstack.org Subject: Re: [Openstack-operators] over commit ratios It's very depend on production type. If you can control guests and predict their memory consumption, use it as base for ratio. If you can't (typical for public clouds) - use 1 or smaller with reserved_host_memory_mb in nova.conf. And one more: some swap sapce is really necessary. Add at least twice of reserved_host_memory_mb - it really improves performance and prevents strange OOMs in the situation of very large host with very small dom0 footprint. On 04/21/2015 10:59 PM, Caius Howcroft wrote: Just a general question: what kind of over commit ratios do people normally run in production with? We currently run 2 for cpu and 1 for memory (with some held back for OS/ceph) i.e.: default['bcpc']['nova']['ram_allocation_ratio'] = 1.0 default['bcpc']['nova']['reserved_host_memory_mb'] = 1024 # often larger default['bcpc']['nova']['cpu_allocation_ratio'] = 2.0 Caius ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] over commit ratios
We had this discussion at the Ops Mid-Cycle meetup. I think the general consensus was 0.9 for memory. if you're running Ceph OSD's on the node you'll almost certainly want to reserve more than a gig for it and the OS. for CPU there was a wide range of ideas and it mainly depended on use-case. If you're running lots of CPU-intensive things like Hadoop, better to be around 2 or even 1. If you've got a ton of web servers, you could go with 16 easily. If you've got a mixed-use cloud, some segregation with host aggregates may be helpful so you can vary the number. If you can't do that though, and you're mixed, you should be able to go better than 2. At least 5 should be OK unless you've packed a massive amount of memory in each node. -Erik On Tue, Apr 21, 2015 at 3:59 PM, Caius Howcroft caius.howcr...@gmail.com wrote: Just a general question: what kind of over commit ratios do people normally run in production with? We currently run 2 for cpu and 1 for memory (with some held back for OS/ceph) i.e.: default['bcpc']['nova']['ram_allocation_ratio'] = 1.0 default['bcpc']['nova']['reserved_host_memory_mb'] = 1024 # often larger default['bcpc']['nova']['cpu_allocation_ratio'] = 2.0 Caius -- Caius Howcroft @caiushowcroft http://www.linkedin.com/in/caius ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
[Openstack-operators] over commit ratios
Just a general question: what kind of over commit ratios do people normally run in production with? We currently run 2 for cpu and 1 for memory (with some held back for OS/ceph) i.e.: default['bcpc']['nova']['ram_allocation_ratio'] = 1.0 default['bcpc']['nova']['reserved_host_memory_mb'] = 1024 # often larger default['bcpc']['nova']['cpu_allocation_ratio'] = 2.0 Caius -- Caius Howcroft @caiushowcroft http://www.linkedin.com/in/caius ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
