Re: [openstack-dev] [puppet][tripleo][fuel] Upcoming changes to defaults around using processor count for worker configurations

2016-10-20 Thread Alex Schultz 
Hey Sergii,

On Thu, Oct 20, 2016 at 3:34 AM, Sergii Golovatiuk
 wrote:
> Hi,
>
>
> On Thu, Sep 29, 2016 at 11:57 PM, Alex Schultz  wrote:
>>
>> Hello all,
>>
>> So for many years we've been using either the service defaults
>> (usually python determined processor count) or the $processorcount
>> fact from facter in puppet for worker configuration options for the
>> OpenStack services.  If you are currently using the default values
>> provided by the puppet modules, you will be affected by this upcoming
>> change. After much discussion and feedback from deployers, we've
>> decided to change this to a default value that has a cap on it.  This
>> is primarily from the feedback when deploying on physical hardware
>> where processor counts can be 32, 48 or even 512.  These values can
>> lead to excessive memory consumption or errors due to connection
>> limits (mysql/rabbit).  As such we've come up with a new fact to that
>> will be used instead of $processorcount.
>>
>> The new fact is called $os_workers[0]. This fact uses the
>> $processorcount to help weigh in on the number of workers to configure
>> and won't be less than 2 but is capped at 8.  The $os_workers fact
>> will use the larger value of either '2' or '# of processors / 4' but
>> will not exceed 8.  The primary goal of this is to improve the user
>> experience when people install services using the puppet modules and
>> without having to tune all of these worker values.  We plan on
>> implementing this for all modules as part of the Ocata cycle.  This
>> work will can be tracked using the os_workers-fact[1] gerrit topic.
>> It should be noted that we have implemented this fact in such a way
>> that operators are free to override it using an external fact to
>> provide their own values as well.  If you are currently specifying
>> your own values for the worker configurations in your manifests then
>> this change will not affect you.  If you have been relying on the
>> defaults and wish to continue to use the $processorcount logic, we
>> would recommend either implementing your own external fact[2] for this
>> or updating your manifests to provide $::processorcount to the workers
>> configuration.
>
>
> This doesn't help a lot. I saw the case where 8 neutron-servers allocated
> 6GB of RAM. From OOM perspective, the biggest process was MySQL (or Rabbit)
> as it doesn't calculate the sum of processes. Instead of killing
> neutron-server it killed MySQL to release some RAM to node. IMO, I would
> focus on cgroup limitation for OpenStack services as that will allow the
> operator to specify some the upper limit of CPU and RAM usage for every
> service.
>

Yea this isn't a fix for excessive ram utilization of the services
themselves. This primarily reduces the default consumption. I recently
went to implement using this fact in tripleo[0] and it reduced the
default memory utilization of the undercloud by a little over 1gig[1]
in our CI runs.  This was on a 4 cpu vhost so we basically halfed the
number of running python processes by switching for the 17 different
services that may run.  So yea this isn't the end solution for dealing
with excessive memory consumption but rather an improvement over the
existing processor count usage.  The primary target for this is really
to prevent accidental blowing up of mysql connections when a host has
large number of cpus but also reducing the overall number of processes
by default.

Thanks,
-Alex

[0] https://review.openstack.org/#/c/386696/
[1] http://people.redhat.com/~aschultz/os_workers.html

>>
>> As always we'd love to hear feedback on this and any other issues
>> people might be facing. We're always available in #puppet-openstack on
>> freenode or via the mailing lists.
>>
>> Thanks,
>> -Alex
>>
>>
>> [0] https://review.openstack.org/#/c/375146/
>> [1] https://review.openstack.org/#/q/topic:os_workers-fact
>> [2] https://docs.puppet.com/facter/3.4/custom_facts.html#external-facts
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet][tripleo][fuel] Upcoming changes to defaults around using processor count for worker configurations

2016-10-20 Thread Sergii Golovatiuk 
Hi,


On Thu, Sep 29, 2016 at 11:57 PM, Alex Schultz  wrote:

> Hello all,
>
> So for many years we've been using either the service defaults
> (usually python determined processor count) or the $processorcount
> fact from facter in puppet for worker configuration options for the
> OpenStack services.  If you are currently using the default values
> provided by the puppet modules, you will be affected by this upcoming
> change. After much discussion and feedback from deployers, we've
> decided to change this to a default value that has a cap on it.  This
> is primarily from the feedback when deploying on physical hardware
> where processor counts can be 32, 48 or even 512.  These values can
> lead to excessive memory consumption or errors due to connection
> limits (mysql/rabbit).  As such we've come up with a new fact to that
> will be used instead of $processorcount.
>
> The new fact is called $os_workers[0]. This fact uses the
> $processorcount to help weigh in on the number of workers to configure
> and won't be less than 2 but is capped at 8.  The $os_workers fact
> will use the larger value of either '2' or '# of processors / 4' but
> will not exceed 8.  The primary goal of this is to improve the user
> experience when people install services using the puppet modules and
> without having to tune all of these worker values.  We plan on
> implementing this for all modules as part of the Ocata cycle.  This
> work will can be tracked using the os_workers-fact[1] gerrit topic.
> It should be noted that we have implemented this fact in such a way
> that operators are free to override it using an external fact to
> provide their own values as well.  If you are currently specifying
> your own values for the worker configurations in your manifests then
> this change will not affect you.  If you have been relying on the
> defaults and wish to continue to use the $processorcount logic, we
> would recommend either implementing your own external fact[2] for this
> or updating your manifests to provide $::processorcount to the workers
> configuration.
>

This doesn't help a lot. I saw the case where 8 neutron-servers allocated
6GB of RAM. From OOM perspective, the biggest process was MySQL (or Rabbit)
as it doesn't calculate the sum of processes. Instead of killing
neutron-server it killed MySQL to release some RAM to node. IMO, I would
focus on cgroup limitation for OpenStack services as that will allow the
operator to specify some the upper limit of CPU and RAM usage for every
service.


> As always we'd love to hear feedback on this and any other issues
> people might be facing. We're always available in #puppet-openstack on
> freenode or via the mailing lists.
>
> Thanks,
> -Alex
>
>
> [0] https://review.openstack.org/#/c/375146/
> [1] https://review.openstack.org/#/q/topic:os_workers-fact
> [2] https://docs.puppet.com/facter/3.4/custom_facts.html#external-facts
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [puppet][tripleo][fuel] Upcoming changes to defaults around using processor count for worker configurations

2016-09-29 Thread Alex Schultz 
Hello all,

So for many years we've been using either the service defaults
(usually python determined processor count) or the $processorcount
fact from facter in puppet for worker configuration options for the
OpenStack services.  If you are currently using the default values
provided by the puppet modules, you will be affected by this upcoming
change. After much discussion and feedback from deployers, we've
decided to change this to a default value that has a cap on it.  This
is primarily from the feedback when deploying on physical hardware
where processor counts can be 32, 48 or even 512.  These values can
lead to excessive memory consumption or errors due to connection
limits (mysql/rabbit).  As such we've come up with a new fact to that
will be used instead of $processorcount.

The new fact is called $os_workers[0]. This fact uses the
$processorcount to help weigh in on the number of workers to configure
and won't be less than 2 but is capped at 8.  The $os_workers fact
will use the larger value of either '2' or '# of processors / 4' but
will not exceed 8.  The primary goal of this is to improve the user
experience when people install services using the puppet modules and
without having to tune all of these worker values.  We plan on
implementing this for all modules as part of the Ocata cycle.  This
work will can be tracked using the os_workers-fact[1] gerrit topic.
It should be noted that we have implemented this fact in such a way
that operators are free to override it using an external fact to
provide their own values as well.  If you are currently specifying
your own values for the worker configurations in your manifests then
this change will not affect you.  If you have been relying on the
defaults and wish to continue to use the $processorcount logic, we
would recommend either implementing your own external fact[2] for this
or updating your manifests to provide $::processorcount to the workers
configuration.

As always we'd love to hear feedback on this and any other issues
people might be facing. We're always available in #puppet-openstack on
freenode or via the mailing lists.

Thanks,
-Alex


[0] https://review.openstack.org/#/c/375146/
[1] https://review.openstack.org/#/q/topic:os_workers-fact
[2] https://docs.puppet.com/facter/3.4/custom_facts.html#external-facts

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev