Re: [Openstack-operators] neutron-server memcached connections
Great, yeah we have also seen these issues with nova-api with keystonemiddle in
newton and ocata.
Thanks for the heads up as I was going to start digging deeper.
Cheers,
Sam
> On 31 Jul 2018, at 10:09 am, iain MacDonnell
> wrote:
>
>
> Following up on my own question, in case it's useful to others
>
> Turns out that keystonemiddleware uses eventlet, and, by default, creates a
> connection to memcached from each green thread (and doesn't clean them up),
> and the green threads are essentially unlimited.
>
> There is a solution for this, which implements a shared connection pool. It's
> enabled via the keystone_authtoken.memcache_use_advanced_pool config option.
>
> Unfortunately it was broken in a few different ways (I guess this means that
> no one is using it?)
>
> I've worked with the keystone devs, and we were able to get a fix (in
> keystonemiddleware) in just in time for the Rocky release. Related fixes have
> also been backported to Queens (for the next update), and a couple needed for
> Pike are pending completion.
>
> With this in place, so-far I have not seen more than one connection to
> memcached for each neutron-api worker process, and everything seems to be
> working well.
>
> Some relevant changes:
>
> master:
>
> https://review.openstack.org/#/c/583695/
>
>
> Queens:
>
> https://review.openstack.org/#/c/583698/
> https://review.openstack.org/#/c/583684/
>
>
> Pike:
>
> https://review.openstack.org/#/c/583699/
> https://review.openstack.org/#/c/583835/
>
>
> I do wonder how others are managing memcached connections for larger
> deployments...
>
>~iain
>
>
>
> On 06/26/2018 12:59 PM, iain MacDonnell wrote:
>> In diagnosing a situation where a Pike deployment was intermittently slower
>> (in general), I discovered that it was (sometimes) exceeding memcached's
>> maximum connection limit, which is set to 4096.
>> Looking closer, ~2750 of the connections are from 8 neutron-server process.
>> neutron-server is configured with 8 API workers, and those 8 processes have
>> a combined total of ~2750 connections to memcached:
>> # lsof -i TCP:11211 | awk '/^neutron-s/ {print $2}' | sort | uniq -c
>> 245 2611
>> 306 2612
>> 228 2613
>> 406 2614
>> 407 2615
>> 385 2616
>> 369 2617
>> 398 2618
>> #
>> There doesn't seem to be much turnover - comparing samples of the
>> connections (incl. source port) 15 mins apart, two were dropped, and one new
>> one added.
>> In neutron.conf, keystone_authtoken.memcached_servers is configured, but
>> nothing else pertaining to caching, so
>> keystone_authtoken.memcache_pool_maxsize should default to 10.
>> Am I misunderstanding something, or shouldn't I see a maximum of 10
>> connections from each of the neutron-server API workers, with this
>> configuration?
>> Any known issues, or pointers to what I'm missing?
>> TIA,
>> ~iain
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] neutron-server memcached connections
Following up on my own question, in case it's useful to others
Turns out that keystonemiddleware uses eventlet, and, by default,
creates a connection to memcached from each green thread (and doesn't
clean them up), and the green threads are essentially unlimited.
There is a solution for this, which implements a shared connection pool.
It's enabled via the keystone_authtoken.memcache_use_advanced_pool
config option.
Unfortunately it was broken in a few different ways (I guess this means
that no one is using it?)
I've worked with the keystone devs, and we were able to get a fix (in
keystonemiddleware) in just in time for the Rocky release. Related fixes
have also been backported to Queens (for the next update), and a couple
needed for Pike are pending completion.
With this in place, so-far I have not seen more than one connection to
memcached for each neutron-api worker process, and everything seems to
be working well.
Some relevant changes:
master:
https://review.openstack.org/#/c/583695/
Queens:
https://review.openstack.org/#/c/583698/
https://review.openstack.org/#/c/583684/
Pike:
https://review.openstack.org/#/c/583699/
https://review.openstack.org/#/c/583835/
I do wonder how others are managing memcached connections for larger
deployments...
~iain
On 06/26/2018 12:59 PM, iain MacDonnell wrote:
In diagnosing a situation where a Pike deployment was intermittently
slower (in general), I discovered that it was (sometimes) exceeding
memcached's maximum connection limit, which is set to 4096.
Looking closer, ~2750 of the connections are from 8 neutron-server
process. neutron-server is configured with 8 API workers, and those 8
processes have a combined total of ~2750 connections to memcached:
# lsof -i TCP:11211 | awk '/^neutron-s/ {print $2}' | sort | uniq -c
245 2611
306 2612
228 2613
406 2614
407 2615
385 2616
369 2617
398 2618
#
There doesn't seem to be much turnover - comparing samples of the
connections (incl. source port) 15 mins apart, two were dropped, and one
new one added.
In neutron.conf, keystone_authtoken.memcached_servers is configured, but
nothing else pertaining to caching, so
keystone_authtoken.memcache_pool_maxsize should default to 10.
Am I misunderstanding something, or shouldn't I see a maximum of 10
connections from each of the neutron-server API workers, with this
configuration?
Any known issues, or pointers to what I'm missing?
TIA,
~iain
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
[Openstack-operators] neutron-server memcached connections
In diagnosing a situation where a Pike deployment was intermittently
slower (in general), I discovered that it was (sometimes) exceeding
memcached's maximum connection limit, which is set to 4096.
Looking closer, ~2750 of the connections are from 8 neutron-server
process. neutron-server is configured with 8 API workers, and those 8
processes have a combined total of ~2750 connections to memcached:
# lsof -i TCP:11211 | awk '/^neutron-s/ {print $2}' | sort | uniq -c
245 2611
306 2612
228 2613
406 2614
407 2615
385 2616
369 2617
398 2618
#
There doesn't seem to be much turnover - comparing samples of the
connections (incl. source port) 15 mins apart, two were dropped, and one
new one added.
In neutron.conf, keystone_authtoken.memcached_servers is configured, but
nothing else pertaining to caching, so
keystone_authtoken.memcache_pool_maxsize should default to 10.
Am I misunderstanding something, or shouldn't I see a maximum of 10
connections from each of the neutron-server API workers, with this
configuration?
Any known issues, or pointers to what I'm missing?
TIA,
~iain
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
