Public bug reported:
With memcache set up for resource caching, when a tenant is created,
deleted, and recreated with the same name, users within that project get
intermittent errors when requesting tokens.
You can recreate this by having memcache with resource caching enabled.
Then create a tenant, delete it, and then recreate it making sure the
name is the same as the first one. Then create a user in this tenant
and continually request tokens. It will gradually start generating
tokens while also failing until the cache is cleaned out.
I believe the intermittent errors we experienced were due to our
environment having a memcache on each keystone node and having the
keystone nodes behind a load balancer.
As I ran this scenario, I was seeing more failures in the beginning and
then it gradually started having more successes until a little after the
cache expiration_time where I was seeing all successes.
We investigated and when this error was originally hit it threw 404 or
401s. The 404s were complaining about not being able to find a certain
project, but when I tried to recreate I was receiving all 401s.
The 404 errors led me to believe that this was due to memcache not
marking cache entries as deleted. Since, when running our tests we used
the name of the project and it would auto resolve the id. So the entry
for the project name in the cache was conflicting with the entry in the
database, but once the cache is expired it isn't an issue.
So it seems that reusing names of projects causes problems with the
resolution of the project id when memcache is enabled.
** Affects: keystone
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Keystone.
https://bugs.launchpad.net/bugs/1504686
Title:
Keystone errors on token requests for users in recreated tenants when
using memcache
Status in Keystone:
New
Bug description:
With memcache set up for resource caching, when a tenant is created,
deleted, and recreated with the same name, users within that project
get intermittent errors when requesting tokens.
You can recreate this by having memcache with resource caching
enabled. Then create a tenant, delete it, and then recreate it making
sure the name is the same as the first one. Then create a user in
this tenant and continually request tokens. It will gradually start
generating tokens while also failing until the cache is cleaned out.
I believe the intermittent errors we experienced were due to our
environment having a memcache on each keystone node and having the
keystone nodes behind a load balancer.
As I ran this scenario, I was seeing more failures in the beginning
and then it gradually started having more successes until a little
after the cache expiration_time where I was seeing all successes.
We investigated and when this error was originally hit it threw 404 or
401s. The 404s were complaining about not being able to find a
certain project, but when I tried to recreate I was receiving all
401s.
The 404 errors led me to believe that this was due to memcache not
marking cache entries as deleted. Since, when running our tests we
used the name of the project and it would auto resolve the id. So the
entry for the project name in the cache was conflicting with the entry
in the database, but once the cache is expired it isn't an issue.
So it seems that reusing names of projects causes problems with the
resolution of the project id when memcache is enabled.
To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1504686/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
