Public bug reported:
Observed on Train.
4 units of Keystone deployed, each backed by shared mysql, with individual
memcached local daemon for caching.
The user is assigned the role "member" in the project, validating new token
returns proper assignment from every keystone unit:
[
{
"id": "758f4dc35c7b4377bfcba1ae083c2808",
"name": "member"
},
{
"id": "d1ea6664378645a788f44b8a1a44e874",
"name": "reader"
}
]
Adding the role of "admin" within the same project, request succeeded,
database is adjusted, query of new roles list in the project for the
user works fine.
User issues new token, validation returns new role in the project only
from one unit of keystone, other 3 still show old list. Eventually
(around 10 minutes timeframe) they all pick up.
$ for i in 43 52 53 60; do curl --insecure -i -H "X-Auth-Token: <redacted>" -H
"X-Subject-Token: <redacted>" https://172.31.248.$i:35347/v3/auth/tokens |grep
\{ |jq .token.roles; done
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 12433 100 12433 0 0 311k 0 --:--:-- --:--:-- --:--:-- 311k
[
{
"id": "a2565b5e5bb440ac9a964f726ccd2f26",
"name": "admin"
},
{
"id": "758f4dc35c7b4377bfcba1ae083c2808",
"name": "member"
},
{
"id": "d1ea6664378645a788f44b8a1a44e874",
"name": "reader"
}
]
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 12372 100 12372 0 0 294k 0 --:--:-- --:--:-- --:--:-- 294k
[
{
"id": "d1ea6664378645a788f44b8a1a44e874",
"name": "reader"
},
{
"id": "758f4dc35c7b4377bfcba1ae083c2808",
"name": "member"
}
]
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 12372 100 12372 0 0 280k 0 --:--:-- --:--:-- --:--:-- 287k
[
{
"id": "758f4dc35c7b4377bfcba1ae083c2808",
"name": "member"
},
{
"id": "d1ea6664378645a788f44b8a1a44e874",
"name": "reader"
}
]
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 12372 100 12372 0 0 246k 0 --:--:-- --:--:-- --:--:-- 246k
[
{
"id": "d1ea6664378645a788f44b8a1a44e874",
"name": "reader"
},
{
"id": "758f4dc35c7b4377bfcba1ae083c2808",
"name": "member"
}
]
The root cause is the "expiration_time" in [cache] section to be set to 600
seconds. During this time the units of keystone which were not yet issuing the
token for the user with new assignments are returning old information.
** Affects: keystone
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1899115
Title:
cached tokens prevent to validate modified assignments
Status in OpenStack Identity (keystone):
New
Bug description:
Observed on Train.
4 units of Keystone deployed, each backed by shared mysql, with individual
memcached local daemon for caching.
The user is assigned the role "member" in the project, validating new token
returns proper assignment from every keystone unit:
[
{
"id": "758f4dc35c7b4377bfcba1ae083c2808",
"name": "member"
},
{
"id": "d1ea6664378645a788f44b8a1a44e874",
"name": "reader"
}
]
Adding the role of "admin" within the same project, request succeeded,
database is adjusted, query of new roles list in the project for the
user works fine.
User issues new token, validation returns new role in the project only
from one unit of keystone, other 3 still show old list. Eventually
(around 10 minutes timeframe) they all pick up.
$ for i in 43 52 53 60; do curl --insecure -i -H "X-Auth-Token: <redacted>"
-H "X-Subject-Token: <redacted>" https://172.31.248.$i:35347/v3/auth/tokens
|grep \{ |jq .token.roles; done
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left Speed
100 12433 100 12433 0 0 311k 0 --:--:-- --:--:-- --:--:-- 311k
[
{
"id": "a2565b5e5bb440ac9a964f726ccd2f26",
"name": "admin"
},
{
"id": "758f4dc35c7b4377bfcba1ae083c2808",
"name": "member"
},
{
"id": "d1ea6664378645a788f44b8a1a44e874",
"name": "reader"
}
]
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left Speed
100 12372 100 12372 0 0 294k 0 --:--:-- --:--:-- --:--:-- 294k
[
{
"id": "d1ea6664378645a788f44b8a1a44e874",
"name": "reader"
},
{
"id": "758f4dc35c7b4377bfcba1ae083c2808",
"name": "member"
}
]
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left Speed
100 12372 100 12372 0 0 280k 0 --:--:-- --:--:-- --:--:-- 287k
[
{
"id": "758f4dc35c7b4377bfcba1ae083c2808",
"name": "member"
},
{
"id": "d1ea6664378645a788f44b8a1a44e874",
"name": "reader"
}
]
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left Speed
100 12372 100 12372 0 0 246k 0 --:--:-- --:--:-- --:--:-- 246k
[
{
"id": "d1ea6664378645a788f44b8a1a44e874",
"name": "reader"
},
{
"id": "758f4dc35c7b4377bfcba1ae083c2808",
"name": "member"
}
]
The root cause is the "expiration_time" in [cache] section to be set to 600
seconds. During this time the units of keystone which were not yet issuing the
token for the user with new assignments are returning old information.
To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1899115/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
