Public bug reported: NOTE - this is happening only on Queens, and possibly earlier, and is already silently fixed in Rocky as part of the major refactor of token model. I am posing this issue here so that anyone still running Queens has a reference and probably a patch to apply.
In Queens release, if I create a token using a nocatalog option: curl -X POST <keystone-endpoint>/v3/auth/tokens?nocatalog and then use this token to e.g. list servers with details curl -X GET <nova-endpoint>/v2.1/servers/detail I get 500 error from nova, with nova api logs containing ERROR nova.api.openstack EmptyCatalog: The service catalog is empty. When repeating the same request with the same token after 5-10 minutes, the token starts to work. Tokens generated with catalog are working as well. AFAIU this goes down to token caching - in Queens tokens are cached/memoized with catalog - or without it if token was requested w/o catalog. Then on token validation, the token validation response takes the token - with or without catalog - from cache and returns it to caller with minimal processing - e.g. removing catalog if token validation call asked for it. It does not however ensures that the catalog is present otherwise. This breaks some other services like Nova, that expects the catalog to be present in the request context constructed from the keystonemiddleware results. Nova needs this for example to make API requests to other services - exactly what happens in server/details call where it has to ask Neutron for some network info about instances. After the cache is invalidated, the catalog starts to be generated for token validation response anew, and everything starts to work as expected. ** Affects: keystone Importance: Undecided Assignee: Pavlo Shchelokovskyy (pshchelo) Status: In Progress ** Changed in: keystone Status: New => In Progress ** Changed in: keystone Assignee: (unassigned) => Pavlo Shchelokovskyy (pshchelo) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1938323 Title: [Queens] tokens generated with nocatalog are not usable in some requests Status in OpenStack Identity (keystone): In Progress Bug description: NOTE - this is happening only on Queens, and possibly earlier, and is already silently fixed in Rocky as part of the major refactor of token model. I am posing this issue here so that anyone still running Queens has a reference and probably a patch to apply. In Queens release, if I create a token using a nocatalog option: curl -X POST <keystone-endpoint>/v3/auth/tokens?nocatalog and then use this token to e.g. list servers with details curl -X GET <nova-endpoint>/v2.1/servers/detail I get 500 error from nova, with nova api logs containing ERROR nova.api.openstack EmptyCatalog: The service catalog is empty. When repeating the same request with the same token after 5-10 minutes, the token starts to work. Tokens generated with catalog are working as well. AFAIU this goes down to token caching - in Queens tokens are cached/memoized with catalog - or without it if token was requested w/o catalog. Then on token validation, the token validation response takes the token - with or without catalog - from cache and returns it to caller with minimal processing - e.g. removing catalog if token validation call asked for it. It does not however ensures that the catalog is present otherwise. This breaks some other services like Nova, that expects the catalog to be present in the request context constructed from the keystonemiddleware results. Nova needs this for example to make API requests to other services - exactly what happens in server/details call where it has to ask Neutron for some network info about instances. After the cache is invalidated, the catalog starts to be generated for token validation response anew, and everything starts to work as expected. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1938323/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp