Public bug reported: Test fails as:- 2024-08-07 17:06:05.650302 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients [-] Unable to authenticate for user c_rally_927546a8_h6aDLbnK in project c_rally_927546a8_ahUUmlp9: keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500) 2024-08-07 17:06:05.650941 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients Traceback (most recent call last): 2024-08-07 17:06:05.651113 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/rally-openstack/rally_openstack/common/osclients.py", line 269, in auth_ref 2024-08-07 17:06:05.651156 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients self.cache["keystone_auth_ref"] = plugin.get_access(sess) 2024-08-07 17:06:05.651193 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/base.py", line 131, in get_access 2024-08-07 17:06:05.651229 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients self.auth_ref = self.get_auth_ref(session) 2024-08-07 17:06:05.651263 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/generic/base.py", line 205, in get_auth_ref 2024-08-07 17:06:05.651334 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients return self._plugin.get_auth_ref(session, **kwargs) 2024-08-07 17:06:05.651348 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/v3/base.py", line 185, in get_auth_ref 2024-08-07 17:06:05.651356 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients resp = session.post(token_url, json=body, headers=headers, 2024-08-07 17:06:05.651363 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/session.py", line 1162, in post 2024-08-07 17:06:05.651370 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients return self.request(url, 'POST', **kwargs) 2024-08-07 17:06:05.651377 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/session.py", line 985, in request 2024-08-07 17:06:05.651384 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients raise exceptions.from_response(resp, method, url) 2024-08-07 17:06:05.651391 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500)
>From keyston log:- Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: CRITICAL keystone [None req-96647e8e-2585-4279-80fa-c4fa97b8c455 None None] Unhandled error: ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone Traceback (most recent call last): Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 1498, in __call__ Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.wsgi_app(environ, start_response) Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/werkzeug/middleware/proxy_fix.py", line 183, in __call__ Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.app(environ, start_response) Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__ Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs) Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/base.py", line 124, in __call__ Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = req.get_response(self.application) Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application( Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response) Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 143, in __call__ Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return resp(environ, start_response) Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__ Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs) Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/base.py", line 124, in __call__ Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = req.get_response(self.application) Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application( Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response) Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__ Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs) Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/web.py", line 131, in __call__ Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone return request.get_response(self.application) Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application( Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response) Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__ Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs) Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/request_id.py", line 58, in __call__ Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = req.get_response(self.application) Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application( Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response) Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/server/flask/request_processing/middleware/url_normalize.py", line 38, in __call__ Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.app(environ, start_response) Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__ Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs) Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/keystonemiddleware/auth_token/__init__.py", line 340, in __call__ Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = req.get_response(self._app) Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application( Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response) Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/werkzeug/middleware/dispatcher.py", line 81, in __call__ Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return app(environ, start_response) Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 1476, in wsgi_app Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = self.handle_exception(e) Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e) Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e) Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e) Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone [Previous line repeated 28 more times] Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 1473, in wsgi_app Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = self.full_dispatch_request() Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 882, in full_dispatch_request Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone rv = self.handle_user_exception(e) Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e) Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e) Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e) Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone [Previous line repeated 28 more times] Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 880, in full_dispatch_request Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone rv = self.dispatch_request() Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 865, in dispatch_request Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return] Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 489, in wrapper Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = resource(*args, **kwargs) Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/views.py", line 110, in view Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return current_app.ensure_sync(self.dispatch_request)(**kwargs) # type: ignore[no-any-return] Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 604, in dispatch_request Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = meth(*args, **kwargs) Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/server/flask/common.py", line 1146, in wrapper Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return f(*args, **kwargs) Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/api/auth.py", line 328, in post Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone token = authentication.authenticate_for_token(auth_data) Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/api/_shared/authentication.py", line 202, in authenticate_for_token Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone authenticate(auth_info, auth_context) Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/api/_shared/authentication.py", line 170, in authenticate Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = method.authenticate(auth_info.get_method_data(method_name)) Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/auth/plugins/password.py", line 34, in authenticate Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone PROVIDERS.identity_api.authenticate( Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/common/manager.py", line 111, in wrapped Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone __ret_val = __f(*args, **kwargs) Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/notifications.py", line 718, in wrapper Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone result = f(wrapped_self, user_id, *args, **kwargs) Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 533, in wrapper Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return f(self, *args, **kwargs) Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 545, in wrapper Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return f(self, *args, **kwargs) Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 1069, in authenticate Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone PROVIDERS.shadow_users_api.set_last_active_at(ref['id']) Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/shadow_backends/sql.py", line 166, in set_last_active_at Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone with sql.session_for_write() as session: Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone next(self.gen) Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/sqlalchemy.py", line 56, in wrap_session Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone with sess as s: Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone next(self.gen) Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1042, in _transaction_scope Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone with current._produce_block( Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone next(self.gen) Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 649, in _session Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone with excutils.save_and_reraise_exception(): Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone self.force_reraise() Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone raise self.value Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 647, in _session Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone self._end_session_transaction(self.session) Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 675, in _end_session_transaction Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone session.commit() Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2017, in commit Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone trans.commit(_to_root=True) Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "<string>", line 2, in commit Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone ret_value = fn(self, *arg, **kw) Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1302, in commit Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone self._prepare_impl() Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "<string>", line 2, in _prepare_impl Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone ret_value = fn(self, *arg, **kw) Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1277, in _prepare_impl Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone self.session.flush() Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4341, in flush Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone self._flush(objects) Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4476, in _flush Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone with util.safe_reraise(): Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__ Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone raise exc_value.with_traceback(exc_tb) Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4437, in _flush Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone flush_context.execute() Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone rec.execute(self) Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone util.preloaded.orm_persistence.save_obj( Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 85, in save_obj Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone _emit_update_statements( Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 912, in _emit_update_statements Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone c = connection.execute( Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1418, in execute Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone return meth( Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone return connection._execute_clauseelement( Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone ret = self._execute_context( Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self._exec_single_context( Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1889, in _exec_single_context Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone str_statement, effective_parameters = fn( Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/events.py", line 176, in wrap_before_cursor_execute Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone orig_fn( Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/sqlalchemy.py", line 71, in handler Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone profiler.start(name, info=info) Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/profiler.py", line 82, in start Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone profiler.start(name, info=info) Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/profiler.py", line 410, in start Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone self._notify("%s-start" % name, info) Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/profiler.py", line 436, in _notify Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone notifier.notify(payload) Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/notifier.py", line 38, in notify Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone __notifier(info) Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/drivers/redis_driver.py", line 75, in notify Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone self.db.lpush(key, jsonutils.dumps(data)) Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_serialization/jsonutils.py", line 198, in dumps Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone return json.dumps(obj, default=default, **kwargs) Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/json/__init__.py", line 238, in dumps Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone **kw).encode(obj) Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/json/encoder.py", line 199, in encode Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone chunks = self.iterencode(o, _one_shot=True) Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone return _iterencode(o, 0) Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_serialization/jsonutils.py", line 176, in to_primitive Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone raise ValueError("Cannot convert %r to primitive" % (value,)) Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: [pid: 56399|app: 0|req: 225/451] 200.225.47.15 () {70 vars in 1374 bytes} [Wed Aug 7 17:03:06 2024] POST /identity/v3/auth/tokens => generated 0 bytes in 61 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0) Example failure:- https://2a378a06b97cbac17818-b80014e5a7a6453c822c4dc8f22159da.ssl.cf5.rackcdn.com/925376/2/check/neutron- ovs-rally-task/fb03014/job-output.txt also cinder-rally-tasks broken Builds:- https://zuul.opendev.org/t/openstack/builds?job_name=neutron- ovs-rally-task&job_name=cinder-rally-task&branch=master&skip=0 Broken since patch https://review.opendev.org/c/openstack/keystone/+/924892 ** Affects: neutron Importance: Undecided Status: New ** Description changed: Test fails as:- 2024-08-07 17:06:05.650302 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients [-] Unable to authenticate for user c_rally_927546a8_h6aDLbnK in project c_rally_927546a8_ahUUmlp9: keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500) 2024-08-07 17:06:05.650941 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients Traceback (most recent call last): 2024-08-07 17:06:05.651113 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/rally-openstack/rally_openstack/common/osclients.py", line 269, in auth_ref 2024-08-07 17:06:05.651156 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients self.cache["keystone_auth_ref"] = plugin.get_access(sess) 2024-08-07 17:06:05.651193 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/base.py", line 131, in get_access 2024-08-07 17:06:05.651229 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients self.auth_ref = self.get_auth_ref(session) 2024-08-07 17:06:05.651263 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/generic/base.py", line 205, in get_auth_ref 2024-08-07 17:06:05.651334 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients return self._plugin.get_auth_ref(session, **kwargs) 2024-08-07 17:06:05.651348 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/v3/base.py", line 185, in get_auth_ref 2024-08-07 17:06:05.651356 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients resp = session.post(token_url, json=body, headers=headers, 2024-08-07 17:06:05.651363 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/session.py", line 1162, in post 2024-08-07 17:06:05.651370 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients return self.request(url, 'POST', **kwargs) 2024-08-07 17:06:05.651377 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/session.py", line 985, in request 2024-08-07 17:06:05.651384 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients raise exceptions.from_response(resp, method, url) 2024-08-07 17:06:05.651391 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500) From keyston log:- Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: CRITICAL keystone [None req-96647e8e-2585-4279-80fa-c4fa97b8c455 None None] Unhandled error: ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone Traceback (most recent call last): Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 1498, in __call__ Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.wsgi_app(environ, start_response) Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/werkzeug/middleware/proxy_fix.py", line 183, in __call__ Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.app(environ, start_response) Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__ Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs) Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/base.py", line 124, in __call__ Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = req.get_response(self.application) Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application( Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response) Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 143, in __call__ Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return resp(environ, start_response) Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__ Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs) Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/base.py", line 124, in __call__ Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = req.get_response(self.application) Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application( Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response) Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__ Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs) Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/web.py", line 131, in __call__ Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone return request.get_response(self.application) Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application( Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response) Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__ Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs) Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/request_id.py", line 58, in __call__ Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = req.get_response(self.application) Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application( Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response) Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/server/flask/request_processing/middleware/url_normalize.py", line 38, in __call__ Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.app(environ, start_response) Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__ Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs) Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/keystonemiddleware/auth_token/__init__.py", line 340, in __call__ Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = req.get_response(self._app) Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application( Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response) Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/werkzeug/middleware/dispatcher.py", line 81, in __call__ Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return app(environ, start_response) Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 1476, in wsgi_app Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = self.handle_exception(e) Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e) Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e) Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e) Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone [Previous line repeated 28 more times] Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 1473, in wsgi_app Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = self.full_dispatch_request() Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 882, in full_dispatch_request Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone rv = self.handle_user_exception(e) Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e) Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e) Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e) Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone [Previous line repeated 28 more times] Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 880, in full_dispatch_request Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone rv = self.dispatch_request() Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 865, in dispatch_request Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return] Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 489, in wrapper Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = resource(*args, **kwargs) Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/views.py", line 110, in view Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return current_app.ensure_sync(self.dispatch_request)(**kwargs) # type: ignore[no-any-return] Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 604, in dispatch_request Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = meth(*args, **kwargs) Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/server/flask/common.py", line 1146, in wrapper Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return f(*args, **kwargs) Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/api/auth.py", line 328, in post Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone token = authentication.authenticate_for_token(auth_data) Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/api/_shared/authentication.py", line 202, in authenticate_for_token Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone authenticate(auth_info, auth_context) Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/api/_shared/authentication.py", line 170, in authenticate Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = method.authenticate(auth_info.get_method_data(method_name)) Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/auth/plugins/password.py", line 34, in authenticate Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone PROVIDERS.identity_api.authenticate( Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/common/manager.py", line 111, in wrapped Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone __ret_val = __f(*args, **kwargs) Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/notifications.py", line 718, in wrapper Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone result = f(wrapped_self, user_id, *args, **kwargs) Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 533, in wrapper Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return f(self, *args, **kwargs) Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 545, in wrapper Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return f(self, *args, **kwargs) Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 1069, in authenticate Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone PROVIDERS.shadow_users_api.set_last_active_at(ref['id']) Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/shadow_backends/sql.py", line 166, in set_last_active_at Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone with sql.session_for_write() as session: Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone next(self.gen) Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/sqlalchemy.py", line 56, in wrap_session Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone with sess as s: Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone next(self.gen) Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1042, in _transaction_scope Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone with current._produce_block( Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone next(self.gen) Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 649, in _session Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone with excutils.save_and_reraise_exception(): Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone self.force_reraise() Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone raise self.value Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 647, in _session Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone self._end_session_transaction(self.session) Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 675, in _end_session_transaction Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone session.commit() Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2017, in commit Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone trans.commit(_to_root=True) Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "<string>", line 2, in commit Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone ret_value = fn(self, *arg, **kw) Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1302, in commit Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone self._prepare_impl() Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "<string>", line 2, in _prepare_impl Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone ret_value = fn(self, *arg, **kw) Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1277, in _prepare_impl Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone self.session.flush() Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4341, in flush Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone self._flush(objects) Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4476, in _flush Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone with util.safe_reraise(): Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__ Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone raise exc_value.with_traceback(exc_tb) Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4437, in _flush Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone flush_context.execute() Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone rec.execute(self) Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone util.preloaded.orm_persistence.save_obj( Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 85, in save_obj Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone _emit_update_statements( Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 912, in _emit_update_statements Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone c = connection.execute( Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1418, in execute Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone return meth( Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone return connection._execute_clauseelement( Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone ret = self._execute_context( Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self._exec_single_context( Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1889, in _exec_single_context Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone str_statement, effective_parameters = fn( Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/events.py", line 176, in wrap_before_cursor_execute Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone orig_fn( Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/sqlalchemy.py", line 71, in handler Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone profiler.start(name, info=info) Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/profiler.py", line 82, in start Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone profiler.start(name, info=info) Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/profiler.py", line 410, in start Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone self._notify("%s-start" % name, info) Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/profiler.py", line 436, in _notify Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone notifier.notify(payload) Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/notifier.py", line 38, in notify Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone __notifier(info) Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/drivers/redis_driver.py", line 75, in notify Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone self.db.lpush(key, jsonutils.dumps(data)) Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_serialization/jsonutils.py", line 198, in dumps Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone return json.dumps(obj, default=default, **kwargs) Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/json/__init__.py", line 238, in dumps Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone **kw).encode(obj) Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/json/encoder.py", line 199, in encode Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone chunks = self.iterencode(o, _one_shot=True) Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone return _iterencode(o, 0) Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_serialization/jsonutils.py", line 176, in to_primitive Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone raise ValueError("Cannot convert %r to primitive" % (value,)) Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive - Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone + Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: [pid: 56399|app: 0|req: 225/451] 200.225.47.15 () {70 vars in 1374 bytes} [Wed Aug 7 17:03:06 2024] POST /identity/v3/auth/tokens => generated 0 bytes in 61 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0) Example failure:- https://2a378a06b97cbac17818-b80014e5a7a6453c822c4dc8f22159da.ssl.cf5.rackcdn.com/925376/2/check/neutron- ovs-rally-task/fb03014/job-output.txt also cinder-rally-tasks broken Builds:- https://zuul.opendev.org/t/openstack/builds?job_name=neutron- ovs-rally-task&job_name=cinder-rally-task&branch=master&skip=0 + + Broken since patch + https://review.opendev.org/c/openstack/keystone/+/924892 -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2076430 Title: rally job broken with recent keystone change and fails with ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive Status in neutron: New Bug description: Test fails as:- 2024-08-07 17:06:05.650302 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients [-] Unable to authenticate for user c_rally_927546a8_h6aDLbnK in project c_rally_927546a8_ahUUmlp9: keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500) 2024-08-07 17:06:05.650941 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients Traceback (most recent call last): 2024-08-07 17:06:05.651113 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/rally-openstack/rally_openstack/common/osclients.py", line 269, in auth_ref 2024-08-07 17:06:05.651156 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients self.cache["keystone_auth_ref"] = plugin.get_access(sess) 2024-08-07 17:06:05.651193 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/base.py", line 131, in get_access 2024-08-07 17:06:05.651229 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients self.auth_ref = self.get_auth_ref(session) 2024-08-07 17:06:05.651263 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/generic/base.py", line 205, in get_auth_ref 2024-08-07 17:06:05.651334 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients return self._plugin.get_auth_ref(session, **kwargs) 2024-08-07 17:06:05.651348 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/v3/base.py", line 185, in get_auth_ref 2024-08-07 17:06:05.651356 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients resp = session.post(token_url, json=body, headers=headers, 2024-08-07 17:06:05.651363 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/session.py", line 1162, in post 2024-08-07 17:06:05.651370 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients return self.request(url, 'POST', **kwargs) 2024-08-07 17:06:05.651377 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/session.py", line 985, in request 2024-08-07 17:06:05.651384 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients raise exceptions.from_response(resp, method, url) 2024-08-07 17:06:05.651391 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500) From keyston log:- Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: CRITICAL keystone [None req-96647e8e-2585-4279-80fa-c4fa97b8c455 None None] Unhandled error: ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone Traceback (most recent call last): Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 1498, in __call__ Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.wsgi_app(environ, start_response) Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/werkzeug/middleware/proxy_fix.py", line 183, in __call__ Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.app(environ, start_response) Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__ Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs) Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/base.py", line 124, in __call__ Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = req.get_response(self.application) Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application( Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response) Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 143, in __call__ Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return resp(environ, start_response) Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__ Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs) Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/base.py", line 124, in __call__ Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = req.get_response(self.application) Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application( Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response) Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__ Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs) Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/web.py", line 131, in __call__ Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone return request.get_response(self.application) Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application( Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response) Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__ Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs) Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/request_id.py", line 58, in __call__ Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = req.get_response(self.application) Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application( Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response) Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/server/flask/request_processing/middleware/url_normalize.py", line 38, in __call__ Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.app(environ, start_response) Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__ Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs) Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/keystonemiddleware/auth_token/__init__.py", line 340, in __call__ Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = req.get_response(self._app) Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application( Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response) Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/werkzeug/middleware/dispatcher.py", line 81, in __call__ Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return app(environ, start_response) Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 1476, in wsgi_app Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = self.handle_exception(e) Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e) Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e) Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e) Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone [Previous line repeated 28 more times] Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 1473, in wsgi_app Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = self.full_dispatch_request() Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 882, in full_dispatch_request Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone rv = self.handle_user_exception(e) Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e) Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e) Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e) Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone [Previous line repeated 28 more times] Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 880, in full_dispatch_request Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone rv = self.dispatch_request() Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 865, in dispatch_request Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return] Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 489, in wrapper Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = resource(*args, **kwargs) Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/views.py", line 110, in view Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return current_app.ensure_sync(self.dispatch_request)(**kwargs) # type: ignore[no-any-return] Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 604, in dispatch_request Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = meth(*args, **kwargs) Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/server/flask/common.py", line 1146, in wrapper Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return f(*args, **kwargs) Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/api/auth.py", line 328, in post Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone token = authentication.authenticate_for_token(auth_data) Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/api/_shared/authentication.py", line 202, in authenticate_for_token Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone authenticate(auth_info, auth_context) Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/api/_shared/authentication.py", line 170, in authenticate Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = method.authenticate(auth_info.get_method_data(method_name)) Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/auth/plugins/password.py", line 34, in authenticate Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone PROVIDERS.identity_api.authenticate( Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/common/manager.py", line 111, in wrapped Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone __ret_val = __f(*args, **kwargs) Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/notifications.py", line 718, in wrapper Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone result = f(wrapped_self, user_id, *args, **kwargs) Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 533, in wrapper Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return f(self, *args, **kwargs) Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 545, in wrapper Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return f(self, *args, **kwargs) Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 1069, in authenticate Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone PROVIDERS.shadow_users_api.set_last_active_at(ref['id']) Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/shadow_backends/sql.py", line 166, in set_last_active_at Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone with sql.session_for_write() as session: Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone next(self.gen) Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/sqlalchemy.py", line 56, in wrap_session Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone with sess as s: Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone next(self.gen) Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1042, in _transaction_scope Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone with current._produce_block( Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone next(self.gen) Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 649, in _session Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone with excutils.save_and_reraise_exception(): Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone self.force_reraise() Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone raise self.value Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 647, in _session Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone self._end_session_transaction(self.session) Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 675, in _end_session_transaction Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone session.commit() Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2017, in commit Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone trans.commit(_to_root=True) Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "<string>", line 2, in commit Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone ret_value = fn(self, *arg, **kw) Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1302, in commit Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone self._prepare_impl() Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "<string>", line 2, in _prepare_impl Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone ret_value = fn(self, *arg, **kw) Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1277, in _prepare_impl Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone self.session.flush() Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4341, in flush Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone self._flush(objects) Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4476, in _flush Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone with util.safe_reraise(): Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__ Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone raise exc_value.with_traceback(exc_tb) Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4437, in _flush Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone flush_context.execute() Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone rec.execute(self) Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone util.preloaded.orm_persistence.save_obj( Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 85, in save_obj Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone _emit_update_statements( Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 912, in _emit_update_statements Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone c = connection.execute( Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1418, in execute Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone return meth( Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone return connection._execute_clauseelement( Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone ret = self._execute_context( Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self._exec_single_context( Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1889, in _exec_single_context Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone str_statement, effective_parameters = fn( Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/events.py", line 176, in wrap_before_cursor_execute Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone orig_fn( Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/sqlalchemy.py", line 71, in handler Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone profiler.start(name, info=info) Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/profiler.py", line 82, in start Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone profiler.start(name, info=info) Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/profiler.py", line 410, in start Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone self._notify("%s-start" % name, info) Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/profiler.py", line 436, in _notify Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone notifier.notify(payload) Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/notifier.py", line 38, in notify Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone __notifier(info) Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/drivers/redis_driver.py", line 75, in notify Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone self.db.lpush(key, jsonutils.dumps(data)) Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_serialization/jsonutils.py", line 198, in dumps Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone return json.dumps(obj, default=default, **kwargs) Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/json/__init__.py", line 238, in dumps Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone **kw).encode(obj) Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/json/encoder.py", line 199, in encode Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone chunks = self.iterencode(o, _one_shot=True) Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone return _iterencode(o, 0) Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_serialization/jsonutils.py", line 176, in to_primitive Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone raise ValueError("Cannot convert %r to primitive" % (value,)) Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: [pid: 56399|app: 0|req: 225/451] 200.225.47.15 () {70 vars in 1374 bytes} [Wed Aug 7 17:03:06 2024] POST /identity/v3/auth/tokens => generated 0 bytes in 61 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0) Example failure:- https://2a378a06b97cbac17818-b80014e5a7a6453c822c4dc8f22159da.ssl.cf5.rackcdn.com/925376/2/check/neutron- ovs-rally-task/fb03014/job-output.txt also cinder-rally-tasks broken Builds:- https://zuul.opendev.org/t/openstack/builds?job_name=neutron- ovs-rally-task&job_name=cinder-rally-task&branch=master&skip=0 Broken since patch https://review.opendev.org/c/openstack/keystone/+/924892 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2076430/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp