Public bug reported: Description of problem:
When deploying with a modified list of Neutron API policies, post deployment, policies which worked on previous versions will result in Neutron API Server returning 'HttpException: 500' when using the API with non admin users. Additional API policies which were passed during deployment: http://paste.openstack.org/show/751347/ Example: 1. Source credentials with non admin user [stack@undercloud-0 ~]$ source /home/stack/overcloudrc_user_tenant 2. Query port list with as non admin user (overcloud) [stack@undercloud-0 ~]$ openstack port list At this point, neutron will return: HttpException: 500: Server Error for url: http://10.35.141.150:9696/v2.0/ports, Internal Server Error And the following exception will be generated inside server.log on controller nodes: server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors [req-98bfd77f-1fc1-4d13-a0fd-02e82f6caa53 2236f6cc04c04964a0b435599ffb7acb ef4de28cbec04ea785b855010e7f46a1 - default default] An error occurred during processing the request: POST /v2.0/ports HTTP/1.0 server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors Traceback (most recent call last): server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/oslo_middleware/catch_errors.py", line 40, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors response = req.get_response(self.application) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors application, catch_exc_info=False) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors app_iter = application(self.environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors resp = self.call_func(req, *args, **kw) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return self.func(req, *args, **kwargs) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return request.get_response(self.application) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors application, catch_exc_info=False) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors app_iter = application(self.environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors resp = self.call_func(req, *args, **kw) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return self.func(req, *args, **kwargs) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 333, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors response = req.get_response(self._app) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors application, catch_exc_info=False) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors app_iter = application(self.environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return resp(environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return resp(environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/routes/middleware.py", line 141, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors response = self.app(environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return resp(environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/pecan/middleware/recursive.py", line 56, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return self.application(environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/pecan/core.py", line 840, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return super(Pecan, self).__call__(environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/pecan/core.py", line 736, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors state server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/pecan/core.py", line 865, in handle_hooks server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return super(Pecan, self).handle_hooks(hooks, *args, **kw) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/pecan/core.py", line 342, in handle_hooks server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors result = getattr(hook, hook_type)(*args) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 185, in after server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors for item in to_process server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 189, in <listcomp> server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors pluralized=collection))] server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 207, in _get_filtered_item server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors neutron_context, controller, resource, collection, data) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 226, in _exclude_attributes_by_policy server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors for attr_name in data.keys(): server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors RuntimeError: dictionary changed size during iteration server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors } Version-Release number of selected component (if applicable): Compose: RHOS_TRUNK-15.0-RHEL-8-20190509.n.1 rpm -qa | grep neutron puppet-neutron-14.4.1-0.20190420042323.400fd54.el8ost.noarch python3-neutronclient-6.12.0-0.20190312100012.680b417.el8ost.noarch How reproducible: Always Steps to Reproduce: 1. Deploy Overcloud with modified Neutron APIs 2. Create non admin user/tenant 3. Attempt to list ports Actual results: Fail to retrieve ports and receive python exceptions Expected results: List of ports is returned Additional info: ** Affects: neutron Importance: Undecided Assignee: Nate Johnston (nate-johnston) Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1829304 Title: Neutron returns HttpException: 500 on certain operations with modified list of policies for non-admin users Status in neutron: New Bug description: Description of problem: When deploying with a modified list of Neutron API policies, post deployment, policies which worked on previous versions will result in Neutron API Server returning 'HttpException: 500' when using the API with non admin users. Additional API policies which were passed during deployment: http://paste.openstack.org/show/751347/ Example: 1. Source credentials with non admin user [stack@undercloud-0 ~]$ source /home/stack/overcloudrc_user_tenant 2. Query port list with as non admin user (overcloud) [stack@undercloud-0 ~]$ openstack port list At this point, neutron will return: HttpException: 500: Server Error for url: http://10.35.141.150:9696/v2.0/ports, Internal Server Error And the following exception will be generated inside server.log on controller nodes: server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors [req-98bfd77f-1fc1-4d13-a0fd-02e82f6caa53 2236f6cc04c04964a0b435599ffb7acb ef4de28cbec04ea785b855010e7f46a1 - default default] An error occurred during processing the request: POST /v2.0/ports HTTP/1.0 server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors Traceback (most recent call last): server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/oslo_middleware/catch_errors.py", line 40, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors response = req.get_response(self.application) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors application, catch_exc_info=False) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors app_iter = application(self.environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors resp = self.call_func(req, *args, **kw) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return self.func(req, *args, **kwargs) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return request.get_response(self.application) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors application, catch_exc_info=False) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors app_iter = application(self.environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors resp = self.call_func(req, *args, **kw) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return self.func(req, *args, **kwargs) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 333, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors response = req.get_response(self._app) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors application, catch_exc_info=False) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors app_iter = application(self.environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return resp(environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return resp(environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/routes/middleware.py", line 141, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors response = self.app(environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return resp(environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/pecan/middleware/recursive.py", line 56, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return self.application(environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/pecan/core.py", line 840, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return super(Pecan, self).__call__(environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/pecan/core.py", line 736, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors state server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/pecan/core.py", line 865, in handle_hooks server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return super(Pecan, self).handle_hooks(hooks, *args, **kw) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/pecan/core.py", line 342, in handle_hooks server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors result = getattr(hook, hook_type)(*args) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 185, in after server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors for item in to_process server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 189, in <listcomp> server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors pluralized=collection))] server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 207, in _get_filtered_item server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors neutron_context, controller, resource, collection, data) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 226, in _exclude_attributes_by_policy server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors for attr_name in data.keys(): server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors RuntimeError: dictionary changed size during iteration server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors } Version-Release number of selected component (if applicable): Compose: RHOS_TRUNK-15.0-RHEL-8-20190509.n.1 rpm -qa | grep neutron puppet-neutron-14.4.1-0.20190420042323.400fd54.el8ost.noarch python3-neutronclient-6.12.0-0.20190312100012.680b417.el8ost.noarch How reproducible: Always Steps to Reproduce: 1. Deploy Overcloud with modified Neutron APIs 2. Create non admin user/tenant 3. Attempt to list ports Actual results: Fail to retrieve ports and receive python exceptions Expected results: List of ports is returned Additional info: To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1829304/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp