Public bug reported: Description =========== Currently, when we get servers list in multi cell, we scatter gather results from cells, but if we get back exception or timeout from cell, we will get 500 error finally.
We should handle raise or timeout after getting back all results. Maybe we could just skip the error result like quota stuff: https://github.com/openstack/nova/blob/e9ce5c4c95edc869ab2cf82ca0733a2821c384ad/nova/quota.py#L1865 Steps to reproduce ================== 1. raise some error exception when gather results. 2. get server list curl -g -i -X GET http://XXX/compute/v2.1/servers -H "OpenStack-API-Version: compute 2.53" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-OpenStack-Nova-API-Version: 2.53" -H "X-Auth-Token: $TOKEN" HTTP/1.1 500 Internal Server Error Date: Wed, 22 Nov 2017 07:11:39 GMT Server: Apache/2.4.18 (Ubuntu) OpenStack-API-Version: compute 2.53 X-OpenStack-Nova-API-Version: 2.53 Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version Content-Type: application/json; charset=UTF-8 Content-Length: 193 x-openstack-request-id: req-429c02e7-84db-4c2d-98fd-dd1af0186383 x-compute-request-id: req-429c02e7-84db-4c2d-98fd-dd1af0186383 Connection: close {"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<type 'exceptions.TypeError'>", "code": 500}} 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/extensions.py", line 336, in wrapped 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/validation/__init__.py", line 181, in wrapper 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/validation/__init__.py", line 181, in wrapper 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 152, in index 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions servers = self._get_servers(req, is_detail=False) 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 308, in _get_servers 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions sort_keys=sort_keys, sort_dirs=sort_dirs) 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 2394, in get_all 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions context, filters, limit, marker, fields, sort_keys, sort_dirs) 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/instance_list.py", line 251, in get_instance_objects_sorted 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions expected_attrs) 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/objects/instance.py", line 1196, in _make_instance_list 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions for db_inst in db_inst_list: 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/instance_list.py", line 228, in get_instances_sorted 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions for i in heapq.merge(*results.values()): 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/heapq.py", line 373, in merge 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions for itnum, it in enumerate(map(iter, iterables)): 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions TypeError: 'object' object is not iterable 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions 2017-11-22 02:11:39.935 9792 INFO nova.api.openstack.wsgi [req-429c02e7-84db-4c2d-98fd-dd1af0186383 - admin] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <type 'exceptions.TypeError'> 2017-11-22 02:11:39.937 9792 DEBUG nova.api.openstack.wsgi [req-429c02e7-84db-4c2d-98fd-dd1af0186383 - admin] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <type 'exceptions.TypeError'> __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1029 2017-11-22 02:11:39.938 9792 INFO nova.api.openstack.requestlog [req-429c02e7-84db-4c2d-98fd-dd1af0186383 - admin] 10.76.6.31 "GET /compute/v2.1/servers" status: 500 len: 193 microversion: 2.53 time: 0.746751 ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1734012 Title: Handle exception in get_instance_sorted when scattering gather results from all cells. Status in OpenStack Compute (nova): New Bug description: Description =========== Currently, when we get servers list in multi cell, we scatter gather results from cells, but if we get back exception or timeout from cell, we will get 500 error finally. We should handle raise or timeout after getting back all results. Maybe we could just skip the error result like quota stuff: https://github.com/openstack/nova/blob/e9ce5c4c95edc869ab2cf82ca0733a2821c384ad/nova/quota.py#L1865 Steps to reproduce ================== 1. raise some error exception when gather results. 2. get server list curl -g -i -X GET http://XXX/compute/v2.1/servers -H "OpenStack-API-Version: compute 2.53" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-OpenStack-Nova-API-Version: 2.53" -H "X-Auth-Token: $TOKEN" HTTP/1.1 500 Internal Server Error Date: Wed, 22 Nov 2017 07:11:39 GMT Server: Apache/2.4.18 (Ubuntu) OpenStack-API-Version: compute 2.53 X-OpenStack-Nova-API-Version: 2.53 Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version Content-Type: application/json; charset=UTF-8 Content-Length: 193 x-openstack-request-id: req-429c02e7-84db-4c2d-98fd-dd1af0186383 x-compute-request-id: req-429c02e7-84db-4c2d-98fd-dd1af0186383 Connection: close {"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<type 'exceptions.TypeError'>", "code": 500}} 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/extensions.py", line 336, in wrapped 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/validation/__init__.py", line 181, in wrapper 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/validation/__init__.py", line 181, in wrapper 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 152, in index 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions servers = self._get_servers(req, is_detail=False) 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 308, in _get_servers 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions sort_keys=sort_keys, sort_dirs=sort_dirs) 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 2394, in get_all 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions context, filters, limit, marker, fields, sort_keys, sort_dirs) 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/instance_list.py", line 251, in get_instance_objects_sorted 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions expected_attrs) 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/objects/instance.py", line 1196, in _make_instance_list 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions for db_inst in db_inst_list: 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/instance_list.py", line 228, in get_instances_sorted 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions for i in heapq.merge(*results.values()): 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/heapq.py", line 373, in merge 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions for itnum, it in enumerate(map(iter, iterables)): 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions TypeError: 'object' object is not iterable 2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions 2017-11-22 02:11:39.935 9792 INFO nova.api.openstack.wsgi [req-429c02e7-84db-4c2d-98fd-dd1af0186383 - admin] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <type 'exceptions.TypeError'> 2017-11-22 02:11:39.937 9792 DEBUG nova.api.openstack.wsgi [req-429c02e7-84db-4c2d-98fd-dd1af0186383 - admin] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <type 'exceptions.TypeError'> __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1029 2017-11-22 02:11:39.938 9792 INFO nova.api.openstack.requestlog [req-429c02e7-84db-4c2d-98fd-dd1af0186383 - admin] 10.76.6.31 "GET /compute/v2.1/servers" status: 500 len: 193 microversion: 2.53 time: 0.746751 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1734012/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp