** Also affects: nova/newton Importance: Undecided Status: New ** Also affects: nova/ocata Importance: Undecided Status: New
** Changed in: nova Importance: Undecided => High ** Changed in: nova/newton Importance: Undecided => High ** Changed in: nova/ocata Importance: Undecided => High ** Changed in: nova/newton Assignee: (unassigned) => Artom Lifshitz (notartom) ** Changed in: nova/ocata Status: New => In Progress ** Changed in: nova/ocata Assignee: (unassigned) => Lee Yarwood (lyarwood) ** Changed in: nova/newton Status: New => In Progress -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1643444 Title: TenantUsagesTestJSON.test_list_usage_all_tenants 500 from Db layer Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) newton series: In Progress Status in OpenStack Compute (nova) ocata series: In Progress Bug description: I have a newton setup with 3 api(controller) node. TenantUsagesTestJSON.test_list_usage_all_tenants failed once , the failure looks similar to one described in the already fixed in https://bugs.launchpad.net/nova/+bug/1487570 , but it is different api call so it can have similar issue. Likely you have an old list of ids, and trying to fetch more info about an already deleted instance. The tempest exception: 2016-11-20 00:07:18,606 27600 INFO [tempest.lib.common.rest_client] Request (TenantUsagesTestJSON:test_list_usage_all_tenants): 500 GET http://[2620:52:0:13b8:5054:ff:fe3e:4]:8774/v2.1/os-simple-tenant-usage?detailed=1&start=2016-11-19T00%3A07%3A17.645313&end=2016-11-21T00%3A07%3A17.645313 0.134s 2016-11-20 00:07:18,607 27600 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'} Body: None Response - Headers: {'status': '500', 'content-length': '205', 'content-location': 'http://[2620:52:0:13b8:5054:ff:fe3e:4]:8774/v2.1/os-simple-tenant-usage?detailed=1&start=2016-11-19T00%3A07%3A17.645313&end=2016-11-21T00%3A07%3A17.645313', 'x-compute-request-id': 'req-3ff84c48-b03e-4f23-8f33-227719a0ced4', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Sun, 20 Nov 2016 05:07:18 GMT', 'content-type': 'application/json; charset=UTF-8'} Body: {"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'nova.exception.InstanceNotFound'>", "code": 500}} File "/home/stack/tempest-dir/tempest/api/compute/admin/test_simple_tenant_usage.py", line 73, in test_list_usage_all_tenants start=self.start, end=self.end, detailed="1")['tenant_usages'][0] File "/home/stack/tempest-dir/tempest/api/compute/admin/test_simple_tenant_usage.py", line 63, in call_until_valid self.assertEqual(test_utils.call_until_true(is_valid, duration, 1), File "/home/stack/tempest-dir/tempest/lib/common/utils/test_utils.py", line 103, in call_until_true if func(): File "/home/stack/tempest-dir/tempest/api/compute/admin/test_simple_tenant_usage.py", line 59, in is_valid self.resp = func(*args, **kwargs) File "/home/stack/tempest-dir/tempest/lib/services/compute/tenant_usages_client.py", line 37, in list_tenant_usages resp, body = self.get(url) File "/home/stack/tempest-dir/tempest/lib/common/rest_client.py", line 291, in get return self.request('GET', url, extra_headers, headers) File "/home/stack/tempest-dir/tempest/lib/services/compute/base_compute_client.py", line 48, in request method, url, extra_headers, headers, body, chunked) File "/home/stack/tempest-dir/tempest/lib/common/rest_client.py", line 664, in request self._error_checker(resp, resp_body) File "/home/stack/tempest-dir/tempest/lib/common/rest_client.py", line 827, in _error_checker message=message) tempest.lib.exceptions.ServerFault: Got server fault Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'nova.exception.InstanceNotFound'> The related nova api log (node-2): 2016-11-20 05:07:18.476 111884 DEBUG nova.api.openstack.wsgi [req-3ff84c48-b03e-4f23-8f33-227719a0ced4 d4852c5eaf2645e2aab0c7485939395a 998c56750d4a4056853829f088ce2be9 - default default] Calling method '<bound method SimpleTenantUsageController.index of <nova.api.openstack.compute.simple_tenant_usage.SimpleTenantUsageController object at 0x7fc62d0>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636 2016-11-20 05:07:18.507 111884 DEBUG nova.objects.instance [req-3ff84c48-b03e-4f23-8f33-227719a0ced4 d4852c5eaf2645e2aab0c7485939395a 998c56750d4a4056853829f088ce2be9 - default default] Lazy-loading 'flavor' on Instance uuid 5f3a04c2-ab22-4378-9512-bfd4f9fb0a52 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1013 2016-11-20 05:07:18.556 111884 DEBUG nova.objects.instance [req-3ff84c48-b03e-4f23-8f33-227719a0ced4 d4852c5eaf2645e2aab0c7485939395a 998c56750d4a4056853829f088ce2be9 - default default] Lazy-loading 'flavor' on Instance uuid 5f3a04c2-ab22-4378-9512-bfd4f9fb0a52 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1013 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions [req-3ff84c48-b03e-4f23-8f33-227719a0ced4 d4852c5eaf2645e2aab0c7485939395a 998c56750d4a4056853829f088ce2be9 - default default] Unexpected exception in API method 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/simple_tenant_usage.py", line 238, in index 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions detailed=detailed) 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/simple_tenant_usage.py", line 123, in _tenant_usages_for_period 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions info['memory_mb'] = instance.flavor.memory_mb 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 67, in getter 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions self.obj_load_attr(name) 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1041, in obj_load_attr 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions self._load_flavor() 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 863, in _load_flavor 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions expected_attrs=['flavor', 'system_metadata']) 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions result = fn(cls, context, *args, **kwargs) 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 457, in get_by_uuid 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions use_slave=use_slave) 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 449, in _db_instance_get_by_uuid 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions columns_to_join=columns_to_join) 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 725, in instance_get_by_uuid 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions return IMPL.instance_get_by_uuid(context, uuid, columns_to_join) 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 169, in wrapper 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions return f(context, *args, **kwargs) 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1867, in instance_get_by_uuid 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions columns_to_join=columns_to_join) 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1876, in _instance_get_by_uuid 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions raise exception.InstanceNotFound(instance_id=uuid) 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions InstanceNotFound: Instance 5f3a04c2-ab22-4378-9512-bfd4f9fb0a52 could not be found. 2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions 2016-11-20 05:07:18.601 111884 INFO nova.api.openstack.wsgi [req-3ff84c48-b03e-4f23-8f33-227719a0ced4 d4852c5eaf2645e2aab0c7485939395a 998c56750d4a4056853829f088ce2be9 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'nova.exception.InstanceNotFound'> 2016-11-20 05:07:18.601 111884 DEBUG nova.api.openstack.wsgi [req-3ff84c48-b03e-4f23-8f33-227719a0ced4 d4852c5eaf2645e2aab0c7485939395a 998c56750d4a4056853829f088ce2be9 - default default] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'nova.exception.InstanceNotFound'> __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1044 2016-11-20 05:07:18.602 111884 INFO nova.osapi_compute.wsgi.server [req-3ff84c48-b03e-4f23-8f33-227719a0ced4 d4852c5eaf2645e2aab0c7485939395a 998c56750d4a4056853829f088ce2be9 - default default] fd00:fd00:fd00:2000::11 "GET /v2.1/os-simple-tenant-usage?detailed=1&start=2016-11-19T00%3A07%3A17.645313&end=2016-11-21T00%3A07%3A17.645313 HTTP/1.1" status: 500 len: 567 time: 0.1304212 2016-11-20 05:07:18.929 111891 DEBUG neutronclient.v2_0.client [req-0063c383-52da-4238-bda6-96079822b4a9 e87a6c64041247c186f36fbeaa1987cb 72ac28353e7f4c03a79ba6abd32c189a - default default] GET call to neutron for http://[fd00:fd00:fd00:2000::18]:9696/v2.0/networks.json?tenant_id=72ac28353e7f4c03a79ba6abd32c189a&shared=False used request id req-9f2b5b0f-5b12-4cb4-981a-000e3bb3fe8c _append_request_id /usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py:127 2016-11-20 05:07:18.989 111891 DEBUG neutronclient.v2_0.client [req-0063c383-52da-4238-bda6-96079822b4a9 e87a6c64041247c186f36fbeaa1987cb 72ac28353e7f4c03a79ba6abd32c189a - default default] GET call to neutron for http://[fd00:fd00:fd00:2000::18]:9696/v2.0/networks.json?shared=True used request id req-4bdabb6e-cbd8-449d-a4b8-61355625bd88 _append_request_id /usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py:127 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1643444/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp