Public bug reported: Example stack trace:
2023-01-13 03:17:37,055 WARNING [oslo_db.sqlalchemy.exc_filters] DBAPIError exception wrapped. Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/urllib3/connectionpool.py", line 440, in _make_request httplib_response = conn.getresponse(buffering=True) TypeError: HTTPConnection.getresponse() got an unexpected keyword argument 'buffering' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/engine/cursor.py", line 977, in fetchall rows = dbapi_cursor.fetchall() sqlite3.InterfaceError: Cursor needed to be reset because of commit/rollback and can no longer be fetched from. 2023-01-13 03:17:37,057 ERROR [nova.api.openstack.wsgi] Unexpected exception in API method Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/urllib3/connectionpool.py", line 440, in _make_request httplib_response = conn.getresponse(buffering=True) TypeError: HTTPConnection.getresponse() got an unexpected keyword argument 'buffering' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/engine/cursor.py", line 977, in fetchall rows = dbapi_cursor.fetchall() sqlite3.InterfaceError: Cursor needed to be reset because of commit/rollback and can no longer be fetched from. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/nova/nova/api/openstack/wsgi.py", line 664, in wrapped return f(*args, **kwargs) File "/home/zuul/src/opendev.org/openstack/nova/nova/api/validation/__init__.py", line 110, in wrapper return func(*args, **kwargs) File "/home/zuul/src/opendev.org/openstack/nova/nova/api/openstack/compute/servers.py", line 1139, in _action_resize self._resize(req, id, flavor_ref, **kwargs) File "/home/zuul/src/opendev.org/openstack/nova/nova/api/openstack/compute/servers.py", line 1053, in _resize self.compute_api.resize(context, instance, flavor_id, File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/api.py", line 388, in inner return function(self, context, instance, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/api.py", line 373, in wrapper return func(self, context, instance, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/api.py", line 356, in wrapper return func(self, context, instance, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/api.py", line 241, in inner return function(self, context, instance, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/api.py", line 167, in inner return f(self, context, instance, *args, **kw) File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/api.py", line 231, in wrapped return function(self, context, instance, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/api.py", line 4244, in resize volume_backed = compute_utils.is_volume_backed_instance( File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/utils.py", line 270, in is_volume_backed_instance root_bdm = get_root_bdm(context, instance, bdms) File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/utils.py", line 263, in get_root_bdm bdms = objects.BlockDeviceMappingList.get_by_instance_uuid( File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper result = fn(cls, context, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/nova/nova/objects/block_device.py", line 420, in get_by_instance_uuid db_bdms = cls._db_block_device_mapping_get_all_by_instance( File "/home/zuul/src/opendev.org/openstack/nova/nova/db/main/api.py", line 179, in wrapper return f(*args, **kwargs) File "/home/zuul/src/opendev.org/openstack/nova/nova/objects/block_device.py", line 415, in _db_block_device_mapping_get_all_by_instance return db.block_device_mapping_get_all_by_instance( File "/home/zuul/src/opendev.org/openstack/nova/nova/db/utils.py", line 35, in wrapper return f(*args, **kwargs) File "/home/zuul/src/opendev.org/openstack/nova/nova/db/main/api.py", line 241, in wrapper return f(context, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/nova/nova/db/main/api.py", line 2931, in block_device_mapping_get_all_by_instance all() File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 2772, in all return self._iter().all() File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 1430, in all return self._allrows() File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 401, in _allrows rows = self._fetchall_impl() File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 1343, in _fetchall_impl return self._real_result._fetchall_impl() File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 1755, in _fetchall_impl return list(self.iterator) File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/orm/loading.py", line 147, in chunks fetch = cursor._raw_all_rows() File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 392, in _raw_all_rows rows = self._fetchall_impl() File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/engine/cursor.py", line 1805, in _fetchall_impl return self.cursor_strategy.fetchall(self, self.cursor) File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/engine/cursor.py", line 981, in fetchall self.handle_exception(result, dbapi_cursor, e) File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/engine/cursor.py", line 941, in handle_exception result.connection._handle_dbapi_exception( File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception util.raise_(newraise, with_traceback=exc_info[2], from_=e) File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_ raise exception File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/engine/cursor.py", line 977, in fetchall rows = dbapi_cursor.fetchall() oslo_db.exception.DBError: (sqlite3.InterfaceError) Cursor needed to be reset because of commit/rollback and can no longer be fetched from. (Background on this error at: https://sqlalche.me/e/14/rvf5) 2023-01-13 03:17:37,059 INFO [nova.api.openstack.wsgi] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'oslo_db.exception.DBError'> 2023-01-13 03:17:37,062 INFO [nova.api.openstack.requestlog] 127.0.0.1 "POST /v2.1/servers/ee5262e1-ddd0-44f4-a9b6-3d3abc1c44ad/action" status: 500 len: 199 microversion: 2.15 time: 0.091336 }}} stderr: {{{ /home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py:541: DeprecationWarning: Using the 'mysql_enable_ndb' argument is deprecated in version '12.1.0': Support for the MySQL NDB Cluster storage engine has been deprecated and will be removed in a future release. engine = engines.create_engine( }}} Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/nova/nova/tests/functional/test_server_group.py", line 406, in test_resize_to_same_host_with_anti_affinity self.admin_api.post_server_action(servers[1]['id'], post) File "/home/zuul/src/opendev.org/openstack/nova/nova/tests/functional/api/client.py", line 273, in post_server_action return self.api_post( File "/home/zuul/src/opendev.org/openstack/nova/nova/tests/functional/api/client.py", line 215, in api_post return APIResponse(self.api_request(relative_uri, **kwargs)) File "/home/zuul/src/opendev.org/openstack/nova/nova/tests/functional/api/client.py", line 191, in api_request raise OpenStackApiException( nova.tests.functional.api.client.OpenStackApiException: Unexpected status code: {"computeFault": {"code": 500, "message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'oslo_db.exception.DBError'>"}} Note that it is not test case specific and seen in multiple functional test cases. Example zuul run: https://zuul.opendev.org/t/openstack/build/c942764f8f3f4d5fb8bf847795875c91/logs It seems to be fairly new failure as I only found 4 hits in the last 20 days and all is since 01-10: $ logsearch log --job-group nova-functional --result FAILURE 'Cursor needed to be reset because of commit/rollback and can no longer be fetched from.' --days 20 [..snip..] Builds with matching logs 4/115: +----------------------------------+---------------------+----------------+----------+-----------------------------------+--------+---------------------------+ | uuid | finished | project | pipeline | review | branch | job | +----------------------------------+---------------------+----------------+----------+-----------------------------------+--------+---------------------------+ | c942764f8f3f4d5fb8bf847795875c91 | 2023-01-13T03:25:13 | openstack/nova | check | https://review.opendev.org/855649 | master | nova-tox-functional-py310 | | e57c4a0052ab43e2943c1dcffd45681f | 2023-01-13T03:30:30 | openstack/nova | check | https://review.opendev.org/855654 | master | nova-tox-functional-py39 | | 42e7e146e6174b2e8fd0949ab671a38a | 2023-01-12T21:48:15 | openstack/nova | check | https://review.opendev.org/863918 | master | nova-tox-functional-py38 | | 71abaf4c3cce47148976a4c2055dc69a | 2023-01-10T19:53:20 | openstack/nova | check | https://review.opendev.org/863915 | master | nova-tox-functional-py39 | +----------------------------------+---------------------+----------------+----------+-----------------------------------+--------+---------------------------+ ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2002782 Title: nova functional test intermittently fail with DB error: Cursor needed to be reset because of commit/rollback and can no longer be fetched from Status in OpenStack Compute (nova): New Bug description: Example stack trace: 2023-01-13 03:17:37,055 WARNING [oslo_db.sqlalchemy.exc_filters] DBAPIError exception wrapped. Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/urllib3/connectionpool.py", line 440, in _make_request httplib_response = conn.getresponse(buffering=True) TypeError: HTTPConnection.getresponse() got an unexpected keyword argument 'buffering' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/engine/cursor.py", line 977, in fetchall rows = dbapi_cursor.fetchall() sqlite3.InterfaceError: Cursor needed to be reset because of commit/rollback and can no longer be fetched from. 2023-01-13 03:17:37,057 ERROR [nova.api.openstack.wsgi] Unexpected exception in API method Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/urllib3/connectionpool.py", line 440, in _make_request httplib_response = conn.getresponse(buffering=True) TypeError: HTTPConnection.getresponse() got an unexpected keyword argument 'buffering' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/engine/cursor.py", line 977, in fetchall rows = dbapi_cursor.fetchall() sqlite3.InterfaceError: Cursor needed to be reset because of commit/rollback and can no longer be fetched from. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/nova/nova/api/openstack/wsgi.py", line 664, in wrapped return f(*args, **kwargs) File "/home/zuul/src/opendev.org/openstack/nova/nova/api/validation/__init__.py", line 110, in wrapper return func(*args, **kwargs) File "/home/zuul/src/opendev.org/openstack/nova/nova/api/openstack/compute/servers.py", line 1139, in _action_resize self._resize(req, id, flavor_ref, **kwargs) File "/home/zuul/src/opendev.org/openstack/nova/nova/api/openstack/compute/servers.py", line 1053, in _resize self.compute_api.resize(context, instance, flavor_id, File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/api.py", line 388, in inner return function(self, context, instance, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/api.py", line 373, in wrapper return func(self, context, instance, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/api.py", line 356, in wrapper return func(self, context, instance, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/api.py", line 241, in inner return function(self, context, instance, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/api.py", line 167, in inner return f(self, context, instance, *args, **kw) File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/api.py", line 231, in wrapped return function(self, context, instance, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/api.py", line 4244, in resize volume_backed = compute_utils.is_volume_backed_instance( File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/utils.py", line 270, in is_volume_backed_instance root_bdm = get_root_bdm(context, instance, bdms) File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/utils.py", line 263, in get_root_bdm bdms = objects.BlockDeviceMappingList.get_by_instance_uuid( File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper result = fn(cls, context, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/nova/nova/objects/block_device.py", line 420, in get_by_instance_uuid db_bdms = cls._db_block_device_mapping_get_all_by_instance( File "/home/zuul/src/opendev.org/openstack/nova/nova/db/main/api.py", line 179, in wrapper return f(*args, **kwargs) File "/home/zuul/src/opendev.org/openstack/nova/nova/objects/block_device.py", line 415, in _db_block_device_mapping_get_all_by_instance return db.block_device_mapping_get_all_by_instance( File "/home/zuul/src/opendev.org/openstack/nova/nova/db/utils.py", line 35, in wrapper return f(*args, **kwargs) File "/home/zuul/src/opendev.org/openstack/nova/nova/db/main/api.py", line 241, in wrapper return f(context, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/nova/nova/db/main/api.py", line 2931, in block_device_mapping_get_all_by_instance all() File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 2772, in all return self._iter().all() File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 1430, in all return self._allrows() File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 401, in _allrows rows = self._fetchall_impl() File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 1343, in _fetchall_impl return self._real_result._fetchall_impl() File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 1755, in _fetchall_impl return list(self.iterator) File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/orm/loading.py", line 147, in chunks fetch = cursor._raw_all_rows() File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 392, in _raw_all_rows rows = self._fetchall_impl() File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/engine/cursor.py", line 1805, in _fetchall_impl return self.cursor_strategy.fetchall(self, self.cursor) File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/engine/cursor.py", line 981, in fetchall self.handle_exception(result, dbapi_cursor, e) File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/engine/cursor.py", line 941, in handle_exception result.connection._handle_dbapi_exception( File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception util.raise_(newraise, with_traceback=exc_info[2], from_=e) File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_ raise exception File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/sqlalchemy/engine/cursor.py", line 977, in fetchall rows = dbapi_cursor.fetchall() oslo_db.exception.DBError: (sqlite3.InterfaceError) Cursor needed to be reset because of commit/rollback and can no longer be fetched from. (Background on this error at: https://sqlalche.me/e/14/rvf5) 2023-01-13 03:17:37,059 INFO [nova.api.openstack.wsgi] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'oslo_db.exception.DBError'> 2023-01-13 03:17:37,062 INFO [nova.api.openstack.requestlog] 127.0.0.1 "POST /v2.1/servers/ee5262e1-ddd0-44f4-a9b6-3d3abc1c44ad/action" status: 500 len: 199 microversion: 2.15 time: 0.091336 }}} stderr: {{{ /home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py:541: DeprecationWarning: Using the 'mysql_enable_ndb' argument is deprecated in version '12.1.0': Support for the MySQL NDB Cluster storage engine has been deprecated and will be removed in a future release. engine = engines.create_engine( }}} Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/nova/nova/tests/functional/test_server_group.py", line 406, in test_resize_to_same_host_with_anti_affinity self.admin_api.post_server_action(servers[1]['id'], post) File "/home/zuul/src/opendev.org/openstack/nova/nova/tests/functional/api/client.py", line 273, in post_server_action return self.api_post( File "/home/zuul/src/opendev.org/openstack/nova/nova/tests/functional/api/client.py", line 215, in api_post return APIResponse(self.api_request(relative_uri, **kwargs)) File "/home/zuul/src/opendev.org/openstack/nova/nova/tests/functional/api/client.py", line 191, in api_request raise OpenStackApiException( nova.tests.functional.api.client.OpenStackApiException: Unexpected status code: {"computeFault": {"code": 500, "message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'oslo_db.exception.DBError'>"}} Note that it is not test case specific and seen in multiple functional test cases. Example zuul run: https://zuul.opendev.org/t/openstack/build/c942764f8f3f4d5fb8bf847795875c91/logs It seems to be fairly new failure as I only found 4 hits in the last 20 days and all is since 01-10: $ logsearch log --job-group nova-functional --result FAILURE 'Cursor needed to be reset because of commit/rollback and can no longer be fetched from.' --days 20 [..snip..] Builds with matching logs 4/115: +----------------------------------+---------------------+----------------+----------+-----------------------------------+--------+---------------------------+ | uuid | finished | project | pipeline | review | branch | job | +----------------------------------+---------------------+----------------+----------+-----------------------------------+--------+---------------------------+ | c942764f8f3f4d5fb8bf847795875c91 | 2023-01-13T03:25:13 | openstack/nova | check | https://review.opendev.org/855649 | master | nova-tox-functional-py310 | | e57c4a0052ab43e2943c1dcffd45681f | 2023-01-13T03:30:30 | openstack/nova | check | https://review.opendev.org/855654 | master | nova-tox-functional-py39 | | 42e7e146e6174b2e8fd0949ab671a38a | 2023-01-12T21:48:15 | openstack/nova | check | https://review.opendev.org/863918 | master | nova-tox-functional-py38 | | 71abaf4c3cce47148976a4c2055dc69a | 2023-01-10T19:53:20 | openstack/nova | check | https://review.opendev.org/863915 | master | nova-tox-functional-py39 | +----------------------------------+---------------------+----------------+----------+-----------------------------------+--------+---------------------------+ To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2002782/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp