Public bug reported:
Our environment has a high amount of ephemeral instances (spawn -> do a
task -> deleted) therefore to keep the Nova DB clean we run periodically
"nova-manage db archive_deleted_rows" to archive and purge old DB
entries, even with a small "--max_rows" it causes occasional DB
deadlocks and leave us with instances stuck forever with BUILD status
and "scheduling" task state.
The nova-conductor error related to such instances:
{'code': 500, 'created': '2025-05-15T05:10:15Z', 'message': 'RemoteError',
'details': 'Traceback (most recent call last):\n File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line
207, in decorated_function\n return function(self, context, *args,
**kwargs)\n File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line
2145, in _do_build_and_run_instance\n (task_states.SCHEDULING, None))\n
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_versionedobjects/base.py",
line 210, in wrapper\n ctxt, self, fn.__name__, args, kwargs)\n File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/conductor/rpcapi.py",
line 246, in object_action\n objmethod=objmethod, args=args,
kwargs=kwargs)\n File
"/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/client.py",
line 181, in call\n transport_options=self.transport_options)\n File
"/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/trans
port.py", line 129, in _send\n transport_options=transport_options)\n File
"/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 674, in send\n transport_options=transport_options)\n File
"/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 664, in _send\n raise result\noslo_messaging.rpc.client.RemoteError:
Remote error: DBDeadlock (pymysql.err.OperationalError) (1213, \'Deadlock found
when trying to get lock; try restarting transaction\')\n[SQL: UPDATE
instance_extra SET updated_at=%(updated_at)s, numa_topology=%(numa_topology)s,
pci_requests=%(pci_requests)s, flavor=%(flavor)s, vcpu_model=%(vcpu_model)s
WHERE instance_extra.deleted = %(deleted_1)s AND instance_extra.instance_uuid =
%(instance_uuid_1)s]\n[parameters: {\'updated_at\': datetime.datetime(2025, 5,
15, 5, 10, 15, 769294), \'numa_topology\': \'{"nova_object.name":
"InstanceNUMATopology", "nova_object.namespac
e": "nova", "nova_object.version": "1.3", "nova_object.data": {"cells":
[{"nova_obj ... (731 characters truncated) ... set_reserved", "id",
"cpu_pinning_raw", "pagesize"]}], "emulator_threads_policy": null},
"nova_object.changes": ["cells", "emulator_threads_policy"]}\',
\'pci_requests\': \'[{"count": 1, "spec": [{"vendor_id": "10de", "product_id":
"1e07", "dev_type": "type-PCI"}], "alias_name": "RTX_2080ti_revA", "is_new":
false, "numa_ ... (538 characters truncated) ... "1ad7", "dev_type":
"type-PCI"}], "alias_name": "RTX_2080tiex2", "is_new": false, "numa_policy":
"legacy", "request_id": null, "requester_id": null}]\', \'flavor\': \'{"cur":
{"nova_object.name": "Flavor", "nova_object.namespace": "nova",
"nova_object.version": "1.2", "nova_object.data": {"id": 651, "name": "ux4-g1
... (547 characters truncated) ... "2025-02-18T20:01:16Z", "updated_at": null,
"deleted_at": null, "deleted": false}, "nova_object.changes": ["extra_specs"]},
"old": null, "new": null}\', \'vcpu_model
\': \'{"nova_object.name": "VirtCPUModel", "nova_object.namespace": "nova",
"nova_object.version": "1.0", "nova_object.data": {"arch": null, "vendor": null
... (462 characters truncated) ... ": "host-passthrough", "model": null,
"match": "exact"}, "nova_object.changes": ["features", "topology", "match",
"model", "mode", "arch", "vendor"]}\', \'deleted_1\': 0, \'instance_uuid_1\':
\'42288ebb-279c-483e-ad00-993ed453f823\'}]\n(Background on this error at:
http://sqlalche.me/e/e3q8)\n[\'Traceback (most recent call last):\\n\', \'
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py",
line 1249, in _execute_context\\n cursor, statement, parameters,
context\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py",
line 552, in do_execute\\n cursor.execute(statement, parameters)\\n\', \'
File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/cursors.py",
line 170, in execute\\n result = self._query(query)\\n
\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 328,
in _query\\n conn.query(q)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line
517, in query\\n self._affected_rows =
self._read_query_result(unbuffered=unbuffered)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line
732, in _read_query_result\\n result.read()\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line
1075, in read\\n first_packet = self.connection._read_packet()\\n\', \'
File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py",
line 684, in _read_packet\\n packet.check_error()\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/protocol.py", line
220, in check_error\\n err.raise_mysql_exception(self._data)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/err.py", line 109, in
raise
_mysql_exception\\n raise errorclass(errno, errval)\\n\',
"pymysql.err.OperationalError: (1213, \'Deadlock found when trying to get lock;
try restarting transaction\')\\n", \'\\nThe above exception was the direct
cause of the following exception:\\n\\n\', \'Traceback (most recent call
last):\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/conductor/manager.py",
line 135, in _object_dispatch\\n return getattr(target, method)(*args,
**kwargs)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_versionedobjects/base.py",
line 226, in wrapper\\n return fn(self, *args, **kwargs)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py",
line 801, in save\\n self._extra_values_to_save)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/api.py", line 979, in
instance_extra_update_by_uuid\\n updates)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py"
, line 224, in wrapped\\n return f(context, *args, **kwargs)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py",
line 3072, in instance_extra_update_by_uuid\\n update(values)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line
3835, in update\\n update_op.exec_()\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py",
line 1694, in exec_\\n self._do_exec()\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-
packages/sqlalchemy/orm/persistence.py", line 1887, in _do_exec\\n
self._execute_stmt(update_stmt)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py",
line 1699, in _execute_stmt\\n self.result =
self.query._execute_crud(stmt, self.mapper)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line
3367, in _execute_crud\\n return conn.execute(st
mt, self._params)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py",
line 988, in execute\\n return meth(self, multiparams, params)\\n\', \'
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/sql/elements.py",
line 287, in _execute_on_connection\\n return
connection._execute_clauseelement(self, multiparams, params)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py",
line 1107, in _execute_clauseelement\\n distilled_params,\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py",
line 1253, in _execute_context\\n e, statement, parameters, cursor,
context\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py",
line 1471, in _handle_dbapi_exception\\n util.raise_from_cause(newraise,
exc_info)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py",
line 398, in raise_fr
om_cause\\n reraise(type(exception), exception, tb=exc_tb,
cause=cause)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py",
line 152, in reraise\\n raise value.with_traceback(tb)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site- packages/sqlalchemy/engine/base.py",
line 1249, in _execute_context\\n cursor, statement, parameters,
context\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py",
line 552, in do_execute\\n cursor.execute(statement, parameters)\\n\', \'
File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/cursors.py", line
170, in execute\\n result = self._query(query)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 328,
in _query\\n conn.query(q)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line
517, in query\\n self._affected_rows = self._read_query_result(unbuffered=un
buffered)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line
732, in _read_query_result\\n result.read()\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line
1075, in read\\n first_packet = self.connection._read_packet()\\n\', \'
File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py",
line 684, in _read_packet\\n packet.check_error()\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/protocol.py", line
220, in check_error\\n err.raise_mysql_exception(self._data)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/err.py", line 109, in
raise_mysql_exception\\n raise errorclass(errno,
errval)\\n\',\'oslo_db.exception.DBDeadlock: (pymysql.err.OperationalError)
(1213, \\\'Deadlock found when trying to get lock; try restarting
transaction\\\')\\n[SQL: UPDATE instance_extra SET updated_at=%(updated_at)s,
numa_topology=%(
numa_topology)s, pci_requests=%(pci_requests)s, flavor=%(flavor)s,
vcpu_model=%(vcpu_model)s WHERE instance_extra.deleted = %(deleted_1)s AND
instance_extra.instance_uuid = %(instance_uuid_1)s]\\n[parameters:
{\\\'updated_at\\\': datetime.datetime(2025, 5, 15, 5, 10, 15, 769294),
\\\'numa_topology\\\': \\\'{"nova_object.name": "InstanceNUMATopology",
"nova_object.namespace": "nova", "nova_object.version": "1.3",
"nova_object.data": {"cells": [{"nova_obj ... (731 characters truncated) ...
set_reserved", "id", "cpu_pinning_raw", "pagesize"]}],
"emulator_threads_policy": null}, "nova_object.changes": ["cells",
"emulator_threads_policy"]}\\\', \\\'pci_requests\\\': \\\'[{"count": 1,
"spec": [{"vendor_id": "10de", "product_id": "1e07", "dev_type": "type-PCI"}],
"alias_name": "RTX_2080ti_revA", "is_new": false, "numa_ ... (538 characters
truncated) ... "1ad7", "dev_type": "type-PCI"}], "alias_name":
"RTX_2080tiex2", "is_new": false, "numa_policy": "legacy", "request_id": null,
"requeste
r_id": null}]\\\', \\\'flavor\\\': \\\'{"cur": {"nova_object.name": "Flavor",
"nova_object.namespace": "nova", "nova_object.version": "1.2",
"nova_object.data": {"id": 651, "name": "ux4-g1 ... (547 characters truncated)
... "2025-02-18T20:01:16Z", "updated_at": null, "deleted_at": null, "deleted":
false}, "nova_object.changes": ["extra_specs"]}, "old": null, "new":
null}\\\', \\\'vcpu_model\\\': \\\'{"nova_object.name": "VirtCPUModel",
"nova_object.namespace": "nova", "nova_object.version": "1.0",
"nova_object.data": {"arch": null, "vendor": null ... (462 characters
truncated) ... ": "host-passthrough", "model": null, "match": "exact"},
"nova_object.changes": ["features", "topology", "match", "model", "mode",
"arch", "vendor"]}\\\', \\\'deleted_1\\\': 0, \\\'instance_uuid_1\\\':
\\\'42288ebb-279c-483e-ad00-993ed453f823\\\'}]\\n(Background on this error at:
http://sqlalche.me/e/e3q8)\\n\'].\n'}
ENGINE INNODB STATUS:
------------------------
LATEST DETECTED DEADLOCK
------------------------
2025-05-15 13:02:46 0x7fda0b6c7700
*** (1) TRANSACTION:
TRANSACTION 33097244744, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 1136, 2 row lock(s)
MySQL thread id 56380264, OS thread handle 140573857470208, query id 3909877498
10.9.64.13 nova_v1 Updating
UPDATE instance_extra SET updated_at='2025-05-15 13:02:46.460570',
numa_topology='{\"nova_object.name\": \"InstanceNUMATopology\",
\"nova_object.namespace\": \"nova\", \"nova_object.version\": \"1.3\",
\"nova_object.data\": {\"cells\": [{\"nova
_object.name\": \"InstanceNUMACell\", \"nova_object.namespace\": \"nova\",
\"nova_object.version\": \"1.4\", \"nova_object.data\": {\"id\": 0, \"cpuset\":
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21,
22, 23, 24
, 25, 26, 27, 28, 29, 30, 31], \"memory\": 122880, \"pagesize\": null,
\"cpu_topology\": {\"nova_object.name\": \"VirtCPUTopology\",
\"nova_object.namespace\": \"nova\", \"nova_object.version\": \"1.0\",
\"nova_object.data\": {\"sockets\": 1, \
"cores\": 16, \"threads\": 2}, \"nova_object.changes\": [\"cores\",
\"sockets\", \"threads\"]}, \"cpu_pinning_raw\": {\"0\": 40, \"1\": 88, \"2\":
18, \"3\": 66, \"4\": 42, \"5\": 90,
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1178 page no 51132 n bits 72 index PRIMARY of table
`nova`.`instance_extra` trx id 33097244744 lock_mode X locks rec but not gap
waiting
Record lock, heap no 2 PHYSICAL RECORD: n_fields 18; compact format; info bits 0
...
*** (2) TRANSACTION:
TRANSACTION 33097244739, ACTIVE 0 sec fetching rows
mysql tables in use 3, locked 3
4450 lock struct(s), heap size 532688, 11991 row lock(s)
MySQL thread id 56380887, OS thread handle 140574471255808, query id 3909877431
10.9.64.13 nova_v1 Sending data
DELETE FROM instance_extra WHERE instance_extra.id in (SELECT T1.id FROM
(SELECT instance_extra.created_at, instance_extra.updated_at,
instance_extra.deleted_at, instance_extra.deleted, instance_extra.id,
instance_extra.instance_uuid, instance_
extra.device_metadata, instance_extra.numa_topology,
instance_extra.pci_requests, instance_extra.flavor, instance_extra.vcpu_model,
instance_extra.migration_context, instance_extra.keypairs,
instance_extra.trusted_certs, instance_extra.resource
s
FROM instance_extra, instances
WHERE instances.deleted != 0 AND instances.uuid = instance_extra.instance_uuid
ORDER BY instance_extra.id
LIMIT 200) as T1)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 1178 page no 51132 n bits 72 index PRIMARY of table
`nova`.`instance_extra` trx id 33097244739 lock mode S
The deadlocks happen in nova.instance_extra and triggered by colliding
"nova-manage db archive_deleted_rows" and "instance_extra_update_by_uuid"
function from nova.db.main.api which does not have any retry logic implemented
** Affects: nova
Importance: Undecided
Assignee: Stanislav Dmitriev (sdmitriev1)
Status: New
** Changed in: nova
Assignee: (unassigned) => Stanislav Dmitriev (sdmitriev1)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2111108
Title:
DBDeadlock in nova.instance_extra
Status in OpenStack Compute (nova):
New
Bug description:
Our environment has a high amount of ephemeral instances (spawn -> do
a task -> deleted) therefore to keep the Nova DB clean we run
periodically "nova-manage db archive_deleted_rows" to archive and
purge old DB entries, even with a small "--max_rows" it causes
occasional DB deadlocks and leave us with instances stuck forever with
BUILD status and "scheduling" task state.
The nova-conductor error related to such instances:
{'code': 500, 'created': '2025-05-15T05:10:15Z', 'message': 'RemoteError',
'details': 'Traceback (most recent call last):\n File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line
207, in decorated_function\n return function(self, context, *args,
**kwargs)\n File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line
2145, in _do_build_and_run_instance\n (task_states.SCHEDULING, None))\n
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_versionedobjects/base.py",
line 210, in wrapper\n ctxt, self, fn.__name__, args, kwargs)\n File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/conductor/rpcapi.py",
line 246, in object_action\n objmethod=objmethod, args=args,
kwargs=kwargs)\n File
"/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/client.py",
line 181, in call\n transport_options=self.transport_options)\n File
"/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/tra
nsport.py", line 129, in _send\n transport_options=transport_options)\n
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 674, in send\n transport_options=transport_options)\n File
"/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 664, in _send\n raise result\noslo_messaging.rpc.client.RemoteError:
Remote error: DBDeadlock (pymysql.err.OperationalError) (1213, \'Deadlock found
when trying to get lock; try restarting transaction\')\n[SQL: UPDATE
instance_extra SET updated_at=%(updated_at)s, numa_topology=%(numa_topology)s,
pci_requests=%(pci_requests)s, flavor=%(flavor)s, vcpu_model=%(vcpu_model)s
WHERE instance_extra.deleted = %(deleted_1)s AND instance_extra.instance_uuid =
%(instance_uuid_1)s]\n[parameters: {\'updated_at\': datetime.datetime(2025, 5,
15, 5, 10, 15, 769294), \'numa_topology\': \'{"nova_object.name":
"InstanceNUMATopology", "nova_object.namesp
ace": "nova", "nova_object.version": "1.3", "nova_object.data": {"cells":
[{"nova_obj ... (731 characters truncated) ... set_reserved", "id",
"cpu_pinning_raw", "pagesize"]}], "emulator_threads_policy": null},
"nova_object.changes": ["cells", "emulator_threads_policy"]}\',
\'pci_requests\': \'[{"count": 1, "spec": [{"vendor_id": "10de", "product_id":
"1e07", "dev_type": "type-PCI"}], "alias_name": "RTX_2080ti_revA", "is_new":
false, "numa_ ... (538 characters truncated) ... "1ad7", "dev_type":
"type-PCI"}], "alias_name": "RTX_2080tiex2", "is_new": false, "numa_policy":
"legacy", "request_id": null, "requester_id": null}]\', \'flavor\': \'{"cur":
{"nova_object.name": "Flavor", "nova_object.namespace": "nova",
"nova_object.version": "1.2", "nova_object.data": {"id": 651, "name": "ux4-g1
... (547 characters truncated) ... "2025-02-18T20:01:16Z", "updated_at": null,
"deleted_at": null, "deleted": false}, "nova_object.changes": ["extra_specs"]},
"old": null, "new": null}\', \'vcpu_mod
el\': \'{"nova_object.name": "VirtCPUModel", "nova_object.namespace": "nova",
"nova_object.version": "1.0", "nova_object.data": {"arch": null, "vendor": null
... (462 characters truncated) ... ": "host-passthrough", "model": null,
"match": "exact"}, "nova_object.changes": ["features", "topology", "match",
"model", "mode", "arch", "vendor"]}\', \'deleted_1\': 0, \'instance_uuid_1\':
\'42288ebb-279c-483e-ad00-993ed453f823\'}]\n(Background on this error at:
http://sqlalche.me/e/e3q8)\n[\'Traceback (most recent call last):\\n\', \'
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py",
line 1249, in _execute_context\\n cursor, statement, parameters,
context\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py",
line 552, in do_execute\\n cursor.execute(statement, parameters)\\n\', \'
File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/cursors.py",
line 170, in execute\\n result = self._query(query)\
\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 328,
in _query\\n conn.query(q)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line
517, in query\\n self._affected_rows =
self._read_query_result(unbuffered=unbuffered)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line
732, in _read_query_result\\n result.read()\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line
1075, in read\\n first_packet = self.connection._read_packet()\\n\', \'
File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py",
line 684, in _read_packet\\n packet.check_error()\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/protocol.py", line
220, in check_error\\n err.raise_mysql_exception(self._data)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/err.py", line 109, in
rai
se_mysql_exception\\n raise errorclass(errno, errval)\\n\',
"pymysql.err.OperationalError: (1213, \'Deadlock found when trying to get lock;
try restarting transaction\')\\n", \'\\nThe above exception was the direct
cause of the following exception:\\n\\n\', \'Traceback (most recent call
last):\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/conductor/manager.py",
line 135, in _object_dispatch\\n return getattr(target, method)(*args,
**kwargs)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_versionedobjects/base.py",
line 226, in wrapper\\n return fn(self, *args, **kwargs)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py",
line 801, in save\\n self._extra_values_to_save)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/api.py", line 979, in
instance_extra_update_by_uuid\\n updates)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.p
y", line 224, in wrapped\\n return f(context, *args, **kwargs)\\n\', \'
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py",
line 3072, in instance_extra_update_by_uuid\\n update(values)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line
3835, in update\\n update_op.exec_()\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py",
line 1694, in exec_\\n self._do_exec()\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-
packages/sqlalchemy/orm/persistence.py", line 1887, in _do_exec\\n
self._execute_stmt(update_stmt)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py",
line 1699, in _execute_stmt\\n self.result =
self.query._execute_crud(stmt, self.mapper)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line
3367, in _execute_crud\\n return conn.execute(
stmt, self._params)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py",
line 988, in execute\\n return meth(self, multiparams, params)\\n\', \'
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/sql/elements.py",
line 287, in _execute_on_connection\\n return
connection._execute_clauseelement(self, multiparams, params)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py",
line 1107, in _execute_clauseelement\\n distilled_params,\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py",
line 1253, in _execute_context\\n e, statement, parameters, cursor,
context\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py",
line 1471, in _handle_dbapi_exception\\n util.raise_from_cause(newraise,
exc_info)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py",
line 398, in raise_
from_cause\\n reraise(type(exception), exception, tb=exc_tb,
cause=cause)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py",
line 152, in reraise\\n raise value.with_traceback(tb)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site- packages/sqlalchemy/engine/base.py",
line 1249, in _execute_context\\n cursor, statement, parameters,
context\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py",
line 552, in do_execute\\n cursor.execute(statement, parameters)\\n\', \'
File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/cursors.py", line
170, in execute\\n result = self._query(query)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 328,
in _query\\n conn.query(q)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line
517, in query\\n self._affected_rows = self._read_query_result(unbuffered=
unbuffered)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line
732, in _read_query_result\\n result.read()\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line
1075, in read\\n first_packet = self.connection._read_packet()\\n\', \'
File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py",
line 684, in _read_packet\\n packet.check_error()\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/protocol.py", line
220, in check_error\\n err.raise_mysql_exception(self._data)\\n\', \' File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/err.py", line 109, in
raise_mysql_exception\\n raise errorclass(errno,
errval)\\n\',\'oslo_db.exception.DBDeadlock: (pymysql.err.OperationalError)
(1213, \\\'Deadlock found when trying to get lock; try restarting
transaction\\\')\\n[SQL: UPDATE instance_extra SET updated_at=%(updated_at)s,
numa_topology=
%(numa_topology)s, pci_requests=%(pci_requests)s, flavor=%(flavor)s,
vcpu_model=%(vcpu_model)s WHERE instance_extra.deleted = %(deleted_1)s AND
instance_extra.instance_uuid = %(instance_uuid_1)s]\\n[parameters:
{\\\'updated_at\\\': datetime.datetime(2025, 5, 15, 5, 10, 15, 769294),
\\\'numa_topology\\\': \\\'{"nova_object.name": "InstanceNUMATopology",
"nova_object.namespace": "nova", "nova_object.version": "1.3",
"nova_object.data": {"cells": [{"nova_obj ... (731 characters truncated) ...
set_reserved", "id", "cpu_pinning_raw", "pagesize"]}],
"emulator_threads_policy": null}, "nova_object.changes": ["cells",
"emulator_threads_policy"]}\\\', \\\'pci_requests\\\': \\\'[{"count": 1,
"spec": [{"vendor_id": "10de", "product_id": "1e07", "dev_type": "type-PCI"}],
"alias_name": "RTX_2080ti_revA", "is_new": false, "numa_ ... (538 characters
truncated) ... "1ad7", "dev_type": "type-PCI"}], "alias_name":
"RTX_2080tiex2", "is_new": false, "numa_policy": "legacy", "request_id": null,
"reques
ter_id": null}]\\\', \\\'flavor\\\': \\\'{"cur": {"nova_object.name":
"Flavor", "nova_object.namespace": "nova", "nova_object.version": "1.2",
"nova_object.data": {"id": 651, "name": "ux4-g1 ... (547 characters truncated)
... "2025-02-18T20:01:16Z", "updated_at": null, "deleted_at": null, "deleted":
false}, "nova_object.changes": ["extra_specs"]}, "old": null, "new":
null}\\\', \\\'vcpu_model\\\': \\\'{"nova_object.name": "VirtCPUModel",
"nova_object.namespace": "nova", "nova_object.version": "1.0",
"nova_object.data": {"arch": null, "vendor": null ... (462 characters
truncated) ... ": "host-passthrough", "model": null, "match": "exact"},
"nova_object.changes": ["features", "topology", "match", "model", "mode",
"arch", "vendor"]}\\\', \\\'deleted_1\\\': 0, \\\'instance_uuid_1\\\':
\\\'42288ebb-279c-483e-ad00-993ed453f823\\\'}]\\n(Background on this error at:
http://sqlalche.me/e/e3q8)\\n\'].\n'}
ENGINE INNODB STATUS:
------------------------
LATEST DETECTED DEADLOCK
------------------------
2025-05-15 13:02:46 0x7fda0b6c7700
*** (1) TRANSACTION:
TRANSACTION 33097244744, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 1136, 2 row lock(s)
MySQL thread id 56380264, OS thread handle 140573857470208, query id
3909877498 10.9.64.13 nova_v1 Updating
UPDATE instance_extra SET updated_at='2025-05-15 13:02:46.460570',
numa_topology='{\"nova_object.name\": \"InstanceNUMATopology\",
\"nova_object.namespace\": \"nova\", \"nova_object.version\": \"1.3\",
\"nova_object.data\": {\"cells\": [{\"nova
_object.name\": \"InstanceNUMACell\", \"nova_object.namespace\": \"nova\",
\"nova_object.version\": \"1.4\", \"nova_object.data\": {\"id\": 0, \"cpuset\":
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21,
22, 23, 24
, 25, 26, 27, 28, 29, 30, 31], \"memory\": 122880, \"pagesize\": null,
\"cpu_topology\": {\"nova_object.name\": \"VirtCPUTopology\",
\"nova_object.namespace\": \"nova\", \"nova_object.version\": \"1.0\",
\"nova_object.data\": {\"sockets\": 1, \
"cores\": 16, \"threads\": 2}, \"nova_object.changes\": [\"cores\",
\"sockets\", \"threads\"]}, \"cpu_pinning_raw\": {\"0\": 40, \"1\": 88, \"2\":
18, \"3\": 66, \"4\": 42, \"5\": 90,
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1178 page no 51132 n bits 72 index PRIMARY of table
`nova`.`instance_extra` trx id 33097244744 lock_mode X locks rec but not gap
waiting
Record lock, heap no 2 PHYSICAL RECORD: n_fields 18; compact format; info
bits 0
...
*** (2) TRANSACTION:
TRANSACTION 33097244739, ACTIVE 0 sec fetching rows
mysql tables in use 3, locked 3
4450 lock struct(s), heap size 532688, 11991 row lock(s)
MySQL thread id 56380887, OS thread handle 140574471255808, query id
3909877431 10.9.64.13 nova_v1 Sending data
DELETE FROM instance_extra WHERE instance_extra.id in (SELECT T1.id FROM
(SELECT instance_extra.created_at, instance_extra.updated_at,
instance_extra.deleted_at, instance_extra.deleted, instance_extra.id,
instance_extra.instance_uuid, instance_
extra.device_metadata, instance_extra.numa_topology,
instance_extra.pci_requests, instance_extra.flavor, instance_extra.vcpu_model,
instance_extra.migration_context, instance_extra.keypairs,
instance_extra.trusted_certs, instance_extra.resource
s
FROM instance_extra, instances
WHERE instances.deleted != 0 AND instances.uuid =
instance_extra.instance_uuid ORDER BY instance_extra.id
LIMIT 200) as T1)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 1178 page no 51132 n bits 72 index PRIMARY of table
`nova`.`instance_extra` trx id 33097244739 lock mode S
The deadlocks happen in nova.instance_extra and triggered by colliding
"nova-manage db archive_deleted_rows" and "instance_extra_update_by_uuid"
function from nova.db.main.api which does not have any retry logic implemented
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2111108/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
