Public bug reported:
This is similar to older bug https://bugs.launchpad.net/bugs/1543937 but
in 'nova-manage db purge'. Purging a large dataset causes this failure
in Galera cluster. Mariadb log has following error when this happens:
2022-07-30 20:22:10 1567 [Warning] WSREP: transaction size limit (2147483647)
exceeded: 2147483648
2022-07-30 20:22:10 1567 [ERROR] WSREP: rbr write fail, data_len: 0, 2
This happens because the transaction is too large for mariadb to handle.
'nova-manage db archive_deleted_rows' works around this by limiting the
number of max_rows using --max_rows argument and checking it against
db.MAX_INT variable, we might have to do similar thing for purge
Traceback:
nova-manage --config-file /etc/nova/nova.conf db purge --verbose --all-cells
--all
An error has occurred:
Traceback (most recent call last):
File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/cmd/manage.py",
line 2793, in main
ret = fn(*fn_args, **fn_kwargs)
File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/cmd/manage.py",
line 454, in purge
status_fn=status)
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py",
line 4426, in purge_shadow_tables
deleted = conn.execute(delete)
File
"/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py",
line 1011, in execute
return meth(self, multiparams, params)
File
"/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/sql/elements.py",
line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File
"/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py",
line 1130, in _execute_clauseelement
distilled_params,
File
"/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py",
line 1317, in _execute_context
e, statement, parameters, cursor, context
File
"/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py",
line 1508, in _handle_dbapi_exception
util.raise_(newraise, with_traceback=exc_info[2], from_=e)
File
"/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py",
line 182, in raise_
raise exception
File
"/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py",
line 1301, in _execute_context
self._root._commit_impl(autocommit=True)
File
"/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py",
line 773, in _commit_impl
self._handle_dbapi_exception(e, None, None, None, None)
File
"/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py",
line 1508, in _handle_dbapi_exception
util.raise_(newraise, with_traceback=exc_info[2], from_=e)
File
"/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py",
line 182, in raise_
raise exception
File
"/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py",
line 771, in _commit_impl
self.engine.dialect.do_commit(self.connection)
File
"/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/dialects/mysql/base.py",
line 2463, in do_commit
dbapi_connection.commit()
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line
422, in commit
self._read_ok_packet()
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line
396, in _read_ok_packet
pkt = self._read_packet()
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line
676, in _read_packet
packet.raise_for_error()
File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/protocol.py",
line 223, in raise_for_error
err.raise_mysql_exception(self._data)
File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/err.py", line
107, in raise_mysql_exception
raise errorclass(errno, errval)
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1180, 'Got
error 90 "Message too long" during COMMIT')
(Background on this error at: http://sqlalche.me/e/13/e3q8)
MariaDB log has following error:
2022-07-30 20:22:10 1567 [Warning] WSREP: transaction size limit (2147483647)
exceeded: 2147483648
2022-07-30 20:22:10 1567 [ERROR] WSREP: rbr write fail, data_len: 0, 2
More information on how to replicate this and workaround this using mariadb is
available here
https://www.percona.com/blog/2015/10/26/how-big-can-your-galera-transactions-be/
** Affects: nova
Importance: Undecided
Status: New
** Tags: nova-manage
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1983188
Title:
nova-manage db purge fails on large datasets
Status in OpenStack Compute (nova):
New
Bug description:
This is similar to older bug https://bugs.launchpad.net/bugs/1543937
but in 'nova-manage db purge'. Purging a large dataset causes this
