** Tags added: oslo ** Also affects: oslo.service Importance: Undecided Status: New
-- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1589880 Title: report state failed Status in OpenStack Compute (nova): Incomplete Status in oslo.service: New Bug description: Description: ================= set master database read_only=on when switching master nova database to slave,after that,I check nova service status # nova-manage service list Binary Host Zone Status State Updated_At nova-consoleauth 11_120 internal enabled XXX 2016-06-07 08:28:46 nova-conductor 11_120 internal enabled XXX 2016-06-07 08:28:45 nova-cert 11_120 internal enabled XXX 2016-05-17 08:12:10 nova-scheduler 11_120 internal enabled XXX 2016-05-17 08:12:24 nova-compute 11_121 bx enabled XXX 2016-06-07 08:28:49 nova-compute 11_122 bx enabled XXX 2016-06-07 08:28:42 ================= Steps to reproduce ================= # mysql MariaDB [nova]> set global read_only=on; ================= Environment ================ Version:Liberty openstack-nova-conductor-12.0.0-1.el7.noarch Logs ================ 2016-05-12 11:01:20.343 9198 ERROR oslo.service.loopingcall 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall [-] Fixed interval looping call 'nova.servicegroup.drivers.db.DbDriver._report_state' failed 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall Traceback (most recent call last): 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 113, in _run_loop 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall result = func(*self.args, **self.kw) 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 87, in _report_state 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall service.service_ref.save() 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 213, in wrapper 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall return fn(self, *args, **kwargs) 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 250, in save 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall db_service = db.service_update(self._context, self.id, updates) 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 153, in service_update 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall return IMPL.service_update(context, service_id, values) 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 146, in wrapper 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall ectxt.value = e.inner_exc 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__ 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall six.reraise(self.type_, self.value, self.tb) 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 136, in wrapper 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall return f(*args, **kwargs) 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 532, in service_update 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall service_ref.update(values) 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 490, in __exit__ 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall self.rollback() 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__ 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall compat.reraise(exc_type, exc_value, exc_tb) 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 487, in __exit__ 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall self.commit() 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 392, in commit 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall self._prepare_impl() 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 372, in _prepare_impl 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall self.session.flush() 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2004, in flush 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall self._flush(objects) 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2122, in _flush 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall transaction.rollback(_capture_exception=True) 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__ 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall compat.reraise(exc_type, exc_value, exc_tb) 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2086, in _flush 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall flush_context.execute() 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall rec.execute(self) 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall uow 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 170, in save_obj 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall mapper, table, update) 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 672, in _emit_update_statements 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall execute(statement, multiparams) 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall return meth(self, multiparams, params) 2016-05-12 11:01:20.473 9178 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection 2016-05-12 11:01:17.575 9177 ERROR oslo.service.loopingcall return connection._execute_clauseelement(self, multiparams, params) 2016-05-12 11:01:17.575 9177 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement 2016-05-12 11:01:17.575 9177 ERROR oslo.service.loopingcall compiled_sql, distilled_params 2016-05-12 11:01:17.575 9177 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context 2016-05-12 11:01:17.575 9177 ERROR oslo.service.loopingcall context) 2016-05-12 11:01:17.575 9177 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1335, in _handle_dbapi_exception 2016-05-12 11:01:17.575 9177 ERROR oslo.service.loopingcall util.raise_from_cause(newraise, exc_info) 2016-05-12 11:01:17.575 9177 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause 2016-05-12 11:01:17.575 9177 ERROR oslo.service.loopingcall reraise(type(exception), exception, tb=exc_tb) 2016-05-12 11:01:17.575 9177 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context 2016-05-12 11:01:17.575 9177 ERROR oslo.service.loopingcall context) 2016-05-12 11:01:17.575 9177 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 442, in do_execute 2016-05-12 11:01:17.575 9177 ERROR oslo.service.loopingcall cursor.execute(statement, parameters) 2016-05-12 11:01:17.575 9177 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute 2016-05-12 11:01:17.575 9177 ERROR oslo.service.loopingcall self.errorhandler(self, exc, value) 2016-05-12 11:01:17.575 9177 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler 2016-05-12 11:01:17.575 9177 ERROR oslo.service.loopingcall raise errorclass, errorvalue 2016-05-12 11:01:17.575 9177 ERROR oslo.service.loopingcall OperationalError: (_mysql_exceptions.OperationalError) (1290, 'The MariaDB server is running with the --read-only option so it cannot execute this statement') [SQL: u'UPDATE services SET updated_at=%s, report_count=%s, last_seen_up=%s WHERE services.id = %s'] [parameters: (datetime.datetime(2016, 5, 12, 3, 1, 17, 572767), 1189606, datetime.datetime(2016, 5, 12, 3, 1, 17, 571934), 2L)] To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1589880/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp