Public bug reported: Jens Harbott reported this in devstack:
https://review.openstack.org/#/c/635519/ He's running the n-api-meta service in the subnode per the multinode guide: https://docs.openstack.org/devstack/latest/guides/multinode-lab.html However with current devstack, which doesn't configure nova.conf with database access on the subnode, n-api-meta fails with this: http://paste.openstack.org/show/744683/ Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: CRITICAL nova [None req-538e6b11-f91b-48d8-9c7a-d177dff7739b None None] Unhandled error: DBNonExistentTable: (sqlite3.OperationalError) no such t Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova Traceback (most recent call last): Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/bin/nova-metadata-wsgi", line 52, in <module> Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova application = init_application() Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/opt/stack/nova/nova/api/metadata/wsgi.py", line 20, in init_application Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova return wsgi_app.init_application(NAME) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/opt/stack/nova/nova/api/openstack/wsgi_app.py", line 82, in init_application Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova _setup_service(CONF.host, name) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/opt/stack/nova/nova/api/openstack/wsgi_app.py", line 49, in _setup_service Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova ctxt, host, binary) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova result = fn(cls, context, *args, **kwargs) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/opt/stack/nova/nova/objects/service.py", line 334, in get_by_host_and_binary Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova host, binary) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/opt/stack/nova/nova/db/api.py", line 127, in service_get_by_host_and_binary Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova return IMPL.service_get_by_host_and_binary(context, host, binary) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 242, in wrapped Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova return f(context, *args, **kwargs) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 500, in service_get_by_host_and_binary Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova filter_by(binary=binary).\ Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2979, in first Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova ret = list(self[0:1]) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2771, in __getitem__ Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova return list(res) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 3081, in __iter__ Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova return self._execute_and_instances(context) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 3106, in _execute_and_instances Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova result = conn.execute(querycontext.statement, self._params) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 980, in execute Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova return meth(self, multiparams, params) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova return connection._execute_clauseelement(self, multiparams, params) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova distilled_params, Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1240, in _execute_context Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova e, statement, parameters, cursor, context Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1456, in _handle_dbapi_exception Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova util.raise_from_cause(newraise, exc_info) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova reraise(type(exception), exception, tb=exc_tb, cause=cause) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova cursor, statement, parameters, context Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 536, in do_execute Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova cursor.execute(statement, parameters) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova DBNonExistentTable: (sqlite3.OperationalError) no such table: services [SQL: u'SELECT services.created_at AS services_created_at, serv Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: unable to load app 0 (mountpoint='') (callable not found or import error) I believe this is due to the nova-metadata service not configuring the conductor indirection API when run in uwsgi mode like it is when run with eventlet: https://github.com/openstack/nova/blob/3cfcd117ce7ddb5a8d94396191cde559db73fb86/nova/cmd/api_metadata.py#L45 It also appears that we don't have any multinode CI jobs that run with the n-api-meta service configured so we don't hit this in the gate. The workaround which most deployments probably use is configuring nova.conf with database access and run n-api-meta with that, even from subnodes. ** Affects: nova Importance: Medium Status: New ** Tags: api metadata -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1815082 Title: "DBNonExistentTable: (sqlite3.OperationalError) no such table: services" when starting nova-metadata under uwsgi Status in OpenStack Compute (nova): New Bug description: Jens Harbott reported this in devstack: https://review.openstack.org/#/c/635519/ He's running the n-api-meta service in the subnode per the multinode guide: https://docs.openstack.org/devstack/latest/guides/multinode-lab.html However with current devstack, which doesn't configure nova.conf with database access on the subnode, n-api-meta fails with this: http://paste.openstack.org/show/744683/ Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: CRITICAL nova [None req-538e6b11-f91b-48d8-9c7a-d177dff7739b None None] Unhandled error: DBNonExistentTable: (sqlite3.OperationalError) no such t Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova Traceback (most recent call last): Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/bin/nova-metadata-wsgi", line 52, in <module> Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova application = init_application() Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/opt/stack/nova/nova/api/metadata/wsgi.py", line 20, in init_application Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova return wsgi_app.init_application(NAME) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/opt/stack/nova/nova/api/openstack/wsgi_app.py", line 82, in init_application Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova _setup_service(CONF.host, name) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/opt/stack/nova/nova/api/openstack/wsgi_app.py", line 49, in _setup_service Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova ctxt, host, binary) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova result = fn(cls, context, *args, **kwargs) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/opt/stack/nova/nova/objects/service.py", line 334, in get_by_host_and_binary Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova host, binary) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/opt/stack/nova/nova/db/api.py", line 127, in service_get_by_host_and_binary Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova return IMPL.service_get_by_host_and_binary(context, host, binary) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 242, in wrapped Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova return f(context, *args, **kwargs) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 500, in service_get_by_host_and_binary Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova filter_by(binary=binary).\ Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2979, in first Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova ret = list(self[0:1]) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2771, in __getitem__ Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova return list(res) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 3081, in __iter__ Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova return self._execute_and_instances(context) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 3106, in _execute_and_instances Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova result = conn.execute(querycontext.statement, self._params) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 980, in execute Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova return meth(self, multiparams, params) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova return connection._execute_clauseelement(self, multiparams, params) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova distilled_params, Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1240, in _execute_context Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova e, statement, parameters, cursor, context Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1456, in _handle_dbapi_exception Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova util.raise_from_cause(newraise, exc_info) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova reraise(type(exception), exception, tb=exc_tb, cause=cause) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova cursor, statement, parameters, context Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 536, in do_execute Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova cursor.execute(statement, parameters) Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova DBNonExistentTable: (sqlite3.OperationalError) no such table: services [SQL: u'SELECT services.created_at AS services_created_at, serv Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: unable to load app 0 (mountpoint='') (callable not found or import error) I believe this is due to the nova-metadata service not configuring the conductor indirection API when run in uwsgi mode like it is when run with eventlet: https://github.com/openstack/nova/blob/3cfcd117ce7ddb5a8d94396191cde559db73fb86/nova/cmd/api_metadata.py#L45 It also appears that we don't have any multinode CI jobs that run with the n-api-meta service configured so we don't hit this in the gate. The workaround which most deployments probably use is configuring nova.conf with database access and run n-api-meta with that, even from subnodes. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1815082/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
