Thanks for the feedback.
I'll invalidate this ticket, afaics this does no longer turn up so probably
DevStack or Tempest devs already improved the default env.
** Changed in: nova
Status: Incomplete => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1658090
Title:
Nova API reports ProgrammingError in mysql layer
Status in OpenStack Compute (nova):
Invalid
Bug description:
Nova API sometimes reports a ProgrammingError in Tempests
TestMinimumBasicScenario in our CI:
2017-01-20 08:56:41.246 10377 DEBUG oslo_db.sqlalchemy.engines
[req-7b026f0c-4071-4425-85be-e98d8ddc2d33
tempest-TestMinimumBasicScenario-878230226
tempest-TestMinimumBasicScenario-878230226] MySQL server mode set to
STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
_check_effective_sql_mode
/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:261
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions
[req-7b026f0c-4071-4425-85be-e98d8ddc2d33
tempest-TestMinimumBasicScenario-878230226
tempest-TestMinimumBasicScenario-878230226] Unexpected exception in API method
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions Traceback
(most recent call last):
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/api/openstack/extensions.py", line 338, in wrapped
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions return
f(*args, **kwargs)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/api/openstack/compute/servers.py", line 196, in index
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions servers
= self._get_servers(req, is_detail=False)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/api/openstack/compute/servers.py", line 347, in
_get_servers
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions
sort_keys=sort_keys, sort_dirs=sort_dirs)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/compute/api.py", line 2407, in get_all
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions
sort_dirs=sort_dirs)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/compute/api.py", line 2484, in _get_instances_by_filters
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions
expected_attrs=fields, sort_keys=sort_keys, sort_dirs=sort_dirs)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
184, in wrapper
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions result
= fn(cls, context, *args, **kwargs)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/objects/instance.py", line 1220, in get_by_filters
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions
use_slave=use_slave, sort_keys=sort_keys, sort_dirs=sort_dirs)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/db/sqlalchemy/api.py", line 226, in wrapper
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions return
f(*args, **kwargs)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/objects/instance.py", line 1204, in _get_by_filters_impl
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions
sort_keys=sort_keys, sort_dirs=sort_dirs)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/db/api.py", line 763, in instance_get_all_by_filters_sort
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions
sort_dirs=sort_dirs)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/db/sqlalchemy/api.py", line 170, in wrapper
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions return
f(*args, **kwargs)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/db/sqlalchemy/api.py", line 271, in wrapped
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions return
f(context, *args, **kwargs)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2250, in
instance_get_all_by_filters_sort
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions return
_instances_fill_metadata(context, query_prefix.all(), manual_joins)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2613, in
all
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions return
list(self)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2761, in
__iter__
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions return
self._execute_and_instances(context)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2776, in
_execute_and_instances
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions result
= conn.execute(querycontext.statement, self._params)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914,
in execute
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions return
meth(self, multiparams, params)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323,
in _execute_on_connection
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions return
connection._execute_clauseelement(self, multiparams, params)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010,
in _execute_clauseelement
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions
compiled_sql, distilled_params
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146,
in _execute_context
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions context)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337,
in _handle_dbapi_exception
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions
util.raise_from_cause(newraise, exc_info)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203,
in raise_from_cause
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions
reraise(type(exception), exception, tb=exc_tb, cause=cause)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139,
in _execute_context
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions context)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line
450, in do_execute
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions
cursor.execute(statement, parameters)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 166, in
execute
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions result
= self._query(query)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 322, in _query
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions
conn.query(q)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 835, in
query
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1019, in
_read_query_result
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions
result.read()
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1302, in
read
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions
first_packet = self.connection._read_packet()
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 981, in
_read_packet
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions
packet.check_error()
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 393, in
check_error
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions
err.raise_mysql_exception(self._data)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions File
"/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 107, in
raise_mysql_exception
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions raise
errorclass(errno, errval)
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions
ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table
'nova_cell0.instances' doesn't exist") [SQL: u'SELECT
anon_1.instances_created_at AS anon_1_instances_created_at,
anon_1.instances_updated_at AS anon_1_instances_updated_at,
anon_1.instances_deleted_at AS anon_1_instances_deleted_at,
anon_1.instances_deleted AS anon_1_instances_deleted, anon_1.instances_id AS
anon_1_instances_id, anon_1.instances_user_id AS anon_1_instances_user_id,
anon_1.instances_project_id AS anon_1_instances_project_id,
anon_1.instances_image_ref AS anon_1_instances_image_ref,
anon_1.instances_kernel_id AS anon_1_instances_kernel_id,
anon_1.instances_ramdisk_id AS anon_1_instances_ramdisk_id,
anon_1.instances_hostname AS anon_1_instances_hostname,
anon_1.instances_launch_index AS anon_1_instances_launch_index,
anon_1.instances_key_name AS anon_1_instances_key_name,
anon_1.instances_key_data AS anon_1_instances_key_data, anon_1.insta
nces_power_state AS anon_1_instances_power_state, anon_1.instances_vm_state AS
anon_1_instances_vm_state, anon_1.instances_task_state AS
anon_1_instances_task_state, anon_1.instances_memory_mb AS
anon_1_instances_memory_mb, anon_1.instances_vcpus AS anon_1_instances_vcpus,
anon_1.instances_root_gb AS anon_1_instances_root_gb,
anon_1.instances_ephemeral_gb AS anon_1_instances_ephemeral_gb,
anon_1.instances_ephemeral_key_uuid AS anon_1_instances_ephemeral_key_uuid,
anon_1.instances_host AS anon_1_instances_host, anon_1.instances_node AS
anon_1_instances_node, anon_1.instances_instance_type_id AS
anon_1_instances_instance_type_id, anon_1.instances_user_data AS
anon_1_instances_user_data, anon_1.instances_reservation_id AS
anon_1_instances_reservation_id, anon_1.instances_launched_at AS
anon_1_instances_launched_at, anon_1.instances_terminated_at AS
anon_1_instances_terminated_at, anon_1.instances_availability_zone AS
anon_1_instances_availability_zone, anon_1.instances_display_name AS
anon_1_instances_display_name, anon_1.instances_display_description AS
anon_1_instances_display_description, anon_1.instances_launched_on AS
anon_1_instances_launched_on, anon_1.instances_locked AS
anon_1_instances_locked, anon_1.instances_locked_by AS
anon_1_instances_locked_by, anon_1.instances_os_type AS
anon_1_instances_os_type, anon_1.instances_architecture AS
anon_1_instances_architecture, anon_1.instances_vm_mode AS
anon_1_instances_vm_mode, anon_1.instances_uuid AS anon_1_instances_uuid,
anon_1.instances_root_device_name AS anon_1_instances_root_device_name,
anon_1.instances_default_ephemeral_device AS
anon_1_instances_default_ephemeral_device, anon_1.instances_default_swap_device
AS anon_1_instances_default_swap_device, anon_1.instances_config_drive AS
anon_1_instances_config_drive, anon_1.instances_access_ip_v4 AS
anon_1_instances_access_ip_v4, anon_1.instances_access_ip_v6 AS
anon_1_instances_access_ip_v6, anon_1.instances_auto_disk_config AS
anon_1_instances_auto_disk_co
nfig, anon_1.instances_progress AS anon_1_instances_progress,
anon_1.instances_shutdown_terminate AS anon_1_instances_shutdown_terminate,
anon_1.instances_disable_terminate AS anon_1_instances_disable_terminate,
anon_1.instances_cell_name AS anon_1_instances_cell_name,
anon_1.instances_cleaned AS anon_1_instances_cleaned,
security_groups_1.created_at AS security_groups_1_created_at,
security_groups_1.updated_at AS security_groups_1_updated_at,
security_groups_1.deleted_at AS security_groups_1_deleted_at,
security_groups_1.deleted AS security_groups_1_deleted, security_groups_1.id AS
security_groups_1_id, security_groups_1.name AS security_groups_1_name,
security_groups_1.description AS security_groups_1_description,
security_groups_1.user_id AS security_groups_1_user_id,
security_groups_1.project_id AS security_groups_1_project_id,
instance_info_caches_1.created_at AS instance_info_caches_1_created_at,
instance_info_caches_1.updated_at AS instance_info_caches_1_updated_at,
instance_
info_caches_1.deleted_at AS instance_info_caches_1_deleted_at,
instance_info_caches_1.deleted AS instance_info_caches_1_deleted,
instance_info_caches_1.id AS instance_info_caches_1_id,
instance_info_caches_1.network_info AS instance_info_caches_1_network_info,
instance_info_caches_1.instance_uuid AS instance_info_caches_1_instance_uuid
\nFROM (SELECT instances.created_at AS instances_created_at,
instances.updated_at AS instances_updated_at, instances.deleted_at AS
instances_deleted_at, instances.deleted AS instances_deleted, instances.id AS
instances_id, instances.user_id AS instances_user_id, instances.project_id AS
instances_project_id, instances.image_ref AS instances_image_ref,
instances.kernel_id AS instances_kernel_id, instances.ramdisk_id AS
instances_ramdisk_id, instances.hostname AS instances_hostname,
instances.launch_index AS instances_launch_index, instances.key_name AS
instances_key_name, instances.key_data AS instances_key_data,
instances.power_state AS instances_power
_state, instances.vm_state AS instances_vm_state, instances.task_state AS
instances_task_state, instances.memory_mb AS instances_memory_mb,
instances.vcpus AS instances_vcpus, instances.root_gb AS instances_root_gb,
instances.ephemeral_gb AS instances_ephemeral_gb, instances.ephemeral_key_uuid
AS instances_ephemeral_key_uuid, instances.host AS instances_host,
instances.node AS instances_node, instances.instance_type_id AS
instances_instance_type_id, instances.user_data AS instances_user_data,
instances.reservation_id AS instances_reservation_id, instances.launched_at AS
instances_launched_at, instances.terminated_at AS instances_terminated_at,
instances.availability_zone AS instances_availability_zone,
instances.display_name AS instances_display_name, instances.display_description
AS instances_display_description, instances.launched_on AS
instances_launched_on, instances.locked AS instances_locked,
instances.locked_by AS instances_locked_by, instances.os_type AS
instances_os_type, i
nstances.architecture AS instances_architecture, instances.vm_mode AS
instances_vm_mode, instances.uuid AS instances_uuid, instances.root_device_name
AS instances_root_device_name, instances.default_ephemeral_device AS
instances_default_ephemeral_device, instances.default_swap_device AS
instances_default_swap_device, instances.config_drive AS
instances_config_drive, instances.access_ip_v4 AS instances_access_ip_v4,
instances.access_ip_v6 AS instances_access_ip_v6, instances.auto_disk_config AS
instances_auto_disk_config, instances.progress AS instances_progress,
instances.shutdown_terminate AS instances_shutdown_terminate,
instances.disable_terminate AS instances_disable_terminate, instances.cell_name
AS instances_cell_name, instances.cleaned AS instances_cleaned \nFROM instances
\nWHERE instances.deleted = %(deleted_1)s AND (instances.vm_state !=
%(vm_state_1)s OR instances.vm_state IS NULL) AND instances.project_id =
%(project_id_1)s ORDER BY instances.created_at DESC, instances.i
d DESC \n LIMIT %(param_1)s) AS anon_1 LEFT OUTER JOIN
(security_group_instance_association AS security_group_instance_association_1
INNER JOIN security_groups AS security_groups_1 ON security_groups_1.id =
security_group_instance_association_1.security_group_id AND
security_group_instance_association_1.deleted = %(deleted_2)s AND
security_groups_1.deleted = %(deleted_3)s) ON
security_group_instance_association_1.instance_uuid = anon_1.instances_uuid AND
anon_1.instances_deleted = %(deleted_4)s LEFT OUTER JOIN instance_info_caches
AS instance_info_caches_1 ON instance_info_caches_1.instance_uuid =
anon_1.instances_uuid ORDER BY anon_1.instances_created_at DESC,
anon_1.instances_id DESC'] [parameters: {u'param_1': 1000, u'deleted_4': 0,
u'deleted_3': 0, u'deleted_2': 0, u'deleted_1': 0, u'vm_state_1':
'soft-delete', u'project_id_1': u'fb9b4c5ccf39462d9cd1ac7403bda55d'}]
2017-01-20 08:56:41.267 10377 ERROR nova.api.openstack.extensions
2017-01-20 08:56:41.271 10377 INFO nova.api.openstack.wsgi
[req-7b026f0c-4071-4425-85be-e98d8ddc2d33
tempest-TestMinimumBasicScenario-878230226
tempest-TestMinimumBasicScenario-878230226] HTTP exception thrown: Unexpected
API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the
Nova API log if possible.
<class 'sqlalchemy.exc.ProgrammingError'>
2017-01-20 08:56:41.272 10377 DEBUG nova.api.openstack.wsgi
[req-7b026f0c-4071-4425-85be-e98d8ddc2d33
tempest-TestMinimumBasicScenario-878230226
tempest-TestMinimumBasicScenario-878230226] Returning 500 to user: Unexpected
API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the
Nova API log if possible.
<class 'sqlalchemy.exc.ProgrammingError'> __call__
/opt/stack/nova/nova/api/openstack/wsgi.py:1039
Example failure at http://78.46.57.153:8081/refs-changes-77-407977-6/
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1658090/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
