Public bug reported: Running Liberty and on one of the image (CN-00) below - while all the other nodes booted and created VOLUME from IMAGE, this one started to FAIL. No idea why.
As with the other instances, this nova boot command should result in the GLANCE Image being used to create the CINDER volume (note - there is 5.5TB available so its not a space issue - and the error doesnt reflect that the call ever makes it to CINDER). EXPECTED RESULT: nova boots the Instance, creating the CINDER volume from the glance image-id as outlined in the command. GLANCE IMAGES: root@node-16:/var/log/nova# glance image-list +--------------------------------------+-----------------+ | ID | Name | +--------------------------------------+-----------------+ | 6be3e076-691e-48cf-b9ed-264de01965b8 | CN-00_R2C25 | | f47da74d-15d0-42b7-8261-c6d3b8f0dc2a | DB-00-1_R2C25 | | fd1d5774-5139-4644-81ac-1d5293d8410e | MS-00_R2C25 | | 9b2dc827-b04a-4def-b36c-b1ee11449705 | NAS | | 3444ce43-99b1-482e-97ef-35d7bb15aecc | PGPOOL-00_R2C25 | | ee11f00a-6ab4-42b3-b86d-f1ee1966ca5f | TestVM | | a1503971-ddcf-4740-bcae-9efc3afe2988 | UTILITY-R2C25 | +--------------------------------------+-----------------+ NOVA BOOT COMMAND: root@node-16:/var/log/nova# nova boot --flavor MDN-8G-8CPU --block-device source=image,id=6be3e076-691e-48cf-b9ed-264de01965b8,dest=volume,bus=scsi,bootindex=0 --nic port-id=5f9c10e9-09b9-4116-9d3a-348882d78817 --nic port-id=1c74b4bd-aaf0-4681-a566-92fdf0f69b0c --nic port-id=d4e59190-24ad-466c-8551-1e29ed3416fa --nic port-id=c08fa16e-11da-49d7-b12c-291dfe836d4c CN-00 ERROR: ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'nova.exception.InvalidBDM'> (HTTP 500) (Request-ID: req-94b9fea7-1f45-486c-9292-47b2a3e6dc56) NOVA-API.LOG 2016-03-29 01:05:05.173 41670 WARNING oslo_config.cfg [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] Option "auth_strategy" from group "neutron" is deprecated for removal. Its value may be silently ignored in the future. 2016-03-29 01:05:05.174 41670 WARNING oslo_config.cfg [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] Option "admin_auth_url" from group "neutron" is deprecated for removal. Its value may be silently ignored in the future. 2016-03-29 01:05:05.175 41670 WARNING oslo_config.cfg [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] Option "admin_username" from group "neutron" is deprecated for removal. Its value may be silently ignored in the future. 2016-03-29 01:05:05.176 41670 WARNING oslo_config.cfg [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] Option "admin_password" from group "neutron" is deprecated for removal. Its value may be silently ignored in the future. 2016-03-29 01:05:05.176 41670 WARNING oslo_config.cfg [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] Option "admin_tenant_name" from group "neutron" is deprecated for removal. Its value may be silently ignored in the future. 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] Unexpected exception in API method 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 611, in create 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions **create_kwargs) 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 149, in inner 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions rv = f(*args, **kwargs) 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1581, in create 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions check_server_group_quota=check_server_group_quota) 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1211, in _create_instance 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions instance_group, check_server_group_quota) 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1060, in _provision_instances 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions quotas.rollback() 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__ 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb) 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1023, in _provision_instances 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions num_instances, i, shutdown_terminate) 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1499, in create_db_entry_for_new_instance 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions instance.destroy() 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__ 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb) 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1495, in create_db_entry_for_new_instance 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions context, instance, instance_type, block_device_mapping) 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1331, in _validate_bdm 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions raise exception.InvalidBDM(message=_("Images with " 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions InvalidBDM: Images with destination_type 'volume' need to have a non-zero size specified 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions 2016-03-29 01:05:06.480 41670 INFO nova.api.openstack.wsgi [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'nova.exception.InvalidBDM'> 2016-03-29 01:05:06.481 41670 INFO nova.osapi_compute.wsgi.server [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] 192.168.0.2 "POST /v2/d37e0a300f154eb099d40ef9cf362123/os-volumes_boot HTTP/1.1" status: 500 len: 425 time: 2.1962600 GLANCE-IMAGE SHOW: (you can see that the IMAGE is NOT zero bytes): root@node-16:/var/log/nova# glance image-show 6be3e076-691e-48cf-b9ed-264de01965b8 +------------------+--------------------------------------+ | Property | Value | +------------------+--------------------------------------+ | checksum | e1177f3cb1625b36c55b7d171ba9d240 | | container_format | bare | | created_at | 2016-03-29T00:27:28Z | | disk_format | qcow2 | | id | 6be3e076-691e-48cf-b9ed-264de01965b8 | | min_disk | 0 | | min_ram | 0 | | name | CN-00_R2C25 | | owner | d37e0a300f154eb099d40ef9cf362123 | | protected | False | | size | 3437887488 | | status | active | | tags | [] | | updated_at | 2016-03-29T00:28:08Z | | virtual_size | None | | visibility | public | +------------------+------------ MORE INFO: root@node-16:/var/log/nova# nova --version 2.30.2 root@node-16:/var/log/nova# IMPOSSIBLE TO ADD SCRIPT - since it would mean pushing the image file to you. ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1563141 Title: Nova boot Fails InvalidBDM from GLANCE IMAGE Status in OpenStack Compute (nova): New Bug description: Running Liberty and on one of the image (CN-00) below - while all the other nodes booted and created VOLUME from IMAGE, this one started to FAIL. No idea why. As with the other instances, this nova boot command should result in the GLANCE Image being used to create the CINDER volume (note - there is 5.5TB available so its not a space issue - and the error doesnt reflect that the call ever makes it to CINDER). EXPECTED RESULT: nova boots the Instance, creating the CINDER volume from the glance image-id as outlined in the command. GLANCE IMAGES: root@node-16:/var/log/nova# glance image-list +--------------------------------------+-----------------+ | ID | Name | +--------------------------------------+-----------------+ | 6be3e076-691e-48cf-b9ed-264de01965b8 | CN-00_R2C25 | | f47da74d-15d0-42b7-8261-c6d3b8f0dc2a | DB-00-1_R2C25 | | fd1d5774-5139-4644-81ac-1d5293d8410e | MS-00_R2C25 | | 9b2dc827-b04a-4def-b36c-b1ee11449705 | NAS | | 3444ce43-99b1-482e-97ef-35d7bb15aecc | PGPOOL-00_R2C25 | | ee11f00a-6ab4-42b3-b86d-f1ee1966ca5f | TestVM | | a1503971-ddcf-4740-bcae-9efc3afe2988 | UTILITY-R2C25 | +--------------------------------------+-----------------+ NOVA BOOT COMMAND: root@node-16:/var/log/nova# nova boot --flavor MDN-8G-8CPU --block-device source=image,id=6be3e076-691e-48cf-b9ed-264de01965b8,dest=volume,bus=scsi,bootindex=0 --nic port-id=5f9c10e9-09b9-4116-9d3a-348882d78817 --nic port-id=1c74b4bd-aaf0-4681-a566-92fdf0f69b0c --nic port-id=d4e59190-24ad-466c-8551-1e29ed3416fa --nic port-id=c08fa16e-11da-49d7-b12c-291dfe836d4c CN-00 ERROR: ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'nova.exception.InvalidBDM'> (HTTP 500) (Request-ID: req-94b9fea7-1f45-486c-9292-47b2a3e6dc56) NOVA-API.LOG 2016-03-29 01:05:05.173 41670 WARNING oslo_config.cfg [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] Option "auth_strategy" from group "neutron" is deprecated for removal. Its value may be silently ignored in the future. 2016-03-29 01:05:05.174 41670 WARNING oslo_config.cfg [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] Option "admin_auth_url" from group "neutron" is deprecated for removal. Its value may be silently ignored in the future. 2016-03-29 01:05:05.175 41670 WARNING oslo_config.cfg [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] Option "admin_username" from group "neutron" is deprecated for removal. Its value may be silently ignored in the future. 2016-03-29 01:05:05.176 41670 WARNING oslo_config.cfg [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] Option "admin_password" from group "neutron" is deprecated for removal. Its value may be silently ignored in the future. 2016-03-29 01:05:05.176 41670 WARNING oslo_config.cfg [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] Option "admin_tenant_name" from group "neutron" is deprecated for removal. Its value may be silently ignored in the future. 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] Unexpected exception in API method 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 611, in create 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions **create_kwargs) 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 149, in inner 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions rv = f(*args, **kwargs) 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1581, in create 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions check_server_group_quota=check_server_group_quota) 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1211, in _create_instance 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions instance_group, check_server_group_quota) 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1060, in _provision_instances 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions quotas.rollback() 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__ 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb) 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1023, in _provision_instances 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions num_instances, i, shutdown_terminate) 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1499, in create_db_entry_for_new_instance 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions instance.destroy() 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__ 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb) 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1495, in create_db_entry_for_new_instance 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions context, instance, instance_type, block_device_mapping) 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1331, in _validate_bdm 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions raise exception.InvalidBDM(message=_("Images with " 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions InvalidBDM: Images with destination_type 'volume' need to have a non-zero size specified 2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions 2016-03-29 01:05:06.480 41670 INFO nova.api.openstack.wsgi [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'nova.exception.InvalidBDM'> 2016-03-29 01:05:06.481 41670 INFO nova.osapi_compute.wsgi.server [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] 192.168.0.2 "POST /v2/d37e0a300f154eb099d40ef9cf362123/os-volumes_boot HTTP/1.1" status: 500 len: 425 time: 2.1962600 GLANCE-IMAGE SHOW: (you can see that the IMAGE is NOT zero bytes): root@node-16:/var/log/nova# glance image-show 6be3e076-691e-48cf-b9ed-264de01965b8 +------------------+--------------------------------------+ | Property | Value | +------------------+--------------------------------------+ | checksum | e1177f3cb1625b36c55b7d171ba9d240 | | container_format | bare | | created_at | 2016-03-29T00:27:28Z | | disk_format | qcow2 | | id | 6be3e076-691e-48cf-b9ed-264de01965b8 | | min_disk | 0 | | min_ram | 0 | | name | CN-00_R2C25 | | owner | d37e0a300f154eb099d40ef9cf362123 | | protected | False | | size | 3437887488 | | status | active | | tags | [] | | updated_at | 2016-03-29T00:28:08Z | | virtual_size | None | | visibility | public | +------------------+------------ MORE INFO: root@node-16:/var/log/nova# nova --version 2.30.2 root@node-16:/var/log/nova# IMPOSSIBLE TO ADD SCRIPT - since it would mean pushing the image file to you. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1563141/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
