Public bug reported: when we configure nova to launch instances with preallocated disk and we fail to spown an instance because we do not have enough disk space, horizon is reporting no valid host found: Error: Failed to launch instance "dafna": Please try again later [Error: No valid host was found. ].
it would be good if we can calculate the amount of space left before launching the instance from horizon (I opened a bug to nova to see if we can add a test in nova before launch - https://bugs.launchpad.net/nova/+bug/1319920). but, it would also be good to report that there is not enough disk space to launch the instance (as reported in compute log) which would be much clearer than no valid hosts which makes the user search for issues in the host. 2014-05-15 19:12:57.878 23617 ERROR nova.compute.manager [req-1607bb0f-88a3-4888-b751-dab00e24f824 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Instance failed to spa wn 2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Traceback (most recent call last): 2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1720, in _spawn 2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] block_device_info) 2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2246, in spawn 2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] admin_pass=admin_password) 2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2615, in _create_image 2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] project_id=instance['project_id']) 2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 196, in cache 2014-05-15 19:12:57.878 23617 TRACE n2014-05-15 19:12:58.471 23617 INFO nova.virt.libvirt.driver [req-1607bb0f-88a3-4888-b751-dab00e24f824 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: c1ad975d-6974-4539-9 a1d-c050a3abd339] Deletion of /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339 complete 2014-05-15 19:12:59.093 23617 ERROR nova.compute.manager [req-1607bb0f-88a3-4888-b751-dab00e24f824 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Error: Unexpected erro r while running command. Command: fallocate -n -l 171798691840 /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk Exit code: 1 Stdout: '' Stderr: 'fallocate: /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk: fallocate failed: No space left on device\n' 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Traceback (most recent call last): 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1311, in _build_instance 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] set_access_ip=set_access_ip) 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 399, in decorated_function 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] return function(self, context, *args, **kwargs) 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1723, in _spawn 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] LOG.exception(_('Instance failed to spawn'), instance=instance) 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__ 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] six.reraise(self.type_, self.value, self.tb) 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1720, in _spawn 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] block_device_info) 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2246, in spawn 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] admin_pass=admin_password) 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2615, in _create_image 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] project_id=instance['project_id']) 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 196, in cache 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] utils.execute('fallocate', '-n', '-l', size, self.path) 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/utils.py", line 164, in execute 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] return processutils.execute(*cmd, **kwargs) 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py", line 193, in execute 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] cmd=' '.join(cmd)) 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] ProcessExecutionError: Unexpected error while running command. 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Command: fallocate -n -l 171798691840 /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Exit code: 1 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Stdout: '' 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Stderr: 'fallocate: /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk: fallocate failed: No space left on device\n' 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] 2014-05-15 19:13:36.350 23617 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2014-05-15 19:13:36.493 23617 AUDIT nova.compute.resource_tracker [-] Free ram (MB): -1008 2014-05-15 19:13:36.493 23617 AUDIT nova.compute.resource_tracker [-] Free disk (GB): -101 : ** Affects: horizon Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Dashboard (Horizon). https://bugs.launchpad.net/bugs/1319926 Title: incorrect error when failing to launch on No space left on device\n Status in OpenStack Dashboard (Horizon): New Bug description: when we configure nova to launch instances with preallocated disk and we fail to spown an instance because we do not have enough disk space, horizon is reporting no valid host found: Error: Failed to launch instance "dafna": Please try again later [Error: No valid host was found. ]. it would be good if we can calculate the amount of space left before launching the instance from horizon (I opened a bug to nova to see if we can add a test in nova before launch - https://bugs.launchpad.net/nova/+bug/1319920). but, it would also be good to report that there is not enough disk space to launch the instance (as reported in compute log) which would be much clearer than no valid hosts which makes the user search for issues in the host. 2014-05-15 19:12:57.878 23617 ERROR nova.compute.manager [req-1607bb0f-88a3-4888-b751-dab00e24f824 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Instance failed to spa wn 2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Traceback (most recent call last): 2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1720, in _spawn 2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] block_device_info) 2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2246, in spawn 2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] admin_pass=admin_password) 2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2615, in _create_image 2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] project_id=instance['project_id']) 2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 196, in cache 2014-05-15 19:12:57.878 23617 TRACE n2014-05-15 19:12:58.471 23617 INFO nova.virt.libvirt.driver [req-1607bb0f-88a3-4888-b751-dab00e24f824 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: c1ad975d-6974-4539-9 a1d-c050a3abd339] Deletion of /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339 complete 2014-05-15 19:12:59.093 23617 ERROR nova.compute.manager [req-1607bb0f-88a3-4888-b751-dab00e24f824 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Error: Unexpected erro r while running command. Command: fallocate -n -l 171798691840 /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk Exit code: 1 Stdout: '' Stderr: 'fallocate: /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk: fallocate failed: No space left on device\n' 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Traceback (most recent call last): 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1311, in _build_instance 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] set_access_ip=set_access_ip) 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 399, in decorated_function 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] return function(self, context, *args, **kwargs) 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1723, in _spawn 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] LOG.exception(_('Instance failed to spawn'), instance=instance) 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__ 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] six.reraise(self.type_, self.value, self.tb) 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1720, in _spawn 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] block_device_info) 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2246, in spawn 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] admin_pass=admin_password) 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2615, in _create_image 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] project_id=instance['project_id']) 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 196, in cache 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] utils.execute('fallocate', '-n', '-l', size, self.path) 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/utils.py", line 164, in execute 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] return processutils.execute(*cmd, **kwargs) 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] File "/usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py", line 193, in execute 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] cmd=' '.join(cmd)) 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] ProcessExecutionError: Unexpected error while running command. 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Command: fallocate -n -l 171798691840 /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Exit code: 1 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Stdout: '' 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Stderr: 'fallocate: /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk: fallocate failed: No space left on device\n' 2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] 2014-05-15 19:13:36.350 23617 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2014-05-15 19:13:36.493 23617 AUDIT nova.compute.resource_tracker [-] Free ram (MB): -1008 2014-05-15 19:13:36.493 23617 AUDIT nova.compute.resource_tracker [-] Free disk (GB): -101 : To manage notifications about this bug go to: https://bugs.launchpad.net/horizon/+bug/1319926/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
