Public bug reported: Summary: When block-migrating a vm between two compute nodes, if the destination node lacks ssh daemon or refuses ssh connection, the block-migration would fail and cause permanent damage to the vm, vm will be unusable and no easy way to recover.
Scenario: Block-migrating a vm between two compute nodes Example: Compute Node 1: ly-compute1 (10.0.0.31) Compute Node 2: ly-compute2 (10.0.0.32) with VM: test11 The below tutorial is what I followed for openstack installation, it doesn't say about installing ssh support on compute nodes, so I didn't install ssh on both compute nodes. OpenStack Installation Guide for Ubuntu 12.04/14.04 (LTS) http://docs.openstack.org/icehouse/install-guide/install/apt/content/ Error occurs when I use the command "nova migrate --poll test11" trying to migrate vm "test11" from ly-compute2 to ly-compute1. both error message and log said that ssh connecting to 10.0.0.31 failed (because ssh daemon is NOT even installed.) And the vm "test11" will enter a vm_state error state. ********************************************************************* Tring to migrate: C:\Windows\system32>nova migrate --poll test11 Server migrating... 0% complete Error migrating server ERROR (InstanceInErrorState): Unexpected error while running command. Command: ssh 10.0.0.31 mkdir -p /var/lib/nova/instances/30c4dac1-f3bc-4e6a-8a38-ee49671eee6a Exit code: 255 Stdout: u'' Stderr: u'ssh: connect to host 10.0.0.31 port 22: Connection refused\r\n' ********************************************************************* dashboard log message: Unexpected error while running command. Command: ssh 10.0.0.31 mkdir -p /var/lib/nova/instances/30c4dac1-f3bc-4e6a-8a38-ee49671eee6a Exit code: 255 Stdout: u'' Stderr: u'ssh: connect to host 10.0.0.31 port 22: Connection refused\r\n' Code 500 Details File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 290, in decorated_function return function(self, context, *args, **kwargs) File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3472, in resize_instance block_device_info) File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4954, in migrate_disk_and_power_off utils.execute('ssh', dest, 'mkdir', '-p', inst_base) File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 165, in execute return processutils.execute(*cmd, **kwargs) File "/usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py", line 195, in execute cmd=sanitized_cmd) ********************************************************************* Second time tring to migrate: C:\Windows\system32>nova migrate --poll test11 ERROR (Conflict): Cannot 'migrate' while instance is in vm_state error (HTTP 409) (Request-ID: req-ba3ca8e1-0753-40ac-9e2e-2f7c319ec691) ********************************************************************* Request: This fault is VERY DANGEROUS, because it will cause permanent damage to the vm of a user. The migrate_disk_and_power_off function from nova/virt/libvirt/driver.py should pre-check if ssh daemon is running on dest node, before the actual block-migration process. ** Affects: nova Importance: Undecided Status: New ** Tags: compute libvirt nova-manage -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1406167 Title: Block migration fails because destination compute node refuses ssh connection Status in OpenStack Compute (Nova): New Bug description: Summary: When block-migrating a vm between two compute nodes, if the destination node lacks ssh daemon or refuses ssh connection, the block-migration would fail and cause permanent damage to the vm, vm will be unusable and no easy way to recover. Scenario: Block-migrating a vm between two compute nodes Example: Compute Node 1: ly-compute1 (10.0.0.31) Compute Node 2: ly-compute2 (10.0.0.32) with VM: test11 The below tutorial is what I followed for openstack installation, it doesn't say about installing ssh support on compute nodes, so I didn't install ssh on both compute nodes. OpenStack Installation Guide for Ubuntu 12.04/14.04 (LTS) http://docs.openstack.org/icehouse/install-guide/install/apt/content/ Error occurs when I use the command "nova migrate --poll test11" trying to migrate vm "test11" from ly-compute2 to ly-compute1. both error message and log said that ssh connecting to 10.0.0.31 failed (because ssh daemon is NOT even installed.) And the vm "test11" will enter a vm_state error state. ********************************************************************* Tring to migrate: C:\Windows\system32>nova migrate --poll test11 Server migrating... 0% complete Error migrating server ERROR (InstanceInErrorState): Unexpected error while running command. Command: ssh 10.0.0.31 mkdir -p /var/lib/nova/instances/30c4dac1-f3bc-4e6a-8a38-ee49671eee6a Exit code: 255 Stdout: u'' Stderr: u'ssh: connect to host 10.0.0.31 port 22: Connection refused\r\n' ********************************************************************* dashboard log message: Unexpected error while running command. Command: ssh 10.0.0.31 mkdir -p /var/lib/nova/instances/30c4dac1-f3bc-4e6a-8a38-ee49671eee6a Exit code: 255 Stdout: u'' Stderr: u'ssh: connect to host 10.0.0.31 port 22: Connection refused\r\n' Code 500 Details File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 290, in decorated_function return function(self, context, *args, **kwargs) File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3472, in resize_instance block_device_info) File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4954, in migrate_disk_and_power_off utils.execute('ssh', dest, 'mkdir', '-p', inst_base) File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 165, in execute return processutils.execute(*cmd, **kwargs) File "/usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py", line 195, in execute cmd=sanitized_cmd) ********************************************************************* Second time tring to migrate: C:\Windows\system32>nova migrate --poll test11 ERROR (Conflict): Cannot 'migrate' while instance is in vm_state error (HTTP 409) (Request-ID: req-ba3ca8e1-0753-40ac-9e2e-2f7c319ec691) ********************************************************************* Request: This fault is VERY DANGEROUS, because it will cause permanent damage to the vm of a user. The migrate_disk_and_power_off function from nova/virt/libvirt/driver.py should pre-check if ssh daemon is running on dest node, before the actual block-migration process. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1406167/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
