Public bug reported: Environment: - Two compute nodes, running Ubuntu 12.04 LTS - KVM Hypervisor - Ceph (dumpling) back-end for Cinder - Grizzly-level Openstack
Steps to reproduce: 1) Create instance and volume 2) Attach volume to instance 3) Attempt a block migration between compute nodes - eg: nova live-migration --block-migrate 9b85b983-dced-4574-b14c-c72e4d92982a Packages: ii ceph 0.67.5-1precise ii ceph-common 0.67.5-1precise ii ceph-fs-common 0.67.5-1precise ii ceph-fuse 0.67.5-1precise ii ceph-mds 0.67.5-1precise ii curl 7.29.0-1precise.ceph ii kvm 1:84+dfsg-0ubuntu16+1.0+noroms+0ubuntu14.13 ii kvm-ipxe 1.0.0+git-3.55f6c88-0ubuntu1 ii libcephfs1 0.67.5-1precise ii libcurl3 7.29.0-1precise.ceph ii libcurl3-gnutls 7.29.0-1precise.ceph ii libleveldb1 1.12.0-1precise.ceph ii nova-common 1:2013.1.4-0ubuntu1~cloud0 ii nova-compute 1:2013.1.4-0ubuntu1~cloud0 ii nova-compute-kvm 1:2013.1.4-0ubuntu1~cloud0 ii python-ceph 0.67.5-1precise ii python-cinderclient 1:1.0.3-0ubuntu1~cloud0 ii python-nova 1:2013.1.4-0ubuntu1~cloud0 ii python-novaclient 1:2.13.0-0ubuntu1~cloud0 ii qemu-common 1.0+noroms-0ubuntu14.13 ii qemu-kvm 1.0+noroms-0ubuntu14.13 ii qemu-utils 1.0+noroms-0ubuntu14.13 ii libvirt-bin 1.0.2-0ubuntu11.13.04.5~cloud1 ii libvirt0 1.0.2-0ubuntu11.13.04.5~cloud1 ii python-libvirt 1.0.2-0ubuntu11.13.04.5~cloud1 /var/log/nova/nova-compute on source: 2014-02-05 16:36:46.014 998 INFO nova.compute.manager [-] Lifecycle event 2 on VM 9b85b983-dced-4574-b14c-c72e4d92982a 2014-02-05 16:36:46.233 998 INFO nova.compute.manager [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] During sync_power_state the instance has a pending task. Skip. 2014-02-05 16:36:46.234 998 INFO nova.compute.manager [-] Lifecycle event 2 on VM 9b85b983-dced-4574-b14c-c72e4d92982a 2014-02-05 16:36:46.468 998 INFO nova.compute.manager [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] During sync_power_state the instance has a pending task. Skip. 2014-02-05 16:41:09.029 998 INFO nova.compute.manager [-] Lifecycle event 1 on VM 9b85b983-dced-4574-b14c-c72e4d92982a 2014-02-05 16:41:09.265 998 INFO nova.compute.manager [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] During sync_power_state the instance has a pending task. Skip. 2014-02-05 16:41:09.640 998 ERROR nova.virt.libvirt.driver [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Live Migration failure: Unable to read from monitor: Connection reset by peer 2014-02-05 16:41:12.165 998 WARNING nova.compute.manager [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Instance shutdown by itself. Calling the stop API. 2014-02-05 16:41:12.398 998 INFO nova.virt.libvirt.driver [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Instance destroyed successfully. /var/log/libvirt/libvirtd.log on source: 2014-02-05 14:41:07.607+0000: 3437: error : qemuMonitorIORead:502 : Unable to read from monitor: Connection reset by peer 2014-02-05 14:41:09.633+0000: 3441: error : virNetClientProgramDispatchError:175 : An error occurred, but the cause is unknown 2014-02-05 14:41:09.634+0000: 3441: error : qemuDomainObjEnterMonitorInternal:997 : operation failed: domain is no longer running 2014-02-05 14:41:09.634+0000: 3441: warning : doPeer2PeerMigrate3:2872 : Guest instance-00000315 probably left in 'paused' state on source /var/log/nova/nova-compute.log on target: 2014-02-05 16:36:38.841 INFO nova.virt.libvirt.driver [req-0f0eaabf-9e29-4d45-88c9-20194be51d49 aaf3e92b69e04958b43348677ab7b38b 1859d80f51ff4180b591f7fe2668fd68] Instance launched has CPU info: {"vendor": "Intel", "model": "SandyBridge", "arch": "x86_64", "features": ["pdpe1gb", "osxsave", "dca", "pcid", "pdcm", "xtpr", "tm2", "est", "smx", "vmx", "ds_cpl", "monitor", "dtes64", "pbe", "tm", "ht", "ss", "acpi", "ds", "vme"], "topology": {"cores": 6, "threads": 2, "sockets": 1}} 2014-02-05 16:36:46.008 28458 INFO nova.compute.manager [-] Lifecycle event 0 on VM 9b85b983-dced-4574-b14c-c72e4d92982a 2014-02-05 16:36:46.244 28458 INFO nova.compute.manager [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] During the sync_power process the instance has moved from host ctpcmp003 to host ctpcmp005 2014-02-05 16:41:09.634 28458 INFO nova.compute.manager [-] Lifecycle event 1 on VM 9b85b983-dced-4574-b14c-c72e4d92982a 2014-02-05 16:41:09.899 28458 INFO nova.compute.manager [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] During the sync_power process the instance has moved from host ctpcmp003 to host ctpcmp005 2014-02-05 16:41:11.056 AUDIT nova.compute.manager [req-0f0eaabf-9e29-4d45-88c9-20194be51d49 aaf3e92b69e04958b43348677ab7b38b 1859d80f51ff4180b591f7fe2668fd68] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Detach volume 7e2f873e-f7ba-4ee1-85ee-cee133cefdcf from mountpoint /dev/vdc 2014-02-05 16:41:11.061 WARNING nova.compute.manager [req-0f0eaabf-9e29-4d45-88c9-20194be51d49 aaf3e92b69e04958b43348677ab7b38b 1859d80f51ff4180b591f7fe2668fd68] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Detaching volume from unknown instance 2014-02-05 16:41:11.066 ERROR nova.compute.manager [req-0f0eaabf-9e29-4d45-88c9-20194be51d49 aaf3e92b69e04958b43348677ab7b38b 1859d80f51ff4180b591f7fe2668fd68] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Failed to detach volume 7e2f873e-f7ba-4ee1-85ee-cee133cefdcf from /dev/vdc 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Traceback (most recent call last): 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2983, in _detach_volume 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] mp) 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1016, in detach_volume 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] virt_dom = self._lookup_by_name(instance_name) 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2386, in _lookup_by_name 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] raise exception.InstanceNotFound(instance_id=instance_name) 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] InstanceNotFound: Instance instance-00000315 could not be found. 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] 2014-02-05 16:41:12.261 28458 ERROR nova.virt.libvirt.driver [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] During wait destroy, instance disappeared. 2014-02-05 16:41:12.872 INFO nova.virt.libvirt.driver [req-0f0eaabf-9e29-4d45-88c9-20194be51d49 aaf3e92b69e04958b43348677ab7b38b 1859d80f51ff4180b591f7fe2668fd68] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Deleting instance files /var/lib/nova/instances/9b85b983-dced-4574-b14c-c72e4d92982a /etc/nova/nova.conf includes the directive: vncserver_listen=0.0.0.0 root@ctpcmp003:~# grep "tls\|tcp" /etc/libvirt/libvirtd.conf | grep -v "^#" listen_tls = 0 listen_tcp = 1 auth_tcp = "none" root@ctpcmp003:~# grep vncserver_listen /etc/nova/nova.conf vncserver_listen=0.0.0.0 root@ctpcmp003:~# ps -ef | grep libvirtd root 20608 1 0 16:29 ? 00:00:11 /usr/sbin/libvirtd -d -l ** Affects: nova Importance: Undecided Status: New ** Description changed: Environment: - - Two compute nodes, running Ubuntu 12.04 LTS - - KVM Hypervisor - - Ceph (dumpling) back-end for Cinder - - Grizzly-level Openstack + - Two compute nodes, running Ubuntu 12.04 LTS + - KVM Hypervisor + - Ceph (dumpling) back-end for Cinder + - Grizzly-level Openstack Steps to reproduce: - 1) Create instance and volume - 2) Attach volume to instance - 3) Attempt a block migration between compute nodes + 1) Create instance and volume + 2) Attach volume to instance + 3) Attempt a block migration between compute nodes Packages: - ii ceph 0.67.5-1precise distributed storage and file system - ii ceph-common 0.67.5-1precise common utilities to mount and interact with a ceph storage cluster - ii ceph-fs-common 0.67.5-1precise common utilities to mount and interact with a ceph file system - ii ceph-fuse 0.67.5-1precise FUSE-based client for the Ceph distributed file system - ii ceph-mds 0.67.5-1precise metadata server for the ceph distributed file system - ii curl 7.29.0-1precise.ceph command line tool for transferring data with URL syntax - ii kvm 1:84+dfsg-0ubuntu16+1.0+noroms+0ubuntu14.13 dummy transitional package from kvm to qemu-kvm - ii kvm-ipxe 1.0.0+git-3.55f6c88-0ubuntu1 PXE ROM's for KVM - ii libcephfs1 0.67.5-1precise Ceph distributed file system client library - ii libcurl3 7.29.0-1precise.ceph easy-to-use client-side URL transfer library (OpenSSL flavour) - ii libcurl3-gnutls 7.29.0-1precise.ceph easy-to-use client-side URL transfer library (GnuTLS flavour) - ii libleveldb1 1.12.0-1precise.ceph fast key-value storage library - ii nova-common 1:2013.1.4-0ubuntu1~cloud0 OpenStack Compute - common files - ii nova-compute 1:2013.1.4-0ubuntu1~cloud0 OpenStack Compute - compute node - ii nova-compute-kvm 1:2013.1.4-0ubuntu1~cloud0 OpenStack Compute - compute node (KVM) - ii python-ceph 0.67.5-1precise Python libraries for the Ceph distributed filesystem - ii python-cinderclient 1:1.0.3-0ubuntu1~cloud0 python bindings to the OpenStack Volume API - ii python-nova 1:2013.1.4-0ubuntu1~cloud0 OpenStack Compute Python libraries - ii python-novaclient 1:2.13.0-0ubuntu1~cloud0 client library for OpenStack Compute API - ii qemu-common 1.0+noroms-0ubuntu14.13 qemu common functionality (bios, documentation, etc) - ii qemu-kvm 1.0+noroms-0ubuntu14.13 Full virtualization on i386 and amd64 hardware - ii qemu-utils 1.0+noroms-0ubuntu14.13 qemu utilities - ii libvirt-bin 1.0.2-0ubuntu11.13.04.5~cloud1 programs for the libvirt library - ii libvirt0 1.0.2-0ubuntu11.13.04.5~cloud1 library for interfacing with different virtualization systems - ii python-libvirt 1.0.2-0ubuntu11.13.04.5~cloud1 libvirt Python bindings + ii ceph 0.67.5-1precise + ii ceph-common 0.67.5-1precise + ii ceph-fs-common 0.67.5-1precise + ii ceph-fuse 0.67.5-1precise + ii ceph-mds 0.67.5-1precise + ii curl 7.29.0-1precise.ceph + ii kvm 1:84+dfsg-0ubuntu16+1.0+noroms+0ubuntu14.13 + ii kvm-ipxe 1.0.0+git-3.55f6c88-0ubuntu1 + ii libcephfs1 0.67.5-1precise + ii libcurl3 7.29.0-1precise.ceph + ii libcurl3-gnutls 7.29.0-1precise.ceph + ii libleveldb1 1.12.0-1precise.ceph + ii nova-common 1:2013.1.4-0ubuntu1~cloud0 + ii nova-compute 1:2013.1.4-0ubuntu1~cloud0 + ii nova-compute-kvm 1:2013.1.4-0ubuntu1~cloud0 + ii python-ceph 0.67.5-1precise + ii python-cinderclient 1:1.0.3-0ubuntu1~cloud0 + ii python-nova 1:2013.1.4-0ubuntu1~cloud0 + ii python-novaclient 1:2.13.0-0ubuntu1~cloud0 + ii qemu-common 1.0+noroms-0ubuntu14.13 + ii qemu-kvm 1.0+noroms-0ubuntu14.13 + ii qemu-utils 1.0+noroms-0ubuntu14.13 + ii libvirt-bin 1.0.2-0ubuntu11.13.04.5~cloud1 + ii libvirt0 1.0.2-0ubuntu11.13.04.5~cloud1 + ii python-libvirt 1.0.2-0ubuntu11.13.04.5~cloud1 /var/log/nova/nova-compute on source: 2014-02-05 16:36:46.014 998 INFO nova.compute.manager [-] Lifecycle event 2 on VM 9b85b983-dced-4574-b14c-c72e4d92982a 2014-02-05 16:36:46.233 998 INFO nova.compute.manager [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] During sync_power_state the instance has a pending task. Skip. 2014-02-05 16:36:46.234 998 INFO nova.compute.manager [-] Lifecycle event 2 on VM 9b85b983-dced-4574-b14c-c72e4d92982a 2014-02-05 16:36:46.468 998 INFO nova.compute.manager [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] During sync_power_state the instance has a pending task. Skip. 2014-02-05 16:41:09.029 998 INFO nova.compute.manager [-] Lifecycle event 1 on VM 9b85b983-dced-4574-b14c-c72e4d92982a 2014-02-05 16:41:09.265 998 INFO nova.compute.manager [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] During sync_power_state the instance has a pending task. Skip. 2014-02-05 16:41:09.640 998 ERROR nova.virt.libvirt.driver [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Live Migration failure: Unable to read from monitor: Connection reset by peer 2014-02-05 16:41:12.165 998 WARNING nova.compute.manager [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Instance shutdown by itself. Calling the stop API. 2014-02-05 16:41:12.398 998 INFO nova.virt.libvirt.driver [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Instance destroyed successfully. - /var/log/libvirt/libvirtd.log on source: 2014-02-05 14:41:07.607+0000: 3437: error : qemuMonitorIORead:502 : Unable to read from monitor: Connection reset by peer 2014-02-05 14:41:09.633+0000: 3441: error : virNetClientProgramDispatchError:175 : An error occurred, but the cause is unknown 2014-02-05 14:41:09.634+0000: 3441: error : qemuDomainObjEnterMonitorInternal:997 : operation failed: domain is no longer running 2014-02-05 14:41:09.634+0000: 3441: warning : doPeer2PeerMigrate3:2872 : Guest instance-00000315 probably left in 'paused' state on source - /var/log/nova/nova-compute.log on target: 2014-02-05 16:36:38.841 INFO nova.virt.libvirt.driver [req-0f0eaabf-9e29-4d45-88c9-20194be51d49 aaf3e92b69e04958b43348677ab7b38b 1859d80f51ff4180b591f7fe2668fd68] Instance launched has CPU info: {"vendor": "Intel", "model": "SandyBridge", "arch": "x86_64", "features": ["pdpe1gb", "osxsave", "dca", "pcid", "pdcm", "xtpr", "tm2", "est", "smx", "vmx", "ds_cpl", "monitor", "dtes64", "pbe", "tm", "ht", "ss", "acpi", "ds", "vme"], "topology": {"cores": 6, "threads": 2, "sockets": 1}} 2014-02-05 16:36:46.008 28458 INFO nova.compute.manager [-] Lifecycle event 0 on VM 9b85b983-dced-4574-b14c-c72e4d92982a 2014-02-05 16:36:46.244 28458 INFO nova.compute.manager [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] During the sync_power process the instance has moved from host ctpcmp003 to host ctpcmp005 2014-02-05 16:41:09.634 28458 INFO nova.compute.manager [-] Lifecycle event 1 on VM 9b85b983-dced-4574-b14c-c72e4d92982a 2014-02-05 16:41:09.899 28458 INFO nova.compute.manager [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] During the sync_power process the instance has moved from host ctpcmp003 to host ctpcmp005 2014-02-05 16:41:11.056 AUDIT nova.compute.manager [req-0f0eaabf-9e29-4d45-88c9-20194be51d49 aaf3e92b69e04958b43348677ab7b38b 1859d80f51ff4180b591f7fe2668fd68] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Detach volume 7e2f873e-f7ba-4ee1-85ee-cee133cefdcf from mountpoint /dev/vdc 2014-02-05 16:41:11.061 WARNING nova.compute.manager [req-0f0eaabf-9e29-4d45-88c9-20194be51d49 aaf3e92b69e04958b43348677ab7b38b 1859d80f51ff4180b591f7fe2668fd68] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Detaching volume from unknown instance 2014-02-05 16:41:11.066 ERROR nova.compute.manager [req-0f0eaabf-9e29-4d45-88c9-20194be51d49 aaf3e92b69e04958b43348677ab7b38b 1859d80f51ff4180b591f7fe2668fd68] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Failed to detach volume 7e2f873e-f7ba-4ee1-85ee-cee133cefdcf from /dev/vdc 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Traceback (most recent call last): 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2983, in _detach_volume 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] mp) 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1016, in detach_volume 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] virt_dom = self._lookup_by_name(instance_name) 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2386, in _lookup_by_name 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] raise exception.InstanceNotFound(instance_id=instance_name) 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] InstanceNotFound: Instance instance-00000315 could not be found. 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] 2014-02-05 16:41:12.261 28458 ERROR nova.virt.libvirt.driver [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] During wait destroy, instance disappeared. 2014-02-05 16:41:12.872 INFO nova.virt.libvirt.driver [req-0f0eaabf-9e29-4d45-88c9-20194be51d49 aaf3e92b69e04958b43348677ab7b38b 1859d80f51ff4180b591f7fe2668fd68] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Deleting instance files /var/lib/nova/instances/9b85b983-dced-4574-b14c-c72e4d92982a /etc/nova/nova.conf includes the directive: vncserver_listen=0.0.0.0 root@ctpcmp003:~# grep "tls\|tcp" /etc/libvirt/libvirtd.conf | grep -v "^#" listen_tls = 0 listen_tcp = 1 auth_tcp = "none" root@ctpcmp003:~# grep vncserver_listen /etc/nova/nova.conf vncserver_listen=0.0.0.0 root@ctpcmp003:~# ps -ef | grep libvirtd root 20608 1 0 16:29 ? 00:00:11 /usr/sbin/libvirtd -d -l -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1276639 Title: block live migration does not work when a volume is attached Status in OpenStack Compute (Nova): New Bug description: Environment: - Two compute nodes, running Ubuntu 12.04 LTS - KVM Hypervisor - Ceph (dumpling) back-end for Cinder - Grizzly-level Openstack Steps to reproduce: 1) Create instance and volume 2) Attach volume to instance 3) Attempt a block migration between compute nodes - eg: nova live-migration --block-migrate 9b85b983-dced-4574-b14c-c72e4d92982a Packages: ii ceph 0.67.5-1precise ii ceph-common 0.67.5-1precise ii ceph-fs-common 0.67.5-1precise ii ceph-fuse 0.67.5-1precise ii ceph-mds 0.67.5-1precise ii curl 7.29.0-1precise.ceph ii kvm 1:84+dfsg-0ubuntu16+1.0+noroms+0ubuntu14.13 ii kvm-ipxe 1.0.0+git-3.55f6c88-0ubuntu1 ii libcephfs1 0.67.5-1precise ii libcurl3 7.29.0-1precise.ceph ii libcurl3-gnutls 7.29.0-1precise.ceph ii libleveldb1 1.12.0-1precise.ceph ii nova-common 1:2013.1.4-0ubuntu1~cloud0 ii nova-compute 1:2013.1.4-0ubuntu1~cloud0 ii nova-compute-kvm 1:2013.1.4-0ubuntu1~cloud0 ii python-ceph 0.67.5-1precise ii python-cinderclient 1:1.0.3-0ubuntu1~cloud0 ii python-nova 1:2013.1.4-0ubuntu1~cloud0 ii python-novaclient 1:2.13.0-0ubuntu1~cloud0 ii qemu-common 1.0+noroms-0ubuntu14.13 ii qemu-kvm 1.0+noroms-0ubuntu14.13 ii qemu-utils 1.0+noroms-0ubuntu14.13 ii libvirt-bin 1.0.2-0ubuntu11.13.04.5~cloud1 ii libvirt0 1.0.2-0ubuntu11.13.04.5~cloud1 ii python-libvirt 1.0.2-0ubuntu11.13.04.5~cloud1 /var/log/nova/nova-compute on source: 2014-02-05 16:36:46.014 998 INFO nova.compute.manager [-] Lifecycle event 2 on VM 9b85b983-dced-4574-b14c-c72e4d92982a 2014-02-05 16:36:46.233 998 INFO nova.compute.manager [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] During sync_power_state the instance has a pending task. Skip. 2014-02-05 16:36:46.234 998 INFO nova.compute.manager [-] Lifecycle event 2 on VM 9b85b983-dced-4574-b14c-c72e4d92982a 2014-02-05 16:36:46.468 998 INFO nova.compute.manager [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] During sync_power_state the instance has a pending task. Skip. 2014-02-05 16:41:09.029 998 INFO nova.compute.manager [-] Lifecycle event 1 on VM 9b85b983-dced-4574-b14c-c72e4d92982a 2014-02-05 16:41:09.265 998 INFO nova.compute.manager [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] During sync_power_state the instance has a pending task. Skip. 2014-02-05 16:41:09.640 998 ERROR nova.virt.libvirt.driver [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Live Migration failure: Unable to read from monitor: Connection reset by peer 2014-02-05 16:41:12.165 998 WARNING nova.compute.manager [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Instance shutdown by itself. Calling the stop API. 2014-02-05 16:41:12.398 998 INFO nova.virt.libvirt.driver [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Instance destroyed successfully. /var/log/libvirt/libvirtd.log on source: 2014-02-05 14:41:07.607+0000: 3437: error : qemuMonitorIORead:502 : Unable to read from monitor: Connection reset by peer 2014-02-05 14:41:09.633+0000: 3441: error : virNetClientProgramDispatchError:175 : An error occurred, but the cause is unknown 2014-02-05 14:41:09.634+0000: 3441: error : qemuDomainObjEnterMonitorInternal:997 : operation failed: domain is no longer running 2014-02-05 14:41:09.634+0000: 3441: warning : doPeer2PeerMigrate3:2872 : Guest instance-00000315 probably left in 'paused' state on source /var/log/nova/nova-compute.log on target: 2014-02-05 16:36:38.841 INFO nova.virt.libvirt.driver [req-0f0eaabf-9e29-4d45-88c9-20194be51d49 aaf3e92b69e04958b43348677ab7b38b 1859d80f51ff4180b591f7fe2668fd68] Instance launched has CPU info: {"vendor": "Intel", "model": "SandyBridge", "arch": "x86_64", "features": ["pdpe1gb", "osxsave", "dca", "pcid", "pdcm", "xtpr", "tm2", "est", "smx", "vmx", "ds_cpl", "monitor", "dtes64", "pbe", "tm", "ht", "ss", "acpi", "ds", "vme"], "topology": {"cores": 6, "threads": 2, "sockets": 1}} 2014-02-05 16:36:46.008 28458 INFO nova.compute.manager [-] Lifecycle event 0 on VM 9b85b983-dced-4574-b14c-c72e4d92982a 2014-02-05 16:36:46.244 28458 INFO nova.compute.manager [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] During the sync_power process the instance has moved from host ctpcmp003 to host ctpcmp005 2014-02-05 16:41:09.634 28458 INFO nova.compute.manager [-] Lifecycle event 1 on VM 9b85b983-dced-4574-b14c-c72e4d92982a 2014-02-05 16:41:09.899 28458 INFO nova.compute.manager [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] During the sync_power process the instance has moved from host ctpcmp003 to host ctpcmp005 2014-02-05 16:41:11.056 AUDIT nova.compute.manager [req-0f0eaabf-9e29-4d45-88c9-20194be51d49 aaf3e92b69e04958b43348677ab7b38b 1859d80f51ff4180b591f7fe2668fd68] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Detach volume 7e2f873e-f7ba-4ee1-85ee-cee133cefdcf from mountpoint /dev/vdc 2014-02-05 16:41:11.061 WARNING nova.compute.manager [req-0f0eaabf-9e29-4d45-88c9-20194be51d49 aaf3e92b69e04958b43348677ab7b38b 1859d80f51ff4180b591f7fe2668fd68] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Detaching volume from unknown instance 2014-02-05 16:41:11.066 ERROR nova.compute.manager [req-0f0eaabf-9e29-4d45-88c9-20194be51d49 aaf3e92b69e04958b43348677ab7b38b 1859d80f51ff4180b591f7fe2668fd68] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Failed to detach volume 7e2f873e-f7ba-4ee1-85ee-cee133cefdcf from /dev/vdc 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Traceback (most recent call last): 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2983, in _detach_volume 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] mp) 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1016, in detach_volume 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] virt_dom = self._lookup_by_name(instance_name) 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2386, in _lookup_by_name 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] raise exception.InstanceNotFound(instance_id=instance_name) 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] InstanceNotFound: Instance instance-00000315 could not be found. 2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] 2014-02-05 16:41:12.261 28458 ERROR nova.virt.libvirt.driver [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] During wait destroy, instance disappeared. 2014-02-05 16:41:12.872 INFO nova.virt.libvirt.driver [req-0f0eaabf-9e29-4d45-88c9-20194be51d49 aaf3e92b69e04958b43348677ab7b38b 1859d80f51ff4180b591f7fe2668fd68] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Deleting instance files /var/lib/nova/instances/9b85b983-dced-4574-b14c-c72e4d92982a /etc/nova/nova.conf includes the directive: vncserver_listen=0.0.0.0 root@ctpcmp003:~# grep "tls\|tcp" /etc/libvirt/libvirtd.conf | grep -v "^#" listen_tls = 0 listen_tcp = 1 auth_tcp = "none" root@ctpcmp003:~# grep vncserver_listen /etc/nova/nova.conf vncserver_listen=0.0.0.0 root@ctpcmp003:~# ps -ef | grep libvirtd root 20608 1 0 16:29 ? 00:00:11 /usr/sbin/libvirtd -d -l To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1276639/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp