Re: [openstack-dev] [openstack-ansible] Live migration issue
Thanks for reply. But « a priori » log say « this error can be safely ignore ». And therefore, this log comes from the live migration that succeeded (compute 2 to compute 1). The ERROR that questions me is (live migration compute 1 to 2), on compute 1: 2017-01-25 11:03:58.475 113231 ERROR nova.virt.libvirt.driver [req-7bd352bf-8818-4f71-9fa0-04fabccebf9c 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Live Migration failure: Requested operation is not valid: domain 'instance-0187' is already active > Le 25 janv. 2017 à 13:24, Lenny Verkhovsky <len...@mellanox.com> a écrit : > > Hi, > > What domain name are you using? > Check for 'Traceback' and ' ERROR ' in the logs, maybe you will get a hint > > > 2017-01-25 11:00:21.215 28309 INFO nova.compute.manager > [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 > 7531f209e3514e3f98eb58aafa480285 - - -] [instance: > c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] You may see the error "libvirt: QEMU > error: Domain not found: no domain with matching name." This error can be > safely ignored. > > > -Original Message- > From: fabrice grelaud [mailto:fabrice.grel...@u-bordeaux.fr] > Sent: Wednesday, January 25, 2017 1:06 PM > To: OpenStack Development Mailing List (not for usage questions) > <openstack-dev@lists.openstack.org> > Subject: [openstack-dev] [openstack-ansible] Live migration issue > > Hi osa team, > > i ‘ve got live migration issue in one direction but not in other. > I deploy openstack with OSA, ubuntu trusty, stable/newton branch, 14.0.5 tag. > > My 2 compute node are same host type and have nova-compute and cinder-volume > (our ceph cluster as backend) services. > > No problem to live migrate instance from Compute 2 to Compute 1 whereas the > reverse is not true. > See log below: > > Live migration instance Compute 2 to 1: OK > > Compute 2 log > 2017-01-25 11:00:15.621 28309 INFO nova.virt.libvirt.migration > [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 > 7531f209e3514e3f98eb58aafa480285 - - -] [instance: > c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Increasing downtime to 46 ms after 0 > sec elapsed time > 2017-01-25 11:00:15.787 28309 INFO nova.virt.libvirt.driver > [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 > 7531f209e3514e3f98eb58aafa480285 - - -] [instance: > c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Migration running for 0 secs, memory > 100% remaining; (bytes processed=0, remaining=0, total=0) > 2017-01-25 11:00:17.737 28309 INFO nova.compute.manager [-] [instance: > c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] VM Paused (Lifecycle Event) > 2017-01-25 11:00:17.794 28309 INFO nova.virt.libvirt.driver > [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 > 7531f209e3514e3f98eb58aafa480285 - - -] [instance: > c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Migration operation has completed > 2017-01-25 11:00:17.795 28309 INFO nova.compute.manager > [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 > 7531f209e3514e3f98eb58aafa480285 - - -] [instance: > c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] _post_live_migration() is started.. > 2017-01-25 11:00:17.815 28309 INFO oslo.privsep.daemon > [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 > 7531f209e3514e3f98eb58aafa480285 - - -] Running privsep helper: ['sudo', > 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', > '--config-file', '/etc/nova/nova.conf', '--privsep_context', > 'os_brick.privileged.default', '--privsep_sock_path', > '/tmp/tmpfL96lI/privsep.sock'] > 2017-01-25 11:00:18.387 28309 INFO oslo.privsep.daemon > [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 > 7531f209e3514e3f98eb58aafa480285 - - -] Spawned new privsep daemon via > rootwrap > 2017-01-25 11:00:18.395 28309 INFO oslo.privsep.daemon [-] privsep daemon > starting > 2017-01-25 11:00:18.396 28309 INFO oslo.privsep.daemon [-] privsep process > running with uid/gid: 0/0 > 2017-01-25 11:00:18.396 28309 INFO oslo.privsep.daemon [-] privsep process > running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none > 2017-01-25 11:00:18.396 28309 INFO oslo.privsep.daemon [-] privsep daemon > running as pid 28815 > 2017-01-25 11:00:18.397 28309 INFO nova.compute.manager > [req-aa0997d7-bf5f-480f-abc5-beadd2d03409 - - - - -] [instance: > c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] During sync_power_state the instance > has a pending task (migrating). Skip. > 2017-01-25 11:00:18.538 28309 INFO nova.compute.manager > [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] R
Re: [openstack-dev] [openstack-ansible] Live migration issue
Hi, What domain name are you using? Check for 'Traceback' and ' ERROR ' in the logs, maybe you will get a hint 2017-01-25 11:00:21.215 28309 INFO nova.compute.manager [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] You may see the error "libvirt: QEMU error: Domain not found: no domain with matching name." This error can be safely ignored. -Original Message- From: fabrice grelaud [mailto:fabrice.grel...@u-bordeaux.fr] Sent: Wednesday, January 25, 2017 1:06 PM To: OpenStack Development Mailing List (not for usage questions) <openstack-dev@lists.openstack.org> Subject: [openstack-dev] [openstack-ansible] Live migration issue Hi osa team, i ‘ve got live migration issue in one direction but not in other. I deploy openstack with OSA, ubuntu trusty, stable/newton branch, 14.0.5 tag. My 2 compute node are same host type and have nova-compute and cinder-volume (our ceph cluster as backend) services. No problem to live migrate instance from Compute 2 to Compute 1 whereas the reverse is not true. See log below: Live migration instance Compute 2 to 1: OK Compute 2 log 2017-01-25 11:00:15.621 28309 INFO nova.virt.libvirt.migration [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Increasing downtime to 46 ms after 0 sec elapsed time 2017-01-25 11:00:15.787 28309 INFO nova.virt.libvirt.driver [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Migration running for 0 secs, memory 100% remaining; (bytes processed=0, remaining=0, total=0) 2017-01-25 11:00:17.737 28309 INFO nova.compute.manager [-] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] VM Paused (Lifecycle Event) 2017-01-25 11:00:17.794 28309 INFO nova.virt.libvirt.driver [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Migration operation has completed 2017-01-25 11:00:17.795 28309 INFO nova.compute.manager [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] _post_live_migration() is started.. 2017-01-25 11:00:17.815 28309 INFO oslo.privsep.daemon [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmpfL96lI/privsep.sock'] 2017-01-25 11:00:18.387 28309 INFO oslo.privsep.daemon [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Spawned new privsep daemon via rootwrap 2017-01-25 11:00:18.395 28309 INFO oslo.privsep.daemon [-] privsep daemon starting 2017-01-25 11:00:18.396 28309 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2017-01-25 11:00:18.396 28309 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none 2017-01-25 11:00:18.396 28309 INFO oslo.privsep.daemon [-] privsep daemon running as pid 28815 2017-01-25 11:00:18.397 28309 INFO nova.compute.manager [req-aa0997d7-bf5f-480f-abc5-beadd2d03409 - - - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] During sync_power_state the instance has a pending task (migrating). Skip. 2017-01-25 11:00:18.538 28309 INFO nova.compute.manager [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] Running instance usage audit for host p-oscompute02 from 2017-01-25 09:00:00 to 2017-01-25 10:00:00. 2 instances. 2017-01-25 11:00:18.691 28309 INFO nova.compute.resource_tracker [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] Auditing locally available compute resources for node p-oscompute02.openstack.local 2017-01-25 11:00:19.634 28309 INFO nova.compute.resource_tracker [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] Total usable vcpus: 40, total allocated vcpus: 4 2017-01-25 11:00:19.635 28309 INFO nova.compute.resource_tracker [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] Final resource view: name=p-oscompute02.openstack.local phys_ram=128700MB used_ram=6144MB phys_disk=33493GB used_disk=22GB total_vcpus=40 used_vcpus=4 pci_stats=[] 2017-01-25 11:00:19.709 28309 INFO nova.compute.resource_tracker [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] Compute_service record updated for p-oscompute02:p-oscompute02.openstack.local 2017-01-25 11:00:20.163 28309 INFO os_vif [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978
[openstack-dev] [openstack-ansible] Live migration issue
Hi osa team, i ‘ve got live migration issue in one direction but not in other. I deploy openstack with OSA, ubuntu trusty, stable/newton branch, 14.0.5 tag. My 2 compute node are same host type and have nova-compute and cinder-volume (our ceph cluster as backend) services. No problem to live migrate instance from Compute 2 to Compute 1 whereas the reverse is not true. See log below: Live migration instance Compute 2 to 1: OK Compute 2 log 2017-01-25 11:00:15.621 28309 INFO nova.virt.libvirt.migration [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Increasing downtime to 46 ms after 0 sec elapsed time 2017-01-25 11:00:15.787 28309 INFO nova.virt.libvirt.driver [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Migration running for 0 secs, memory 100% remaining; (bytes processed=0, remaining=0, total=0) 2017-01-25 11:00:17.737 28309 INFO nova.compute.manager [-] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] VM Paused (Lifecycle Event) 2017-01-25 11:00:17.794 28309 INFO nova.virt.libvirt.driver [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Migration operation has completed 2017-01-25 11:00:17.795 28309 INFO nova.compute.manager [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] _post_live_migration() is started.. 2017-01-25 11:00:17.815 28309 INFO oslo.privsep.daemon [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmpfL96lI/privsep.sock'] 2017-01-25 11:00:18.387 28309 INFO oslo.privsep.daemon [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Spawned new privsep daemon via rootwrap 2017-01-25 11:00:18.395 28309 INFO oslo.privsep.daemon [-] privsep daemon starting 2017-01-25 11:00:18.396 28309 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2017-01-25 11:00:18.396 28309 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none 2017-01-25 11:00:18.396 28309 INFO oslo.privsep.daemon [-] privsep daemon running as pid 28815 2017-01-25 11:00:18.397 28309 INFO nova.compute.manager [req-aa0997d7-bf5f-480f-abc5-beadd2d03409 - - - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] During sync_power_state the instance has a pending task (migrating). Skip. 2017-01-25 11:00:18.538 28309 INFO nova.compute.manager [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] Running instance usage audit for host p-oscompute02 from 2017-01-25 09:00:00 to 2017-01-25 10:00:00. 2 instances. 2017-01-25 11:00:18.691 28309 INFO nova.compute.resource_tracker [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] Auditing locally available compute resources for node p-oscompute02.openstack.local 2017-01-25 11:00:19.634 28309 INFO nova.compute.resource_tracker [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] Total usable vcpus: 40, total allocated vcpus: 4 2017-01-25 11:00:19.635 28309 INFO nova.compute.resource_tracker [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] Final resource view: name=p-oscompute02.openstack.local phys_ram=128700MB used_ram=6144MB phys_disk=33493GB used_disk=22GB total_vcpus=40 used_vcpus=4 pci_stats=[] 2017-01-25 11:00:19.709 28309 INFO nova.compute.resource_tracker [req-115a99b8-48ef-43d5-908b-5ff7aadc3df4 - - - - -] Compute_service record updated for p-oscompute02:p-oscompute02.openstack.local 2017-01-25 11:00:20.163 28309 INFO os_vif [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] Successfully unplugged vif VIFBridge(active=True,address=fa:16:3e:d2:7c:83,bridge_name='brqc434ace8-45',has_traffic_filtering=True,id=dff20b91-a654-437d-8a74-dc55aeac8ab7,network=Network(c434ace8-45f6-4bb1-95bc-d52dadb557c7),plugin='linux_bridge',port_profile=,preserve_on_delete=False,vif_name='tapdff20b91-a6') 2017-01-25 11:00:20.201 28309 INFO nova.virt.libvirt.driver [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470 0329776bd1634978a7fed35a70c77479 7531f209e3514e3f98eb58aafa480285 - - -] [instance: c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3] Deleting instance files /var/lib/nova/instances/c7a5e5d1-bc22-4143-a85a-ee5c3b6777b3_del 2017-01-25 11:00:20.202 28309 INFO nova.virt.libvirt.driver [req-6f21e4a4-28a8-48e3-bf2f-2e1ad3b52470
