[Yahoo-eng-team] [Bug 1662626] Re: live-migrate left in migrating as domain not found

2017-10-19 Thread Corey Bryant 
** Also affects: cloud-archive/newton
   Importance: Undecided
   Status: New

** Also affects: cloud-archive/ocata
   Importance: Undecided
   Status: New

** Also affects: cloud-archive/pike
   Importance: Medium
 Assignee: Shane Peters (shaner)
   Status: Fix Released

** Changed in: cloud-archive/newton
   Importance: Undecided => Medium

** Changed in: cloud-archive/newton
   Status: New => Fix Released

** Changed in: cloud-archive/ocata
   Importance: Undecided => Medium

** Changed in: cloud-archive/ocata
   Status: New => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1662626

Title:
  live-migrate left in migrating as domain not found

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive mitaka series:
  Triaged
Status in Ubuntu Cloud Archive newton series:
  Fix Released
Status in Ubuntu Cloud Archive ocata series:
  Fix Released
Status in Ubuntu Cloud Archive pike series:
  Fix Released
Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) newton series:
  Fix Committed
Status in OpenStack Compute (nova) ocata series:
  Fix Committed
Status in nova package in Ubuntu:
  Fix Released
Status in nova source package in Xenial:
  Triaged
Status in nova source package in Zesty:
  Fix Released
Status in nova source package in Artful:
  Fix Released

Bug description:
  A live-migration stress test was working fine when suddenly a VM
  stopped migrating. It failed with this error:

  ERROR nova.virt.libvirt.driver [req-df91ac40-820f-4aa9-945b-
  b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c
  669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-
  9c2c-8a792aed3d6b] Error from libvirt during undefine. Code=42
  Error=Domain not found: no domain with matching uuid '62034d78-3144
  -4efd-9c2c-8a792aed3d6b' (instance-0431)

  The full stack trace:

  2017-02-05 02:33:41.787 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Migration running for 240 secs, memory 9% 
remaining; (bytes processed=15198240264, remaining=1680875520, 
total=17314955264)
  2017-02-05 02:33:45.795 19770 INFO nova.compute.manager 
[req-abff9c69-5f82-4ed6-af8a-fd1dc81a72a6 - - - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] VM Paused (Lifecycle Event)
  2017-02-05 02:33:45.870 19770 INFO nova.compute.manager 
[req-abff9c69-5f82-4ed6-af8a-fd1dc81a72a6 - - - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] During sync_power_state the instance has 
a pending task (migrating). Skip.
  2017-02-05 02:33:45.883 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Migration operation has completed
  2017-02-05 02:33:45.884 19770 INFO nova.compute.manager 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] _post_live_migration() is started..
  2017-02-05 02:33:46.156 19770 INFO os_vif 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] Successfully unplugged vif 
VIFBridge(active=True,address=fa:16:3e:a2:90:55,bridge_name='brq476ab6ba-b3',has_traffic_filtering=True,id=98d476b3-0ead-4adb-ad54-1dff63edcd65,network=Network(476ab6ba-b32e-409e-9711-9412e8475ea0),plugin='linux_bridge',port_profile=,preserve_on_delete=True,vif_name='tap98d476b3-0e')
  2017-02-05 02:33:46.189 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Deleting instance files 
/var/lib/nova/instances/62034d78-3144-4efd-9c2c-8a792aed3d6b_del
  2017-02-05 02:33:46.195 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Deletion of 
/var/lib/nova/instances/62034d78-3144-4efd-9c2c-8a792aed3d6b_del complete

  2017-02-05 02:33:46.334 19770 ERROR nova.virt.libvirt.driver [req-
  df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c
  669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-
  9c2c-8a792aed3d6b] Error from libvirt during undefine. Code=42
  Error=Domain not found: no domain with matching uuid '62034d78-3144
  -4efd-9c2c-8a792aed3d6b' (instance-0431)

  2017-02-05 02:33:46.363 19770 WARNING nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c

[Yahoo-eng-team] [Bug 1662626] Re: live-migrate left in migrating as domain not found

2017-10-19 Thread Corey Bryant 
** Changed in: nova (Ubuntu Zesty)
   Status: New => Fix Released

** Changed in: nova (Ubuntu Xenial)
   Status: New => Triaged

** Also affects: cloud-archive/mitaka
   Importance: Undecided
   Status: New

** Changed in: cloud-archive/mitaka
   Status: New => Triaged

** Changed in: cloud-archive/mitaka
   Importance: Undecided => Medium

** Changed in: nova (Ubuntu Xenial)
   Importance: Undecided => Medium

** Changed in: nova (Ubuntu Zesty)
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1662626

Title:
  live-migrate left in migrating as domain not found

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive mitaka series:
  Triaged
Status in Ubuntu Cloud Archive newton series:
  Fix Released
Status in Ubuntu Cloud Archive ocata series:
  Fix Released
Status in Ubuntu Cloud Archive pike series:
  Fix Released
Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) newton series:
  Fix Committed
Status in OpenStack Compute (nova) ocata series:
  Fix Committed
Status in nova package in Ubuntu:
  Fix Released
Status in nova source package in Xenial:
  Triaged
Status in nova source package in Zesty:
  Fix Released
Status in nova source package in Artful:
  Fix Released

Bug description:
  A live-migration stress test was working fine when suddenly a VM
  stopped migrating. It failed with this error:

  ERROR nova.virt.libvirt.driver [req-df91ac40-820f-4aa9-945b-
  b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c
  669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-
  9c2c-8a792aed3d6b] Error from libvirt during undefine. Code=42
  Error=Domain not found: no domain with matching uuid '62034d78-3144
  -4efd-9c2c-8a792aed3d6b' (instance-0431)

  The full stack trace:

  2017-02-05 02:33:41.787 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Migration running for 240 secs, memory 9% 
remaining; (bytes processed=15198240264, remaining=1680875520, 
total=17314955264)
  2017-02-05 02:33:45.795 19770 INFO nova.compute.manager 
[req-abff9c69-5f82-4ed6-af8a-fd1dc81a72a6 - - - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] VM Paused (Lifecycle Event)
  2017-02-05 02:33:45.870 19770 INFO nova.compute.manager 
[req-abff9c69-5f82-4ed6-af8a-fd1dc81a72a6 - - - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] During sync_power_state the instance has 
a pending task (migrating). Skip.
  2017-02-05 02:33:45.883 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Migration operation has completed
  2017-02-05 02:33:45.884 19770 INFO nova.compute.manager 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] _post_live_migration() is started..
  2017-02-05 02:33:46.156 19770 INFO os_vif 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] Successfully unplugged vif 
VIFBridge(active=True,address=fa:16:3e:a2:90:55,bridge_name='brq476ab6ba-b3',has_traffic_filtering=True,id=98d476b3-0ead-4adb-ad54-1dff63edcd65,network=Network(476ab6ba-b32e-409e-9711-9412e8475ea0),plugin='linux_bridge',port_profile=,preserve_on_delete=True,vif_name='tap98d476b3-0e')
  2017-02-05 02:33:46.189 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Deleting instance files 
/var/lib/nova/instances/62034d78-3144-4efd-9c2c-8a792aed3d6b_del
  2017-02-05 02:33:46.195 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Deletion of 
/var/lib/nova/instances/62034d78-3144-4efd-9c2c-8a792aed3d6b_del complete

  2017-02-05 02:33:46.334 19770 ERROR nova.virt.libvirt.driver [req-
  df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c
  669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-
  9c2c-8a792aed3d6b] Error from libvirt during undefine. Code=42
  Error=Domain not found: no domain with matching uuid '62034d78-3144
  -4efd-9c2c-8a792aed3d6b' (instance-0431)

  2017-02-05 02:33:46.363 19770 WARNING nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b]

[Yahoo-eng-team] [Bug 1662626] Re: live-migrate left in migrating as domain not found

2017-10-19 Thread Corey Bryant 
** Changed in: cloud-archive
   Status: Confirmed => Fix Released

** Also affects: nova (Ubuntu)
   Importance: Undecided
   Status: New

** Changed in: nova (Ubuntu)
   Status: New => Fix Released

** Changed in: nova (Ubuntu)
   Importance: Undecided => Medium

** Also affects: nova (Ubuntu Artful)
   Importance: Medium
   Status: Fix Released

** Also affects: nova (Ubuntu Zesty)
   Importance: Undecided
   Status: New

** Also affects: nova (Ubuntu Xenial)
   Importance: Undecided
   Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1662626

Title:
  live-migrate left in migrating as domain not found

Status in Ubuntu Cloud Archive:
  Fix Released
Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) newton series:
  Fix Committed
Status in OpenStack Compute (nova) ocata series:
  Fix Committed
Status in nova package in Ubuntu:
  Fix Released
Status in nova source package in Xenial:
  New
Status in nova source package in Zesty:
  New
Status in nova source package in Artful:
  Fix Released

Bug description:
  A live-migration stress test was working fine when suddenly a VM
  stopped migrating. It failed with this error:

  ERROR nova.virt.libvirt.driver [req-df91ac40-820f-4aa9-945b-
  b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c
  669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-
  9c2c-8a792aed3d6b] Error from libvirt during undefine. Code=42
  Error=Domain not found: no domain with matching uuid '62034d78-3144
  -4efd-9c2c-8a792aed3d6b' (instance-0431)

  The full stack trace:

  2017-02-05 02:33:41.787 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Migration running for 240 secs, memory 9% 
remaining; (bytes processed=15198240264, remaining=1680875520, 
total=17314955264)
  2017-02-05 02:33:45.795 19770 INFO nova.compute.manager 
[req-abff9c69-5f82-4ed6-af8a-fd1dc81a72a6 - - - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] VM Paused (Lifecycle Event)
  2017-02-05 02:33:45.870 19770 INFO nova.compute.manager 
[req-abff9c69-5f82-4ed6-af8a-fd1dc81a72a6 - - - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] During sync_power_state the instance has 
a pending task (migrating). Skip.
  2017-02-05 02:33:45.883 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Migration operation has completed
  2017-02-05 02:33:45.884 19770 INFO nova.compute.manager 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] _post_live_migration() is started..
  2017-02-05 02:33:46.156 19770 INFO os_vif 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] Successfully unplugged vif 
VIFBridge(active=True,address=fa:16:3e:a2:90:55,bridge_name='brq476ab6ba-b3',has_traffic_filtering=True,id=98d476b3-0ead-4adb-ad54-1dff63edcd65,network=Network(476ab6ba-b32e-409e-9711-9412e8475ea0),plugin='linux_bridge',port_profile=,preserve_on_delete=True,vif_name='tap98d476b3-0e')
  2017-02-05 02:33:46.189 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Deleting instance files 
/var/lib/nova/instances/62034d78-3144-4efd-9c2c-8a792aed3d6b_del
  2017-02-05 02:33:46.195 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Deletion of 
/var/lib/nova/instances/62034d78-3144-4efd-9c2c-8a792aed3d6b_del complete

  2017-02-05 02:33:46.334 19770 ERROR nova.virt.libvirt.driver [req-
  df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c
  669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-
  9c2c-8a792aed3d6b] Error from libvirt during undefine. Code=42
  Error=Domain not found: no domain with matching uuid '62034d78-3144
  -4efd-9c2c-8a792aed3d6b' (instance-0431)

  2017-02-05 02:33:46.363 19770 WARNING nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Error monitoring migration: Domain not 
found: no domain with matching uuid '62034d78-3144-4efd-9c2c-8a792aed3d6b' 
(instance-0431)
  2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance:

[Yahoo-eng-team] [Bug 1662626] Re: live-migrate left in migrating as domain not found

2017-10-05 Thread Shane Peters 
** Also affects: cloud-archive
   Importance: Undecided
   Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1662626

Title:
  live-migrate left in migrating as domain not found

Status in Ubuntu Cloud Archive:
  Confirmed
Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) newton series:
  Fix Committed
Status in OpenStack Compute (nova) ocata series:
  Fix Committed

Bug description:
  A live-migration stress test was working fine when suddenly a VM
  stopped migrating. It failed with this error:

  ERROR nova.virt.libvirt.driver [req-df91ac40-820f-4aa9-945b-
  b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c
  669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-
  9c2c-8a792aed3d6b] Error from libvirt during undefine. Code=42
  Error=Domain not found: no domain with matching uuid '62034d78-3144
  -4efd-9c2c-8a792aed3d6b' (instance-0431)

  The full stack trace:

  2017-02-05 02:33:41.787 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Migration running for 240 secs, memory 9% 
remaining; (bytes processed=15198240264, remaining=1680875520, 
total=17314955264)
  2017-02-05 02:33:45.795 19770 INFO nova.compute.manager 
[req-abff9c69-5f82-4ed6-af8a-fd1dc81a72a6 - - - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] VM Paused (Lifecycle Event)
  2017-02-05 02:33:45.870 19770 INFO nova.compute.manager 
[req-abff9c69-5f82-4ed6-af8a-fd1dc81a72a6 - - - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] During sync_power_state the instance has 
a pending task (migrating). Skip.
  2017-02-05 02:33:45.883 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Migration operation has completed
  2017-02-05 02:33:45.884 19770 INFO nova.compute.manager 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] _post_live_migration() is started..
  2017-02-05 02:33:46.156 19770 INFO os_vif 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] Successfully unplugged vif 
VIFBridge(active=True,address=fa:16:3e:a2:90:55,bridge_name='brq476ab6ba-b3',has_traffic_filtering=True,id=98d476b3-0ead-4adb-ad54-1dff63edcd65,network=Network(476ab6ba-b32e-409e-9711-9412e8475ea0),plugin='linux_bridge',port_profile=,preserve_on_delete=True,vif_name='tap98d476b3-0e')
  2017-02-05 02:33:46.189 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Deleting instance files 
/var/lib/nova/instances/62034d78-3144-4efd-9c2c-8a792aed3d6b_del
  2017-02-05 02:33:46.195 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Deletion of 
/var/lib/nova/instances/62034d78-3144-4efd-9c2c-8a792aed3d6b_del complete

  2017-02-05 02:33:46.334 19770 ERROR nova.virt.libvirt.driver [req-
  df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c
  669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-
  9c2c-8a792aed3d6b] Error from libvirt during undefine. Code=42
  Error=Domain not found: no domain with matching uuid '62034d78-3144
  -4efd-9c2c-8a792aed3d6b' (instance-0431)

  2017-02-05 02:33:46.363 19770 WARNING nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Error monitoring migration: Domain not 
found: no domain with matching uuid '62034d78-3144-4efd-9c2c-8a792aed3d6b' 
(instance-0431)
  2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Traceback (most recent call last):
  2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b]   File 
"/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
 line 6345, in _live_migration
  2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] finish_event, disk_paths)
  2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b]   File 
"/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
 line

[Yahoo-eng-team] [Bug 1662626] Re: live-migrate left in migrating as domain not found

2017-08-11 Thread Matt Riedemann 
** Changed in: nova/ocata
   Importance: Undecided => Medium

** Also affects: nova/newton
   Importance: Undecided
   Status: New

** Changed in: nova/newton
 Assignee: (unassigned) => Shane Peters (shaner)

** Changed in: nova/newton
   Status: New => In Progress

** Changed in: nova/newton
   Importance: Undecided => Medium

** Tags removed: ocata-backport-potential
** Tags added: libvirt

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1662626

Title:
  live-migrate left in migrating as domain not found

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) newton series:
  In Progress
Status in OpenStack Compute (nova) ocata series:
  Fix Committed

Bug description:
  A live-migration stress test was working fine when suddenly a VM
  stopped migrating. It failed with this error:

  ERROR nova.virt.libvirt.driver [req-df91ac40-820f-4aa9-945b-
  b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c
  669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-
  9c2c-8a792aed3d6b] Error from libvirt during undefine. Code=42
  Error=Domain not found: no domain with matching uuid '62034d78-3144
  -4efd-9c2c-8a792aed3d6b' (instance-0431)

  The full stack trace:

  2017-02-05 02:33:41.787 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Migration running for 240 secs, memory 9% 
remaining; (bytes processed=15198240264, remaining=1680875520, 
total=17314955264)
  2017-02-05 02:33:45.795 19770 INFO nova.compute.manager 
[req-abff9c69-5f82-4ed6-af8a-fd1dc81a72a6 - - - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] VM Paused (Lifecycle Event)
  2017-02-05 02:33:45.870 19770 INFO nova.compute.manager 
[req-abff9c69-5f82-4ed6-af8a-fd1dc81a72a6 - - - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] During sync_power_state the instance has 
a pending task (migrating). Skip.
  2017-02-05 02:33:45.883 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Migration operation has completed
  2017-02-05 02:33:45.884 19770 INFO nova.compute.manager 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] _post_live_migration() is started..
  2017-02-05 02:33:46.156 19770 INFO os_vif 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] Successfully unplugged vif 
VIFBridge(active=True,address=fa:16:3e:a2:90:55,bridge_name='brq476ab6ba-b3',has_traffic_filtering=True,id=98d476b3-0ead-4adb-ad54-1dff63edcd65,network=Network(476ab6ba-b32e-409e-9711-9412e8475ea0),plugin='linux_bridge',port_profile=,preserve_on_delete=True,vif_name='tap98d476b3-0e')
  2017-02-05 02:33:46.189 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Deleting instance files 
/var/lib/nova/instances/62034d78-3144-4efd-9c2c-8a792aed3d6b_del
  2017-02-05 02:33:46.195 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Deletion of 
/var/lib/nova/instances/62034d78-3144-4efd-9c2c-8a792aed3d6b_del complete

  2017-02-05 02:33:46.334 19770 ERROR nova.virt.libvirt.driver [req-
  df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c
  669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-
  9c2c-8a792aed3d6b] Error from libvirt during undefine. Code=42
  Error=Domain not found: no domain with matching uuid '62034d78-3144
  -4efd-9c2c-8a792aed3d6b' (instance-0431)

  2017-02-05 02:33:46.363 19770 WARNING nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Error monitoring migration: Domain not 
found: no domain with matching uuid '62034d78-3144-4efd-9c2c-8a792aed3d6b' 
(instance-0431)
  2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Traceback (most recent call last):
  2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b]   File 
"/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
 line 6345, in _live_migration
  2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver

[Yahoo-eng-team] [Bug 1662626] Re: live-migrate left in migrating as domain not found

2017-03-27 Thread OpenStack Infra 
Reviewed:  https://review.openstack.org/430404
Committed: 
https://git.openstack.org/cgit/openstack/nova/commit/?id=b56f8fc2d1392f4675a5baae0977e4817a362159
Submitter: Jenkins
Branch:master

commit b56f8fc2d1392f4675a5baae0977e4817a362159
Author: John Garbutt 
Date:   Tue Feb 7 19:12:50 2017 +

Stop failed live-migrates getting stuck migrating

When there are failures in driver.cleanup, we are seeing live-migrations
that get stuck in the live-migrating state. While there has been a patch
to stop the cause listed in the bug this closes, there are other
failures (such as a token timeout when talking to cinder or neutron)
that could trigger this same failure mode.

When we hit an error this late in live-migration, it should be a very
rare event, so its best to just put the instance and migration into an
error state, and help alert both the operator and API user to the
failure that has occurred.

Closes-Bug: #1662626

Change-Id: Idfdce9e7dd8106af01db0358ada15737cb846395


** Changed in: nova
   Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1662626

Title:
  live-migrate left in migrating as domain not found

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) ocata series:
  New

Bug description:
  A live-migration stress test was working fine when suddenly a VM
  stopped migrating. It failed with this error:

  ERROR nova.virt.libvirt.driver [req-df91ac40-820f-4aa9-945b-
  b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c
  669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-
  9c2c-8a792aed3d6b] Error from libvirt during undefine. Code=42
  Error=Domain not found: no domain with matching uuid '62034d78-3144
  -4efd-9c2c-8a792aed3d6b' (instance-0431)

  The full stack trace:

  2017-02-05 02:33:41.787 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Migration running for 240 secs, memory 9% 
remaining; (bytes processed=15198240264, remaining=1680875520, 
total=17314955264)
  2017-02-05 02:33:45.795 19770 INFO nova.compute.manager 
[req-abff9c69-5f82-4ed6-af8a-fd1dc81a72a6 - - - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] VM Paused (Lifecycle Event)
  2017-02-05 02:33:45.870 19770 INFO nova.compute.manager 
[req-abff9c69-5f82-4ed6-af8a-fd1dc81a72a6 - - - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] During sync_power_state the instance has 
a pending task (migrating). Skip.
  2017-02-05 02:33:45.883 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Migration operation has completed
  2017-02-05 02:33:45.884 19770 INFO nova.compute.manager 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] _post_live_migration() is started..
  2017-02-05 02:33:46.156 19770 INFO os_vif 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] Successfully unplugged vif 
VIFBridge(active=True,address=fa:16:3e:a2:90:55,bridge_name='brq476ab6ba-b3',has_traffic_filtering=True,id=98d476b3-0ead-4adb-ad54-1dff63edcd65,network=Network(476ab6ba-b32e-409e-9711-9412e8475ea0),plugin='linux_bridge',port_profile=,preserve_on_delete=True,vif_name='tap98d476b3-0e')
  2017-02-05 02:33:46.189 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Deleting instance files 
/var/lib/nova/instances/62034d78-3144-4efd-9c2c-8a792aed3d6b_del
  2017-02-05 02:33:46.195 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Deletion of 
/var/lib/nova/instances/62034d78-3144-4efd-9c2c-8a792aed3d6b_del complete

  2017-02-05 02:33:46.334 19770 ERROR nova.virt.libvirt.driver [req-
  df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c
  669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-
  9c2c-8a792aed3d6b] Error from libvirt during undefine. Code=42
  Error=Domain not found: no domain with matching uuid '62034d78-3144
  -4efd-9c2c-8a792aed3d6b' (instance-0431)

  2017-02-05 02:33:46.363 19770 WARNING nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c

[Yahoo-eng-team] [Bug 1662626] Re: live-migrate left in migrating as domain not found

2017-02-16 Thread Matt Riedemann 
** Also affects: nova/ocata
   Importance: Undecided
   Status: New

** Tags removed: ocata-rc-potential
** Tags added: ocata-backport-potential

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1662626

Title:
  live-migrate left in migrating as domain not found

Status in OpenStack Compute (nova):
  In Progress
Status in OpenStack Compute (nova) ocata series:
  New

Bug description:
  A live-migration stress test was working fine when suddenly a VM
  stopped migrating. It failed with this error:

  ERROR nova.virt.libvirt.driver [req-df91ac40-820f-4aa9-945b-
  b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c
  669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-
  9c2c-8a792aed3d6b] Error from libvirt during undefine. Code=42
  Error=Domain not found: no domain with matching uuid '62034d78-3144
  -4efd-9c2c-8a792aed3d6b' (instance-0431)

  The full stack trace:

  2017-02-05 02:33:41.787 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Migration running for 240 secs, memory 9% 
remaining; (bytes processed=15198240264, remaining=1680875520, 
total=17314955264)
  2017-02-05 02:33:45.795 19770 INFO nova.compute.manager 
[req-abff9c69-5f82-4ed6-af8a-fd1dc81a72a6 - - - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] VM Paused (Lifecycle Event)
  2017-02-05 02:33:45.870 19770 INFO nova.compute.manager 
[req-abff9c69-5f82-4ed6-af8a-fd1dc81a72a6 - - - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] During sync_power_state the instance has 
a pending task (migrating). Skip.
  2017-02-05 02:33:45.883 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Migration operation has completed
  2017-02-05 02:33:45.884 19770 INFO nova.compute.manager 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] _post_live_migration() is started..
  2017-02-05 02:33:46.156 19770 INFO os_vif 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] Successfully unplugged vif 
VIFBridge(active=True,address=fa:16:3e:a2:90:55,bridge_name='brq476ab6ba-b3',has_traffic_filtering=True,id=98d476b3-0ead-4adb-ad54-1dff63edcd65,network=Network(476ab6ba-b32e-409e-9711-9412e8475ea0),plugin='linux_bridge',port_profile=,preserve_on_delete=True,vif_name='tap98d476b3-0e')
  2017-02-05 02:33:46.189 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Deleting instance files 
/var/lib/nova/instances/62034d78-3144-4efd-9c2c-8a792aed3d6b_del
  2017-02-05 02:33:46.195 19770 INFO nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Deletion of 
/var/lib/nova/instances/62034d78-3144-4efd-9c2c-8a792aed3d6b_del complete

  2017-02-05 02:33:46.334 19770 ERROR nova.virt.libvirt.driver [req-
  df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c
  669472610b194bfa9bf03f50f86d725a - - -] [instance: 62034d78-3144-4efd-
  9c2c-8a792aed3d6b] Error from libvirt during undefine. Code=42
  Error=Domain not found: no domain with matching uuid '62034d78-3144
  -4efd-9c2c-8a792aed3d6b' (instance-0431)

  2017-02-05 02:33:46.363 19770 WARNING nova.virt.libvirt.driver 
[req-df91ac40-820f-4aa9-945b-b2fce73461f8 29c0371e35f84fdaa033f2dbfe2c042c 
669472610b194bfa9bf03f50f86d725a - - -] [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Error monitoring migration: Domain not 
found: no domain with matching uuid '62034d78-3144-4efd-9c2c-8a792aed3d6b' 
(instance-0431)
  2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] Traceback (most recent call last):
  2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b]   File 
"/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
 line 6345, in _live_migration
  2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b] finish_event, disk_paths)
  2017-02-05 02:33:46.363 19770 ERROR nova.virt.libvirt.driver [instance: 
62034d78-3144-4efd-9c2c-8a792aed3d6b]   File 
"/openstack/venvs/nova-14.0.4/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
 line 6255, in _live_migration_monitor
  2017-02-05