** Summary changed: - Masakari delays BFV instance evacuation when Cinder volume has multiattach=False on shared storage backend + [SRU] Masakari marks host recovery finished with pending VM moves after Nova or Keystone API failure
** Description changed: + [ Impact ] + + When Masakari handles a compute-host failure, a transient failure while + querying Nova or Keystone can escape from an evacuation worker before the corresponding + VM move is updated. + + The VM move remains in PENDING state, but Masakari can mark the overall + host-failure notification as finished. Operators are therefore told that + recovery completed even though an instance was not evacuated and may remain + unavailable. + + The update ensures that these failures mark the VM move and host-recovery + notification as failed instead of silently reporting successful recovery. + + [ Test Plan ] + + This test requires an OpenStack deployment with Masakari and at least two Nova + compute hosts. + + 1. Install the release version of Masakari from the Resolute archive. + + 2. Create an ACTIVE instance on the first compute host and set its + HA_Enabled property to True. + + 3. While processing a compute-host failure notification, make the Nova + server-detail request for that instance return HTTP 503. + + 4. Confirm the existing bug: + + * the Masakari notification reaches FINISHED; + * the instance has not been evacuated; and + * the corresponding vmove remains PENDING without completion timestamps. + + 5. Restore the compute host and Masakari host state. + + 6. Install the Masakari packages from resolute-proposed and confirm the + installed binary package version. + + 7. Repeat steps 2 through 4 with the same injected Nova failure. + + 8. Confirm the corrected behaviour: + + * the Masakari notification reaches FAILED; + * the associated vmove records reach FAILED; + * failed vmoves have an end time and error message; and + * no vmove associated with the notification remains PENDING. + + 9. Remove the injected Nova failure and perform another host evacuation with + an ACTIVE HA-enabled instance. + + 10. Confirm that the normal evacuation succeeds: + + * the notification reaches FINISHED; + * the vmove reaches SUCCEEDED; + * Nova reports the instance ACTIVE on the second compute; + * the instance is running under libvirt on the second compute; + * its Neutron port is ACTIVE and bound to the second compute; and + * both compute services can be restored to the up/enabled state. + + [ Where problems could occur ] + + The change affects the compute-host recovery path, including instance lookup, + instance locking and unlocking, and the final determination of whether host + recovery succeeded. + + An error in this logic could incorrectly fail a successful evacuation, leave + an instance locked, or allow a failed evacuation to be reported as successful. + + The added pending-vmove check could also expose an existing workflow or + persistence race as a failed notification. The test plan therefore covers both + the error path and a normal successful evacuation between two compute hosts. + + [ Other Info ] + + Upstream fix: + + https://opendev.org/openstack/masakari/commit/4e6dd2e645d71439eb289a1523a5d04c402e90bb + + The upstream source containing this fix uses a newer + oslo.versionedobjects decorator API than the version available in Resolute. + The backport retains Resolute's existing decorator API and includes only the + behavioural changes required to fix this bug. + + --- previous description ## Summary Masakari fails to evacuate boot-from-volume (BFV) instances promptly during host failure when the Cinder volume type has `multiattach=False`. The evacuation is delayed by ~13 minutes before being re-queued, and completes only after the source hypervisor recovers and clears the iSCSI session, or after manual Cinder volume state reset. ## Environment - **Deployment method:** OpenStack Sunbeam - **OpenStack release:** 2024.1 (Nova 29.2.0) - **Masakari:** 2024.1/stable (masakari-k8s charm rev 137) - **Cinder backend:** Pure Storage FA iSCSI (PureISCSIDriver) - **Volume multiattach:** False (default) - **Cluster:** 3-node, all nodes are control + compute + storage ## Steps to reproduce 1. Create a BFV instance using a Cinder volume with `multiattach=False` on a Pure Storage iSCSI backend 2. Configure Masakari with a failover segment covering the host 3. Hard power-off the host (IPMI/BMC hard power cut) 4. Observe Masakari's evacuation behavior ## Observed behavior Masakari's `EvacuateInstancesTask` logs `"Evacuation of instance started"` for all instances, including BFV ones, at T+0. However: - **Ephemeral instances (4/6):** evacuated successfully within 2min 26s - **BFV instances (2/6):** no Nova server event recorded for ~13 minutes after Masakari triggered evacuation. Nova migration records are only created at T+13min. Actual rebuild completed in under 1 minute once Nova accepted the request. Total time from host failure to BFV instance availability on destination: **~37 minutes** (13min delay + wait for source node recovery to clear iSCSI session when powering on the host). ## Evidence **Masakari notification detail (EvacuateInstancesTask):** ``` 19:24:20 - "Evacuation of instance started: 'caad8b3d-c2c3-43ec-845f-7c5f056ebcd4'" 19:24:20 - "Evacuation of instance started: '4efcc1bf-8c3b-4467-8f75-420ff0b3863a'" 19:26:47 - "Evacuation process completed!" ``` **Nova server events (`openstack server event list --long`):** - No evacuation event recorded between 19:24:20 and 19:37:51 for either BFV VM - First Nova evacuation event: 19:37:51 (caad8b3d) and 19:37:59 (4efcc1bf) - compute_rebuild_instance completed: 19:38:21 and 19:38:55 (~1min after Nova accepted) **Volume state:** ``` $ openstack volume show 35ab2ef8-5508-4401-bff4-06c2f119c225 | grep multiattach | multiattach | False | ``` ## Note on log availability The Masakari, Nova conductor, Nova API, and Cinder API pod logs covering the critical 19:24–19:37 window were no longer available at the time of the investigation. The failed node was rebooted as part of the test recovery procedure, resulting in the loss of all pod logs on that node. The surviving nodes' pod logs had also rotated by the time the investigation was conducted (~2 days after the test). As a result, it was not possible to confirm from the logs whether: - Masakari submitted the initial evacuation request to Nova and received an error response (e.g., volume `in-use`, Nova rejecting the evacuate call), or - Masakari's internal evacuation loop detected the volume issue and deferred the request before reaching Nova The ~13-minute gap between Masakari's "Evacuation of instance started" log (19:24:20) and the first Nova server event (19:37:51) is consistent with Masakari's `FixedIntervalWithTimeoutLoopingCall` polling at `verify_interval=60s` - suggesting approximately 13 polling cycles elapsed before Masakari re-submitted the evacuation. The exact failure mode during the first attempt could not be confirmed due to log rotation. To reproduce and capture the full log sequence, the test should be repeated with extended log retention and pod log `--previous` capture immediately after the event. ## Root cause analysis Masakari's `EvacuateInstancesTask` calls Nova's evacuate API for all instances without first checking whether instances are boot-from-volume and whether their Cinder volumes need to be force-detached from the dead host. With `multiattach=False`, Nova cannot attach the volume to the destination host until it has detached it from the source. Since the source host is powered off, Nova cannot cleanly detach the iSCSI session - it requires either: 1. The source hypervisor to come back online and clear the session, or 2. A `os-force_detach` call to Cinder to forcibly clear the volume attachment Masakari currently does neither. It calls Nova's evacuate API and then polls for confirmation (`FixedIntervalWithTimeoutLoopingCall` with `verify_interval=60s`). When Nova cannot complete the evacuation (volume still `in-use`), Masakari eventually re-queues the evacuation after its internal retry logic (~13 minutes in our test), at which point it succeeds because enough time has passed for the volume state to change. ## Expected behavior Masakari should detect BFV instances and call `os-force_detach` on their Cinder volumes before or during the evacuation call to the Nova API. Since the backend is shared storage (Pure Storage FA, Ceph, etc.), the storage array can terminate the dead host's iSCSI/RBD session without requiring the hypervisor to be alive. This would allow Nova to complete the evacuation immediately, reducing BFV evacuation time from ~37 minutes to under 2 minutes (matching ephemeral instances). ## Workaround Manually reset Cinder volume state immediately after host power-off: ```bash openstack volume set --state available <volume-id> ``` Or enable `multiattach` on the Pure Storage volume types, which allows Nova to attach to the destination before detaching from the source: ```bash openstack volume type set --property multiattach='<is> True' <volume-type> ``` ## Impact Any Masakari deployment that uses BFV instances with `multiattach=False` on a shared storage backend (Pure Storage, NetApp, etc.) will experience significantly delayed evacuation in the event of a hard host failure. This makes Masakari HA effectively unusable for BFV workloads without the multiattach workaround. ## References - https://canonical-openstack.readthedocs-hosted.com/en/latest/how- to/features/instance-recovery/#supplementary-information ** Changed in: masakari (Ubuntu Resolute) Status: New => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2158101 Title: [SRU] Masakari marks host recovery finished with pending VM moves after Nova or Keystone API failure To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/caracal/+bug/2158101/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
