GitHub user muthukrishnang1100 added a comment to the discussion: VM HA not
working when there's an issue with vm restart
Hi @kiranchavala
Thank you for checking.
Our environment is different from the original NFS-primary-storage case.
Environment
Apache CloudStack: `4.20.3.0`
Hypervisor: KVM on Ubuntu 24.04
Management servers: 3-node management cluster
Database: 3-node MariaDB Galera cluster behind HAProxy
VM primary storage: Ceph RBD
Ceph pool for VM disks: `cloudstack-primary`
Ceph health during the test: `HEALTH_OK`
QEMU version on all compute hosts:
```text
QEMU emulator version 8.2.2
(Debian 1:8.2.2+ds-0ubuntu1.16)
```
Important: NFS is not used as VM primary storage in our environment.
NFS usage
NFSv4.2 is used only for the KVM Host HA heartbeat path.
Example heartbeat mount from compute hosts:
```text
/mnt/<heartbeat-mount-id> from
<management-server-ip>:/var/cloudstack-ha-heartbeat
Flags:
rw,nosuid,nodev,noexec,relatime,vers=4.2,rsize=1048576,wsize=1048576,hard,proto=tcp,timeo=600,retrans=2,sec=sys,local_lock=none
```
We also use NFSv4.2 for CloudStack secondary storage. However, all user VM root
and data disks are on Ceph RBD.
Before adding the dedicated NFS heartbeat storage, Host HA did not trigger
correctly with Ceph RBD primary storage. After configuring the NFS heartbeat
storage, Host HA detection and fencing are working correctly.
Relevant HA settings
```text
vm.ha.enabled = true
vm.ha.alerts.enabled = true
ha.workers = 5
enable.ha.storage.migration = true
ha.fence.builders.exclude = RecreatableFencer
ha.investigators.order = KVMInvestigator, PingInvestigator, SimpleInvestigator
ha.max.concurrent.activity.check.operations = 25
ha.max.concurrent.fence.operations = 25
ha.max.concurrent.health.check.operations = 50
ha.max.concurrent.recovery.operations = 25
kvm.ha.activity.check.failure.ratio = 0.7
kvm.ha.activity.check.interval = 60
kvm.ha.activity.check.max.attempts = 10
kvm.ha.activity.check.timeout = 60
kvm.ha.degraded.max.period = 300
kvm.ha.fence.on.storage.heartbeat.failure = true
kvm.ha.fence.timeout = 60
kvm.ha.health.check.timeout = 10
kvm.ha.recover.failure.threshold = 1
kvm.ha.recover.timeout = 60
kvm.ha.recover.wait.period = 600
ha.vm.restart.hostup = true
vm.ha.migration.max.retries = 5
```
Test performed
We performed repeated ungraceful host-failure tests by hard-powering off a KVM
compute host through OOBM/IPMI.
Example failed host:
```text
Host name: ltchl1pacscom04
CloudStack Host ID: 19
Host IP: <redacted>
```
At the time of one test, there were 9 user VMs running on this host.
Expected behavior:
```text
Host Down
-> fencing succeeds
-> all VMs restart on another KVM host
```
Actual behavior:
```text
Host Down
-> KVM fencing succeeds
-> most VMs restart successfully on other hosts
-> a few VMs remain permanently in Stopping state
-> HA work later becomes Done
-> manual recovery is required
```
For example, during one 9-VM test, 6 VMs restarted successfully and the
following VMs remained in `Stopping`:
```text
i-22-248-VM
i-22-257-VM
i-22-258-VM
```
Management-server log observations
CloudStack correctly identifies the failed host as Down and KVM fencing
succeeds:
```text
Fencer KVMFenceBuilder returned true
```
For an affected VM, the HA stop/network cleanup workflow encountered a database
transaction failure:
```text
Unable to release some network resources for the VM in Stopping state
com.cloud.utils.exception.CloudRuntimeException:
Unable to commit or close the connection.
Caused by:
com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException:
Deadlock found when trying to get lock; try restarting transaction
```
For other affected VMs, HA logs:
```text
Encountered unhandled exception during HA process, reschedule work
com.cloud.utils.exception.CloudRuntimeException:
Unable to find by id on DB, due to:
Deadlock found when trying to get lock; try restarting transaction
```
The HA work is initially rescheduled. During the next HA attempt, CloudStack
detects that the VM state has changed from `Running` to `Stopping`:
```text
HA on VM instance {... state="Stopping" ...}
VM ... has been changed.
Current State = Stopping
Previous State = Running
```
Then the HA work is completed:
```text
Completed work HAWork[...]
```
However, the VM remains in `Stopping` and is not restarted on another host.
Database observation
For the affected VMs, the database shows:
```text
vm_instance.state = Stopping
vm_instance.host_id = failed host ID
Latest op_ha_work:
step = Done
```
The MariaDB Galera cluster was healthy after the test:
```text
wsrep_cluster_status = Primary
wsrep_cluster_size = 3
wsrep_connected = ON
wsrep_ready = ON
wsrep_local_state_comment = Synced
```
Current counters on one database node were:
```text
Innodb_deadlocks = 0
wsrep_local_bf_aborts = 68
wsrep_local_cert_failures = 75
wsrep_local_replays = 4
```
These Galera counters are cumulative and were not captured before and
immediately after every HA test, so we cannot confirm that they are directly
caused by this specific HA event. However, the CloudStack management-server
logs clearly show `MySQLTransactionRollbackException: Deadlock found when
trying to get lock` during the affected HA recovery workflow.
Ceph RBD lock observation
For VMs that restarted successfully, the RBD exclusive locks moved from the
failed host to the destination compute hosts.
For VMs stuck in `Stopping`, their root-volume RBD locks remained on the failed
source host. Example:
```text
rbd lock ls cloudstack-primary/<rbd-image>
There is 1 exclusive lock on this image.
Locker ID Address
client.<id> auto <id>
<failed-compute-host-ip>:0/<client-session>
```
Since the source host was hard powered off, this stale RBD lock is expected.
However, CloudStack does not complete the HA recovery because the VM remains in
`Stopping` and the HA work becomes `Done`.
Comparison with issue #13510
I understand that the original issue uses shared NFSv4.2 primary storage and
the first restart failure is caused by a QEMU/NFS write-lock error.
Our initial HA failure appears different:
```text
Issue #13510:
QEMU cannot acquire NFS write lock
-> initial restart fails
-> HA retry detects update counter mismatch
-> HA work completes without retry
Our environment:
CloudStack HA stop/network cleanup encounters DB transaction deadlock
-> VM remains Stopping
-> HA retry detects changed VM state/update state
-> HA work completes without successful restart
```
The storage backend is different, but the final behavior appears similar: after
an initial HA recovery failure, CloudStack stops retrying and completes the HA
work even though the VM remains unavailable.
GitHub link:
https://github.com/apache/cloudstack/discussions/13729#discussioncomment-17810565
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
