danivogel90 commented on issue #14326:
URL: https://github.com/apache/cloudstack/issues/14326#issuecomment-6034902680
Thanks for the quick response! I tested a fix and found two more things that
matter for the PR:
**1. The suggested fix in the issue description alone is dangerous.**
Further down, `kvmsmpheartbeat.sh` has a second check:
```bash
if grep -q "^[^ ]\+ $MountPoint " /proc/mounts
then
:
else
if [ "$rflag" == "0" ]
then
deleteVMs $MountPoint # kill -9 on every qemu process using this
path
fi
fi
```
Today this is never reached for a subdirectory pool, because the `mountpoint
-q` check exits earlier.
If only the `mountpoint -q` check is relaxed, every heartbeat write would
`kill -9` all VMs on the pool.
So both checks have to use the same logic. My patch adds a helper
`is_on_mounted_fs()`
(`findmnt -n -o TARGET -T`, fallback `df -P`). It returns true if the path
is on a mounted filesystem
other than `/`, and it replaces both checks.
**2. `check_hbLog` reports a live host as DEAD when diff == 0.**
`diff=`expr $now - $hb`` is followed by `if [ $? -ne 0 ]`, but `expr` exits
with status 1 when the result is 0.
A read in the same second as the write therefore returns `999997` →
"Considering host as DEAD".
Reproduced with the unmodified 4.23.0.0 script: write, then read immediately
with `-r -t 60` → DEAD.
The patch validates `$hb` as numeric and uses `$((now - hb))`.
**Tests** (copy of the script with `kill -9` disarmed, then live in the
agent):
| Case | Expected | Result |
|---|---|---|
| original, subdirectory of GPFS mount | fails | exit 1 |
| patched, subdirectory of GPFS mount, write + read | ALIVE, no deleteVMs |
OK |
| patched, real mount point (tmpfs) | ALIVE | OK |
| patched, NFS pool mount point (read) | ALIVE | OK |
| patched, path on root FS | refuse, nothing written | exit 1, dir empty |
| patched, path does not exist | refuse | exit 1 |
| original, read right after write (diff=0) | ALIVE | **DEAD** |
| patched, read right after write / after 3 s, `-t 60` | ALIVE | OK / OK |
| patched, after 3 s with `-t 1` / corrupt hb file | DEAD | OK / OK |
The patched script runs live in the KVM agent on one host since 2026-10-07
(Ubuntu 24.04, 4.23.0.0,
IBM Storage Scale 5.2.3-5, pool `/data01/vol01`). No heartbeat errors since
then. I will open a PR (against 4.22, `Fixes #14326`) next.
**Update – HA failover test (2026-10-07):** 3 KVM hosts, HA-enabled VM on
the SharedMountPoint pool on host A,
host A hard-reset via sysrq (agent autostart disabled so it stays away). The
neighbour running the patched script
reported "Heart is not beating" for both pools (NFS + SharedMountPoint)
about 85 s after the reset. The host was marked
Down and the VM was restarted on another host about 110 s after the reset.
No `deleteVMs` side effects.
Side observation: while the clustered FS was recovering the failed node,
heartbeat writes on the surviving hosts hit
the script timeout once. The killed write leaves an empty `hb-<ip>.<pid>`
temp file behind. I can add a `trap` cleanup
to the PR if you like.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
