danivogel90 opened a new issue, #14328:
URL: https://github.com/apache/cloudstack/issues/14328
### ISSUE TYPE
Bug Report
### COMPONENT NAME
KVM agent / HA (scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh)
### CLOUDSTACK VERSION
4.22.1.0, 4.23.0.0 (introduced by #12773)
### CONFIGURATION
KVM, Ubuntu 24.04, SharedMountPoint primary storage on IBM Storage Scale
(GPFS).
GPFS is mounted at `/data01`, the primary storage pool path is
`/data01/vol01`.
### SUMMARY
#12773 added HA heartbeats for SharedMountPoint pools. The new script
`kvmsmpheartbeat.sh` requires the pool path itself to be a mount point:
if ! mountpoint -q "$MountPoint"; then
echo "Mount point is not a mounted filesystem: $MountPoint" >&2
exit 1
According to the docs, SharedMountPoint is "a file system path local to each
server". A subdirectory of a clustered filesystem (GPFS, OCFS2, GFS2, CephFS
...)
is a valid and common setup. For such a pool the heartbeat write never
succeeds:
WARN [kvm.resource.KVMHAMonitor] Write heartbeat for pool [27454846-...]
failed:
Mount point is not a mounted filesystem: /data01/vol01; try: 5 of 5.
### IMPACT
- VM-HA cannot use the heartbeat for this pool (the host looks dead to the
investigator).
- With `reboot.host.and.alert.management.on.heartbeat.timeout=true` (default
before 4.23,
see #10111), **every host in the cluster** hard-resets via sysrq about 5
minutes after
the agent upgrade, all at the same time. In hyperconverged setups this
also takes down
the storage cluster quorum.
### STEPS TO REPRODUCE
1. Mount a shared FS at `/data01`. Create a SharedMountPoint pool with path
`/data01/vol01`.
2. Upgrade or install the agent to >= 4.22.1.
3. Agent log shows the warning above every minute; `KVMHA/` is never created.
### EXPECTED RESULTS
The heartbeat is written to `/data01/vol01/KVMHA/hb-<ip>`.
### SUGGESTED FIX
Keep the safety intent (do not write the heartbeat to the local root FS when
the shared
FS is not mounted). Check the filesystem *containing* the path instead of
requiring the
path itself to be a mount point:
```bash
if command -v findmnt >/dev/null 2>&1; then
mnt_target=$(findmnt -n -o TARGET -T "$MountPoint" 2>/dev/null)
if [ -z "$mnt_target" ] || [ "$mnt_target" = "/" ]; then
echo "Mount point is not on a mounted (shared) filesystem: $MountPoint"
>&2
exit 1
fi
fi
```
Workaround for now: bind-mount the directory onto itself (`mount --bind
/data01/vol01 /data01/vol01`).
Related: #12773 (SMP heartbeat), #13090 (configurable fence action), #10111
(reboot on heartbeat timeout disabled by default).
Broader context: see enhancement issue #14329 (first-class support for
parallel/clustered filesystems).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
