jmsperu commented on code in PR #13090: URL: https://github.com/apache/cloudstack/pull/13090#discussion_r4356700210
########## scripts/vm/hypervisor/kvm/kvmha-fence.sh: ########## @@ -0,0 +1,71 @@ +#!/bin/bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# Shared fence-action helper for kvmheartbeat.sh, kvmspheartbeat.sh and kvmsmpheartbeat.sh. +# Sourced by those scripts; do not invoke directly. +# +# kvm.heartbeat.fence.action in agent.properties selects the action: +# hard-reboot (default), graceful-reboot, restart-agent, or log-only (only writes to +# syslog and takes no action). Any other value falls back to hard-reboot. +# +# Usage from caller: +# source "$(dirname "$0")/kvmha-fence.sh" +# fence_action "kvmheartbeat.sh" # script name passed for log tagging + +AGENT_PROPS="${AGENT_PROPS:-/etc/cloudstack/agent/agent.properties}" + +fence_action() { + local source_script="${1:-kvmha}" + local FENCE_ACTION="hard-reboot" + + if [ -r "$AGENT_PROPS" ]; then + local val + val=$(grep "^kvm.heartbeat.fence.action=" "$AGENT_PROPS" | tail -n 1 | cut -d= -f2- | tr -d '[:space:]') + [ -n "$val" ] && FENCE_ACTION="$val" + fi + + case "$FENCE_ACTION" in + log-only) Review Comment: Ran the hung storage case with `restart-agent` on a KVM lab, twice: once with the stock 60 s `heartbeat.update.timeout`, once with it set to 15 s so the agent kills the fence script before the script's own hard-reboot fallback can fire. Both runs ended in a hard reboot. Timelines below, times are UTC from the host's journal (the journal was streamed off-host so nothing was lost at the reboot). **Lab.** Ubuntu 24.04, libvirt 10.0.0, QEMU 8.2.2, same versions as our production hosts. The host is itself a VM, so the guest ran under TCG (`--virt-type qemu`), which does not change anything on the storage side. NFS primary storage was exported from the same host and mounted hard over loopback at `/mnt/primary`, like a libvirt NFS pool. The guest `i-2-1-VM` had its qcow2 on that mount and ran a `dd ... oflag=direct conv=fsync` loop so it was writing when the storage hung. The agent was a bash stand-in for `KVMHAMonitor`: it writes the heartbeat with `kvmheartbeat.sh`, runs `kvmheartbeat.sh ... -c` after the retries are exhausted, and kills either script with SIGKILL when the timeout runs out, the same thing `Script.destroyForcibly()` does. Only the retry count and the write timeout were shortened so the test does not take five minutes to reach the fence. `agent.properties` had `kvm.heartbeat.fence.action=restart-agent` and `kvm.heartbeat.fence.stop.vms.timeout=30`. The hang wa s `iptables -I OUTPUT -o lo -p tcp --dport 2049 -j DROP` (plus INPUT), with an `at` job to remove it. **Run 1, agent kill timeout 60 s (default).** The fence script reached the fallback on its own. ``` 05:59:18 NFS blocked 05:59:39 agent: heartbeat write killed at 20 s, try 1 of 2 06:00:01 agent: heartbeat write killed at 20 s, try 2 of 2 06:00:03 agent: running kvmheartbeat.sh -c (kill timeout 60 s) 06:00:03 heartbeat: stopping VMs for fence action 'restart-agent' with a budget of 30 s ... at most 10 s per virsh call 06:00:04 heartbeat: fence deadman (pid 12880) armed; the host will be hard-rebooted in 30 s unless ... 06:00:04 heartbeat: stopping VM i-2-1-VM so it cannot run here while HA starts it on another host 06:00:14 virsh destroy killed by the 10 s per-call timeout (qemu stuck in the NFS write; libvirt did not return) 06:00:24 heartbeat: could not stop every VM ... falling back to hard-reboot 06:00:24 heartbeat: kvmheartbeat.sh will reboot system because it was unable to write the heartbeat to the storage 06:00:29 host rebooted (sysrq b from the fence script, 21 s into the 60 s kill timeout) ``` The second `virsh list` after the destroy also hit the 10 s timeout, which is why `stop_vms` reported failure. The qemu process was in state `Zl` just before the reboot: it had been killed but libvirt, itself stuck on the hung mount, had not reaped it. So from the fence's point of view the VM was never confirmed stopped, and it rebooted. **Run 2, agent kill timeout 15 s.** This is the case you raised: the agent kills the script before it reaches `echo b > /proc/sysrq-trigger`. ``` 06:04:45 NFS blocked 06:05:09 agent: heartbeat write killed at 20 s, try 1 of 2 06:05:31 agent: heartbeat write killed at 20 s, try 2 of 2 06:05:33 agent: running kvmheartbeat.sh -c (kill timeout 15 s) 06:05:33 heartbeat: stopping VMs for fence action 'restart-agent' with a budget of 30 s ... 06:05:33 heartbeat: fence deadman (pid 4184) armed; the host will be hard-rebooted in 30 s unless ... 06:05:33 heartbeat: stopping VM i-2-1-VM ... 06:05:44 heartbeat: could not stop every VM ... falling back to hard-reboot 06:05:44 heartbeat: kvmheartbeat.sh will reboot system because it was unable to write the heartbeat to the storage 06:05:48 agent: fence script returned rc=137 <- SIGKILL from the agent, during the script's "sleep 5" before the sysrq write 06:05:48 deadman pid 4184 still alive (own session via setsid, outside the killed process) 06:06:03 heartbeat: VMs were not confirmed stopped within kvm.heartbeat.fence.stop.vms.timeout (30 s); fence deadman is hard-rebooting the host ... 06:06:08 host rebooted (sysrq b from the deadman, 30 s after arming) ``` So with the agent killing the script mid-fence, the host still went down 30 s after the stop phase began, and nothing on that host could carry on writing to the shared disks while HA started the VM elsewhere. In this run the qemu process happened to die at 06:05:47, after the script had already been killed, so nobody could disarm the deadman; it rebooted anyway, which is the intended fail-safe (the deadman only stands down when the fence confirms every VM is gone). Both runs also show why the per-call `timeout` matters: `virsh destroy` and `virsh list` each blocked for the full 10 s on the hung mount, so without it the stop phase alone would have eaten the whole budget. Happy to run any other variant you want to see (`include.system.vms=false`, `log-only`, or a hung mount that is not the one holding the VM disks). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
