jmsperu commented on code in PR #13090: URL: https://github.com/apache/cloudstack/pull/13090#discussion_r4348449339
########## scripts/vm/hypervisor/kvm/kvmha-fence.sh: ########## @@ -0,0 +1,71 @@ +#!/bin/bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# Shared fence-action helper for kvmheartbeat.sh, kvmspheartbeat.sh and kvmsmpheartbeat.sh. +# Sourced by those scripts; do not invoke directly. +# +# kvm.heartbeat.fence.action in agent.properties selects the action: +# hard-reboot (default), graceful-reboot, restart-agent, or log-only (only writes to +# syslog and takes no action). Any other value falls back to hard-reboot. +# +# Usage from caller: +# source "$(dirname "$0")/kvmha-fence.sh" +# fence_action "kvmheartbeat.sh" # script name passed for log tagging + +AGENT_PROPS="${AGENT_PROPS:-/etc/cloudstack/agent/agent.properties}" + +fence_action() { + local source_script="${1:-kvmha}" + local FENCE_ACTION="hard-reboot" + + if [ -r "$AGENT_PROPS" ]; then + local val + val=$(grep "^kvm.heartbeat.fence.action=" "$AGENT_PROPS" | tail -n 1 | cut -d= -f2- | tr -d '[:space:]') + [ -n "$val" ] && FENCE_ACTION="$val" + fi + + case "$FENCE_ACTION" in + log-only) Review Comment: Good catch, thanks. You are right: with a hung NFS mount virsh destroy can block in D state, the agent kills the script once heartbeat.update.timeout runs out, and the hard-reboot fallback never runs. Fixed in 8fa1937d3f. Before restart-agent or log-only stop any VM, the fence now arms a deadman. It is a separate bash process started with setsid in its own session (nohup on hosts without setsid), with its output closed, ignoring HUP and TERM, so it survives the agent force-killing the script. It checks an arm file once a second. If the file is still there when the budget runs out, it logs, runs sync and writes b to the sysrq trigger, the same as the existing hard-reboot. The script removes the arm file and kills the deadman only after stop_vms has confirmed that every VM it had to stop is gone. If stop_vms fails before the budget runs out, the script hard-reboots itself as before. The budget is a new agent property, kvm.heartbeat.fence.stop.vms.timeout, in seconds, default 30, documented in agent.properties, AgentProperties and the script header with a note to keep it below heartbeat.update.timeout. Any value that is not a positive integer falls back to 30 with a syslog warning. Each virsh call (destroy and the list used to verify) is also wrapped in coreutils timeout when installed, at 10 s or the budget if lower, so one stuck domain cannot use up the whole budget. Without timeout the calls run unbounded and the deadman still covers them. One limit worth knowing: the deadman lives in the agent's cgroup. If systemd restarts cloudstack-agent during the stop phase it ignores the TERM and still fires, unless TimeoutStopSec is set below the budget. New tests in KvmHaFenceScriptTest run the real script against stand-ins: a virsh destroy that hangs with a 2 s budget while the test force-kills the script the way Script.execute does (the deadman writes b about 2 s later and the agent is never restarted); a normal stop within the budget (arm and pid files removed, deadman gone, no reboot after the budget); and the invalid values abc, 0, -5 and 1.5 each falling back to the default with a warning. The existing tests still pass and keep.vms.running, hard-reboot and graceful-reboot behave as before. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
