bhouse-nexthop opened a new issue, #14744:
URL: https://github.com/apache/cloudstack/issues/14744

   ### problem
   
   When an HA-enabled VM stops unexpectedly while its host stays Up, the HA 
restart first tries the VM's last host, which is usually the host that just 
stopped it. If that host is Up and passes the capacity check, the VM goes 
straight back there. The deployment planner and `vm.allocation.algorithm` are 
never consulted.
   
   That's the right choice when the VM itself failed, for example a guest 
crash. It's the wrong one when the host stopped the VM, for example when the 
host's kernel OOM-killed the QEMU process because the host's VMs together used 
more memory than it has. With memory overcommit, the capacity check counts 
allocated memory times the overcommit ratio, so the host that killed the VM 
still "has capacity". The VM is restarted on the same full host, where it, or 
one of its neighbours, can be killed again.
   
   We hit this on KVM with `memoryOvercommitRatio` 4: Kubernetes worker VMs 
killed by the host's OOM killer were restarted by HA on the hosts that had 
killed them.
   
   Cause:
   - `HighAvailabilityManagerImpl.restart()` starts the VM with `HaTag` (when 
`ha.tag` is set) and `HaOperation`, but never sets 
`VirtualMachineProfile.Param.ConsiderLastHost`.
   - `DeploymentPlanningManagerImpl.deployInVmLastHost()` therefore tries the 
last host before the planner.
   - A successful start sets the VM's last host to the host it starts on, and a 
stop the VM didn't ask for doesn't change it, so the last host is the host the 
VM was stopped on (or, after a migration, the migration's source host).
   
   When the host itself fails, this doesn't arise: a host that isn't Up is 
never used as the last host.
   
   ### versions
   
   4.22 and main. The code involved is the same on both.
   
   ### The steps to reproduce the bug
   
   1. Use a KVM cluster with two or more hosts, and an HA-enabled VM running on 
host A.
   2. Kill the VM's QEMU process on A (`kill -9 <pid>`), the same thing the OOM 
killer does.
   3. CloudStack reports the VM stopped out of band and schedules an HA restart.
   4. The VM is restarted on A, even with `vm.allocation.algorithm` set to an 
algorithm that would choose another host. The management server log shows `Last 
host [Host {... A ...}] of VM [VM instance {...}] is UP and has enough 
capacity`.
   
   Reproduced on a three-host lab cluster with 
`vm.allocation.algorithm=firstfitleastconsumed`, the VM's host A being the most 
consumed: HA restarted it on A, with an empty host available.
   
   ### What to do about it?
   
   Changing the default would surprise people who rely on HA restarting a VM in 
place, so make it a setting. Add a cluster-scoped 
`vm.ha.restart.consider.last.host`, defaulting to `true` (today's behaviour). 
When it is `false`, the HA restart sets `ConsiderLastHost=false`, so the 
planner and allocators choose the host. The planner may still choose the same 
host when it is the best fit.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to