bhouse-nexthop opened a new pull request, #14745:
URL: https://github.com/apache/cloudstack/pull/14745
### Description
When an HA-enabled VM stops unexpectedly while its host stays Up, the HA
restart first tries the VM's last host, which is usually the host that just
stopped it. If that host is Up and passes the capacity check, the VM goes
straight back there, and the planner and `vm.allocation.algorithm` are never
asked.
That's right when the VM itself failed. It's wrong when the host stopped it,
for example when the host's kernel OOM-killed the QEMU process. Under memory
overcommit, the capacity check still counts that host as having room, so the VM
is restarted on the same full host.
Changing that unconditionally would surprise anyone who relies on HA
restarting a VM in place. So this adds a cluster-scoped setting,
`vm.ha.restart.consider.last.host`, defaulting to `true`, which is today's
behaviour. When it's `false`, the HA restart sets `ConsiderLastHost=false`, so
the planner and allocators choose the host. They may still choose the same host
when it's the best fit.
Fixes #14744
#### Changes
- New `ConfigKey<Boolean>` `vm.ha.restart.consider.last.host` in
`HighAvailabilityManagerImpl`: Advanced category, dynamic, cluster scope,
default `true`. It's registered in `getConfigKeys()`.
- `restart()` builds its params in a new `getRestartParams(workType,
clusterId)`:
- `HaTag` when `ha.tag` is set, and `HaOperation` for HA work, both as
before;
- `ConsiderLastHost=false` when the setting is `false` for the cluster of
the host the VM ran on.
- With `ha.tag` set, the last host is never preferred anyway, so the setting
has no effect there. The description says so.
#### Not changed here
- The default keeps today's behaviour.
- The last host isn't excluded, only no longer preferred. Excluding it would
make the restart fail outright when it's the only host that fits.
- Host failure isn't affected: a host that isn't Up is never used as the
last host.
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Bug fix (non-breaking change which fixes an issue)
- [x] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] Build/CI
- [x] Test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Bug Severity
- [ ] BLOCKER
- [ ] Critical
- [ ] Major
- [x] Minor
- [ ] Trivial
### How Has This Been Tested?
- New unit tests in `HighAvailabilityManagerImplTest`:
- by default the params don't set `ConsiderLastHost`;
- with the setting `false` they set it to `false`;
- `HaOperation` is set only for HA work.
- Lab test: a nested KVM cluster of three hosts, with
`vm.allocation.algorithm=firstfitleastconsumed`. The lab runs a main-based
(24.0) build with this commit cherry-picked, not the 4.22 branch itself; the
code changed here is the same on both.
- Setup: an HA-enabled VM and two filler VMs on host 1, the most consumed
host. Then the VM's QEMU process is killed with `kill -9` on host 1, as the OOM
killer does, and HA restarts it.
| `vm.ha.restart.consider.last.host` (cluster) | HA restarted the VM on |
Planner log |
|---|---|---|
| `true` (default) | host 1, its last host | `Last host [...] of VM [...] is
UP and has enough capacity` |
| `false` | host 2, the least consumed | no last-host line |
#### How did you try to break this feature and the system with this change?
- With `false` and no other host that fits, the planner still picks the last
host, so the restart doesn't fail where it used to succeed.
- The value is read per cluster at restart time, so a cluster override
applies without a restart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]