bhouse-nexthop opened a new pull request, #14109: URL: https://github.com/apache/cloudstack/pull/14109
### Description This PR adds an opt-in VM allocation algorithm, `balancedweighted`, for `vm.allocation.algorithm`. Existing algorithms and the default are untouched. **The problem.** The existing algorithms rank hosts on allocated capacity alone. Under a large overprovisioning factor that reads badly: allocation is measured against a total that has already been multiplied by the factor, so a host under real strain can still report a low percentage allocated and keep attracting new VMs. Anything allocation cannot see - guests using far more than they asked for, VMs the management server has lost track of - is invisible. Concurrent deployments make it worse. Capacity is only charged once a VM starts, so every decision taken in the same moment reads the same figures, and strict ordering makes them all agree on one host. **What it ranks on.** A blend, lower is better: | Term | Source | |---|---| | CPU allocated | `op_host_capacity`, over the overprovisioned total | | CPU utilisation | moving average of measured usage | | Memory allocated | `op_host_capacity`, over the overprovisioned total | | Memory utilisation | moving average of measured usage | | VM count | VMs on the host | | Recent starts | VMs started within the last few minutes | Recent starts exist because a VM that started moments ago is invisible to both allocation lag and a moving average, while often working hardest. A dominant-resource term is added on top of the weighted mean, taking the larger of allocated and measured per resource, so that a host nearly out of any one resource does not rank well on a good average. Then two things happen that ranking alone does not do: - hosts measurably too busy are held back, unless that would leave nowhere to deploy - selection is random among the best few rather than strictly ordered, so simultaneous deployments do not all pick the same host **Measured effect.** Simulation over a churning, heavily overprovisioned fleet where part of the real load is invisible to allocation, measuring how unevenly real load ends up distributed (max/mean across hosts, lower is better): | Ranking | Load skew | |---|---| | allocation only | 1.84 - 2.01 | | allocation only, plus the random spread | 1.65 - 2.08 | | weighted | **1.27 - 1.40** | The middle row is a control: it isolates what the scoring contributes from what the randomisation contributes. **Utilisation figures** come from a moving average of what `StatsCollector` already polls but nothing used for placement. It is per management server and not persisted; every management server polls every host, so they converge. Until a server has samples, ranking falls back to allocation figures, and hosts that cannot be measured rank behind hosts that can rather than being assumed idle. Note the utilisation terms are only meaningful as intended on KVM. `getCpuUtilization` reports a reservation figure on VMware and is scaled by core count on XenServer; this is documented on the collector. All weights and thresholds are settings, most cluster scoped. ### Types of changes - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] New feature (non-breaking change which adds functionality) - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] Enhancement (improves an existing feature and functionality) - [ ] Cleanup (Code refactoring and cleanup, that may add test cases) - [ ] Build/CI - [ ] Test (unit or integration test code) ### Feature/Enhancement Scale or Bug Severity #### Feature/Enhancement Scale - [x] Major - [ ] Minor ### How Has This Been Tested? Unit tests, 39 new cases: - `WeightedHostScorerTest` - the scoring function: each term's direction, the dominant-resource term, unmeasured hosts, the utilisation gates, the selection spread. - `WeightedHostScorerRankTest` - `rank()` end to end as the allocator calls it, with the capacity and VM-count queries mocked. Covers the overprovisioned denominator, a busy host ranking behind a quiet one at equal allocation, an over-threshold host never leading while a healthy one exists, unmeasured hosts ranking last, fallback to allocation when nothing is measured, and a host missing a capacity row. - `HostLoadTrackerTest` - the moving average: first sample, a single spike not dominating, convergence, half-life, missed samples decaying by elapsed time rather than sample count, unchanged readings not folded twice, and a host that stops reporting becoming unusable. - `WeightedPlacementDistributionTest` - the simulation above. Deterministic; the workload is fixed before any arm runs so all arms see identical VMs. Full `mvn test` on `api`, `engine/schema` and `server`, checkstyle and license checks enabled: 0 failures. #### How did you try to break this feature and the system with this change? This went through two rounds of adversarial review. The defects found and fixed are worth listing, since they are the interesting part: - **The allocated fraction was measured against the wrong total.** `op_host_capacity` stores totals raw and applies overprovisioning when they are read. Dividing by the stored total made the fraction reach 1 at the host's physical size, so on a cluster overcommitted ten times every host clamped to 1 and both the allocation term and the dominant-resource term went dead - on exactly the clusters this is for. - **The utilisation threshold could be bypassed.** Held-back hosts were re-appended before the random spread was applied, so with one healthy host and a spread of three, two thirds of deployments picked an over-threshold host. - **A host with no samples was treated as idle**, exempt from the thresholds and scoring well on the dominant term - so a host with broken statistics became a deployment magnet. - **A host that stopped reporting kept vouching for itself.** `StatsCollector` hands back the previous entry when a poll fails, and that unchanged reading was folded again every interval. - **The sampler ran on a `Timer`**, which dies permanently and silently on one escaping error, leaving placement quietly back on allocation alone. It also ran regardless of whether the algorithm was selected. - **A negative weight** would have ranked the most loaded host first. - **The simulation drew different random streams per arm**, so the arms were not seeing the same workload; and it bypassed `rank()`, which is why it caught none of the above. Both fixed, and the spread-only control arm added. Other things checked: hosts missing a capacity row, all-zero weights, `podId`/`clusterId` being null, concurrent access to the shared average, and that the two per-ranking queries became one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
