bhouse-nexthop commented on PR #14109: URL: https://github.com/apache/cloudstack/pull/14109#issuecomment-5683015701
Pushed one more commit, and added a section to the description on the thundering herd, which the write-up had been treating as an implementation detail of the selection spread rather than as the problem it is. **What prompted it.** The failure we actually see is not an allocation failure: a burst of deployments lands on one host and the VMs then sit in `Starting` for minutes instead of seconds, while that host works through a queue of starts its storage and agent command path cannot absorb. Nothing in the algorithm saw that. Capacity is not charged until a VM reaches `Starting` — `CapacityManagerImpl.postStateTransitionEvent` calls `allocateVmCapacity` on that transition, which is after the allocator has returned — and the utilisation average is a sample interval and a half life behind, so a burst is over before either notices. The recent-start term is the closest thing to a guard and it cannot do the job: - **It expires on a clock, and the load does not.** It counts VMs whose state changed within the window, so a VM stuck in `Starting` for ten minutes stops counting after five. The signal decays fastest on exactly the hosts that are worst off. - **It is capped too low to act as a brake.** Its contribution is capped at `0.111` of the score, so a host with 200 VMs stuck in `Starting` ranks identically to one with 50. So `host.weighted.starting.vms.threshold` (default 10, `0` disables) counts the VMs actually *in* `Starting` and gates on it rather than weighting it. It does not expire while the condition persists, it is read fresh on every ranking, and a gate cannot be outvoted by the other terms. It costs nothing — the per-host count query already filters on `vm.state`, so it is one more `SUM(IF(...))` in a query that was already being made, and the one-query-per-ranking property holds. Worth being explicit that this is a brake and not a cure: a host taking minutes to start VMs is usually saturated on its storage path or agent command queue, and the allocator cannot make that faster, only stop adding to it. If the bottleneck is a shared primary storage pool rather than the host, per-host gating will not help. **A defect found on the way.** The selection spread was applied to the healthy list only. Whenever no host was eligible, that list was empty, the shuffle was a no-op, and the held-back hosts were handed out in strict score order — so concurrent deployments agreed on one host in exactly the conditions that had caused the pile-up. Same class of defect as the "held-back hosts re-appended before the spread" one from the earlier review, in the other branch of the same `if`. The spread now applies to whichever tier leads, which removes the branch it was hiding in. On the question that came up while writing this — what happens if every host trips a threshold — the answer is unchanged and now stated in the description: thresholds decide *preference*, never *eligibility*. Held-back hosts are moved down the list, never dropped, so no threshold can turn a deployment that would have succeeded into an `InsufficientServerCapacity` failure. Within the held-back group the order is by starting count, so the least backed up host is tried first. Six new test cases (45 total). Full `mvn test` on `api`, `engine/schema` and `server`: 0 failures. The three Copilot threads were addressed in the previous push; I have resolved them. @DaanHoogland this is a behavioural change on top of what you approved, so flagging it rather than assuming the approval carries — happy to split the gate into its own PR if you would rather keep this one to the scoring change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
