bhouse-nexthop opened a new pull request, #14139:
URL: https://github.com/apache/cloudstack/pull/14139
### Description
On a redundant VPC, creating several tiers with VMs in parallel corrupts
both virtual routers' NIC layout. Their keepalived configurations diverge and
VRRP breaks: routers end up FAULT or both PRIMARY, and the VPC is unusable
until restarted. Root cause analysis in #11710.
**Cause**
1. Each concurrently-deploying VM implements its VPC tier inside its own
`VmWorkStart` work job, which calls `addVpcRouterToGuestNetwork(router, ...)` →
`addVmToNetwork(router, ...)`.
2. `addVmToNetwork()` runs directly (no queue) whenever the calling thread
is inside *any* VM work job:
`jobContext.isJobDispatchedBy(VM_WORK_JOB_DISPATCHER)`. It does not check whose
work job it is.
3. So N parallel tier implements run N concurrent, unserialized
`orchestrateAddVmToNetwork()` calls on the same router. `getFreeDeviceId()`
hands out the same device id more than once.
`nics` rows from one router during reproduction, all live simultaneously:
| nic id | network | device_id | created |
|---|---|---|---|
| 3499 | 300 | **2** | 19:33:40 |
| 3502 | 301 | **2** | 19:33:40 |
| 3520 | 302 | **3** | 19:33:46 |
| 3523 | 303 | **3** | 19:33:46 |
`vm_work_job` contained no `VmWorkAddVmToNetwork` rows at all — every tier
attach took the direct path.
**Fix**
- Run directly only when the current work job belongs to the VM being
modified (new helper `isRunningVmWorkJobForVm()`); otherwise dispatch through
that VM's own work job queue.
- Apply the same rule to `removeNicFromVm()` and `removeVmFromNetwork()`.
The latter always ran directly, so tier removals were never serialized either.
- Scope `removeVmFromNetworkThroughJobQueue()`'s pending-job lookup by
network uuid, like the add path already does, so concurrent removals of
different networks do not collapse into one job.
- Drop an unused `retrievePendingWorkJob()` call from
`addVmToNetworkThroughJobQueue()`.
Fixes: #11710
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] Build/CI
- [ ] Test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Bug Severity
- [ ] BLOCKER
- [x] Critical
- [ ] Major
- [ ] Minor
- [ ] Trivial
### How Has This Been Tested?
Environment: 4.22.1.0, KVM, two management servers, 8 hosts, VXLAN isolation.
Repro: redundant VPC ("VPC HA" offering), then 8 tiers each with one VM, all
created in parallel through the API (mimics a `terraform apply`).
- Without the fix: corrupted on the first attempt — duplicate device ids
(table above), no `VmWorkAddVmToNetwork` jobs, routers in FAULT/PRIMARY with
keepalived `virtual_ipaddress` blocks that disagree between the two routers.
- The fix compiles against both 4.22 HEAD and the 4.22.1.0 tag. A live
re-test on the same environment with the patched class deployed on both
management servers is in progress; results will be posted here.
#### How did you try to break this feature and the system with this change?
- Deadlock: a user VM's work job now blocks while the router's NIC job runs
on the router's queue. The queues are distinct and router NIC jobs never
enqueue work back onto user VMs, so no cycle. The wait uses the same
`joinJob`/outcome machinery the API path already uses.
- Pool exhaustion would need more concurrent cross-VM NIC operations than
worker threads (`(db.cloud.maxActive * 2) / 3`, ~166 by default).
- Same-VM callers (a VM operating on itself from its own work job) still
take the direct path, unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
