Dale Richardson created YUNIKORN-3377:
-----------------------------------------
Summary: Add fault-injection tests with phantom-allocation oracles
for the shim
Key: YUNIKORN-3377
URL: https://issues.apache.org/jira/browse/YUNIKORN-3377
Project: Apache YuniKorn
Issue Type: Test
Components: shim - kubernetes
Reporter: Dale Richardson
Companion to the invariant property fuzzer of YUNIKORN-3373, one layer up:
where the fuzzer checks the scheduler cache as a data structure, this drives
the REAL scheduling stack (Context, Application/Task state machines,
dispatcher, embedded core through the MockScheduler harness) while injecting
the faults a real cluster produces, and asserts end-state oracles once the
cluster settles.
Two scenarios:
# *Bind failures*: the mock kube client fails a seeded fraction (p=0.3) of Bind
calls; a kubelet simulator turns each successful bind into assigned/Running
informer updates, and a resync simulator keeps re-reporting never-bound pods as
unassigned -- the informer model is truthful to what a real cluster does.
# *Node flaps*: the same bind churn plus seeded node remove/re-add cycles
delivered through the informer path, modelling a Node object deleted and
re-created while its pods keep running ({{kubectl delete node}} + kubelet
self-re-registration, inside the pod-GC ~40s orphan quarantine).
Oracles (end-state invariants, valid under any interleaving):
- O1: a pod that never bound successfully has no node assignment in the shim
cache (the YUNIKORN-3355 phantom oracle)
- O2: a bound pod is in the cache on exactly the node it was bound to
- O3: no pod is left assumed once the cluster is idle
- O4: no pod's assignment points at a node absent from the cache
- O7: every pod the shim has placed on a node has a matching core allocation on
that node (shim/core consistency, read directly from the core partition state)
*These tests currently FAIL on master by design*:
- The bind-failure scenario fails via O1 -- the YUNIKORN-3355 cache poisoning
(a failed bind leaves a phantom assignment; on master 100% of never-bound pods
leak one).
- The node-flap scenario additionally fails via O7 -- a node delete/re-add
permanently desynchronizes shim and core allocation state and corrupts queue
accounting (separate bug report to follow; link to be added here).
The PR stays a draft until those fixes merge, after which the bind scenario
passes and the flap scenario passes with the re-report fix; the tests then
become permanent regression guards. Running the scenarios under {{-race}} also
surfaced a data race on {{Task.pod}} (separate minor bug report to follow).
Additions to the two issue tickets (from my previous message):
- Ticket 1 (node-flap desync), add under the title line:
Found by: the fault-injection tests of <chaos JIRA key> (node-flap scenario,
oracle O7). Related: YUNIKORN-3373, YUNIKORN-3355, YUNIKORN-3374.
- Ticket 2 (Task.pod race), add:
Found by: running the <chaos JIRA key> fault-injection scenarios under -race.
- The YUNIKORN-3128 comment (item 3) can gain a first line: Observed while
building the fault-injection tests of <chaos JIRA key>:
The coder is meanwhile producing the master-based test/chaos-vs-master branch:
one file, both scenarios, the campaign's improved stability-window settle (the
original settle had a race against the task re-creation path), verified
red-on-master for the right reasons and — on the fixed cache tree —
bind-scenario green with the flap scenario failing only via O7. When you come
back with the keys (chaos hub + the two issue JIRAs), I'll do the same dance as
before: substitute keys, push, open the chaos PR as draft plus wire the
cross-references.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
