[
https://issues.apache.org/jira/browse/YUNIKORN-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dale Richardson updated YUNIKORN-3377:
--------------------------------------
Description:
Companion to the invariant property fuzzer of YUNIKORN-3373, one layer up:
where the fuzzer checks the scheduler cache as a data structure, this drives
the REAL scheduling stack (Context, Application/Task state machines,
dispatcher, embedded core through the MockScheduler harness) while injecting
the faults a real cluster produces, and asserts end-state oracles once the
cluster settles.
Two scenarios:
# *Bind failures*: the mock kube client fails a seeded fraction (p=0.3) of Bind
calls; a kubelet simulator turns each successful bind into assigned/Running
informer updates, and a resync simulator keeps re-reporting never-bound pods as
unassigned -- the informer model is truthful to what a real cluster does.
# *Node flaps*: the same bind churn plus seeded node remove/re-add cycles
delivered through the informer path, modelling a Node object deleted and
re-created while its pods keep running ({{kubectl delete node}} + kubelet
self-re-registration, inside the pod-GC ~40s orphan quarantine).
Oracles (end-state invariants, valid under any interleaving):
- O1: a pod that never bound successfully has no node assignment in the shim
cache (the YUNIKORN-3355 phantom oracle)
- O2: a bound pod is in the cache on exactly the node it was bound to
- O3: no pod is left assumed once the cluster is idle
- O4: no pod's assignment points at a node absent from the cache
- O7: every pod the shim has placed on a node has a matching core allocation on
that node (shim/core consistency, read directly from the core partition state)
*These tests currently FAIL on master by design*:
- The bind-failure scenario fails via O1 -- the YUNIKORN-3355 cache poisoning
(a failed bind leaves a phantom assignment; on master 100% of never-bound pods
leak one).
- The node-flap scenario additionally fails via O7 -- a node delete/re-add
permanently desynchronizes shim and core allocation state and corrupts queue
accounting: YUNIKORN-3375.
and
Running the scenarios under {{-race}} also surfaced a data race on
{{Task.pod}}: YUNIKORN-3376.
The PR stays a draft until those fixes merge, after which the bind scenario
passes and the flap scenario passes with the re-report fix; the tests then
become permanent regression guards.
Issues found:
- [YUNIKORN-3375|https://issues.apache.org/jira/browse/YUNIKORN-3375]: Node
delete/re-add silently corrupts queue accounting: adopted pods are never
re-reported to the core
Found by: the fault-injection tests of <chaos JIRA key> (node-flap scenario,
oracle O7). Related: YUNIKORN-3373, YUNIKORN-3355, YUNIKORN-3374.
- [YUNIKORN-3376|https://issues.apache.org/jira/browse/YUNIKORN-3376]: Data
race on Task.pod: checkPodMetadataBeforeScheduling reads without the task lock
Found by: running the <chaos JIRA key> fault-injection scenarios under -race.
Issues re-discoveered:
- [YUNIKORN-3128|https://issues.apache.org/jira/browse/YUNIKORN-3128]: Data
race on Task.pod: checkPodMetadataBeforeScheduling reads without the task lock
The coder is meanwhile producing the master-based test/chaos-vs-master branch:
one file, both scenarios, the campaign's improved stability-window settle (the
original settle had a race against the task re-creation path), verified
red-on-master for the right reasons and — on the fixed cache tree —
bind-scenario green with the flap scenario failing only via O7. When you come
back with the keys (chaos hub + the two issue JIRAs), I'll do the same dance as
before: substitute keys, push, open the chaos PR as draft plus wire the
cross-references.
was:
Companion to the invariant property fuzzer of YUNIKORN-3373, one layer up:
where the fuzzer checks the scheduler cache as a data structure, this drives
the REAL scheduling stack (Context, Application/Task state machines,
dispatcher, embedded core through the MockScheduler harness) while injecting
the faults a real cluster produces, and asserts end-state oracles once the
cluster settles.
Two scenarios:
# *Bind failures*: the mock kube client fails a seeded fraction (p=0.3) of Bind
calls; a kubelet simulator turns each successful bind into assigned/Running
informer updates, and a resync simulator keeps re-reporting never-bound pods as
unassigned -- the informer model is truthful to what a real cluster does.
# *Node flaps*: the same bind churn plus seeded node remove/re-add cycles
delivered through the informer path, modelling a Node object deleted and
re-created while its pods keep running ({{kubectl delete node}} + kubelet
self-re-registration, inside the pod-GC ~40s orphan quarantine).
Oracles (end-state invariants, valid under any interleaving):
- O1: a pod that never bound successfully has no node assignment in the shim
cache (the YUNIKORN-3355 phantom oracle)
- O2: a bound pod is in the cache on exactly the node it was bound to
- O3: no pod is left assumed once the cluster is idle
- O4: no pod's assignment points at a node absent from the cache
- O7: every pod the shim has placed on a node has a matching core allocation on
that node (shim/core consistency, read directly from the core partition state)
*These tests currently FAIL on master by design*:
- The bind-failure scenario fails via O1 -- the YUNIKORN-3355 cache poisoning
(a failed bind leaves a phantom assignment; on master 100% of never-bound pods
leak one).
- The node-flap scenario additionally fails via O7 -- a node delete/re-add
permanently desynchronizes shim and core allocation state and corrupts queue
accounting (separate bug report to follow; link to be added here).
The PR stays a draft until those fixes merge, after which the bind scenario
passes and the flap scenario passes with the re-report fix; the tests then
become permanent regression guards. Running the scenarios under {{-race}} also
surfaced a data race on {{Task.pod}} (separate minor bug report to follow).
Issues found:
- [YUNIKORN-3375|https://issues.apache.org/jira/browse/YUNIKORN-3375]: Node
delete/re-add silently corrupts queue accounting: adopted pods are never
re-reported to the core
Found by: the fault-injection tests of <chaos JIRA key> (node-flap scenario,
oracle O7). Related: YUNIKORN-3373, YUNIKORN-3355, YUNIKORN-3374.
- [YUNIKORN-3376|https://issues.apache.org/jira/browse/YUNIKORN-3376]: Data
race on Task.pod: checkPodMetadataBeforeScheduling reads without the task lock
Found by: running the <chaos JIRA key> fault-injection scenarios under -race.
Issues re-discoveered:
- [YUNIKORN-3128|https://issues.apache.org/jira/browse/YUNIKORN-3128]: Data
race on Task.pod: checkPodMetadataBeforeScheduling reads without the task lock
The coder is meanwhile producing the master-based test/chaos-vs-master branch:
one file, both scenarios, the campaign's improved stability-window settle (the
original settle had a race against the task re-creation path), verified
red-on-master for the right reasons and — on the fixed cache tree —
bind-scenario green with the flap scenario failing only via O7. When you come
back with the keys (chaos hub + the two issue JIRAs), I'll do the same dance as
before: substitute keys, push, open the chaos PR as draft plus wire the
cross-references.
> Add fault-injection tests with phantom-allocation oracles for the shim
> ----------------------------------------------------------------------
>
> Key: YUNIKORN-3377
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3377
> Project: Apache YuniKorn
> Issue Type: Test
> Components: shim - kubernetes
> Reporter: Dale Richardson
> Priority: Major
>
> Companion to the invariant property fuzzer of YUNIKORN-3373, one layer up:
> where the fuzzer checks the scheduler cache as a data structure, this drives
> the REAL scheduling stack (Context, Application/Task state machines,
> dispatcher, embedded core through the MockScheduler harness) while injecting
> the faults a real cluster produces, and asserts end-state oracles once the
> cluster settles.
> Two scenarios:
> # *Bind failures*: the mock kube client fails a seeded fraction (p=0.3) of
> Bind calls; a kubelet simulator turns each successful bind into
> assigned/Running informer updates, and a resync simulator keeps re-reporting
> never-bound pods as unassigned -- the informer model is truthful to what a
> real cluster does.
> # *Node flaps*: the same bind churn plus seeded node remove/re-add cycles
> delivered through the informer path, modelling a Node object deleted and
> re-created while its pods keep running ({{kubectl delete node}} + kubelet
> self-re-registration, inside the pod-GC ~40s orphan quarantine).
> Oracles (end-state invariants, valid under any interleaving):
> - O1: a pod that never bound successfully has no node assignment in the shim
> cache (the YUNIKORN-3355 phantom oracle)
> - O2: a bound pod is in the cache on exactly the node it was bound to
> - O3: no pod is left assumed once the cluster is idle
> - O4: no pod's assignment points at a node absent from the cache
> - O7: every pod the shim has placed on a node has a matching core allocation
> on that node (shim/core consistency, read directly from the core partition
> state)
> *These tests currently FAIL on master by design*:
> - The bind-failure scenario fails via O1 -- the YUNIKORN-3355 cache poisoning
> (a failed bind leaves a phantom assignment; on master 100% of never-bound
> pods leak one).
> - The node-flap scenario additionally fails via O7 -- a node delete/re-add
> permanently desynchronizes shim and core allocation state and corrupts queue
> accounting: YUNIKORN-3375.
> and
> Running the scenarios under {{-race}} also surfaced a data race on
> {{Task.pod}}: YUNIKORN-3376.
> The PR stays a draft until those fixes merge, after which the bind scenario
> passes and the flap scenario passes with the re-report fix; the tests then
> become permanent regression guards.
> Issues found:
> - [YUNIKORN-3375|https://issues.apache.org/jira/browse/YUNIKORN-3375]: Node
> delete/re-add silently corrupts queue accounting: adopted pods are never
> re-reported to the core
> Found by: the fault-injection tests of <chaos JIRA key> (node-flap scenario,
> oracle O7). Related: YUNIKORN-3373, YUNIKORN-3355, YUNIKORN-3374.
> - [YUNIKORN-3376|https://issues.apache.org/jira/browse/YUNIKORN-3376]: Data
> race on Task.pod: checkPodMetadataBeforeScheduling reads without the task lock
> Found by: running the <chaos JIRA key> fault-injection scenarios under -race.
> Issues re-discoveered:
> - [YUNIKORN-3128|https://issues.apache.org/jira/browse/YUNIKORN-3128]: Data
> race on Task.pod: checkPodMetadataBeforeScheduling reads without the task lock
> The coder is meanwhile producing the master-based test/chaos-vs-master
> branch: one file, both scenarios, the campaign's improved stability-window
> settle (the original settle had a race against the task re-creation path),
> verified red-on-master for the right reasons and — on the fixed cache tree —
> bind-scenario green with the flap scenario failing only via O7. When you come
> back with the keys (chaos hub + the two issue JIRAs), I'll do the same dance
> as before: substitute keys, push, open the chaos PR as draft plus wire the
> cross-references.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
