[
https://issues.apache.org/jira/browse/YUNIKORN-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dale Richardson updated YUNIKORN-3377:
--------------------------------------
Description:
Companion to the invariant property fuzzer of YUNIKORN-3373, one layer up:
where the fuzzer checks the scheduler cache as a data structure, this drives
the REAL scheduling stack (Context, Application/Task state machines,
dispatcher, embedded core through the MockScheduler harness) while injecting
the faults a real cluster produces, and asserts end-state oracles once the
cluster settles.
Two scenarios:
# *Bind failures*: the mock kube client fails a seeded fraction (p=0.3) of Bind
calls; a kubelet simulator turns each successful bind into assigned/Running
informer updates, and a resync simulator keeps re-reporting never-bound pods as
unassigned -- the informer model is truthful to what a real cluster does.
# *Node flaps*: the same bind churn plus seeded node remove/re-add cycles
delivered through the informer path, modelling a Node object deleted and
re-created while its pods keep running ({{kubectl delete node}} + kubelet
self-re-registration, inside the pod-GC ~40s orphan quarantine).
Oracles (end-state invariants, valid under any interleaving):
- O1: a pod that never bound successfully has no node assignment in the shim
cache (the YUNIKORN-3355 phantom oracle)
- O2: a bound pod is in the cache on exactly the node it was bound to
- O3: no pod is left assumed once the cluster is idle
- O4: no pod's assignment points at a node absent from the cache
- O5: every pod the shim has placed on a node has a matching core allocation on
that node (shim/core consistency, read directly from the core partition state)
*These tests currently FAIL on master by design*:
- The bind-failure scenario fails via O1 -- the YUNIKORN-3355 cache poisoning
(a failed bind leaves a phantom assignment; on master 100% of never-bound pods
leak one).
- The node-flap scenario additionally fails via O5 -- a node delete/re-add
permanently desynchronizes shim and core allocation state and corrupts queue
accounting: YUNIKORN-3375.
- Running the scenarios under {{-race}} also surfaced a data race on
{{Task.pod}}: YUNIKORN-3376.
The accompanying PR stays a draft until those fixes merge, after which the bind
scenario passes and the flap scenario passes with the re-report fix; the tests
then become permanent regression guards.
*New issues found by these tests*:
- YUNIKORN-3375: Node delete/re-add silently corrupts queue accounting: adopted
pods are never re-reported to the core (node-flap scenario, oracle O5)
- YUNIKORN-3376: Data race on Task.pod: checkPodMetadataBeforeScheduling reads
without the task lock (any scenario under -race)
*Pre-existing issues deterministically reproduced*:
- YUNIKORN-3355: Failed bind leaves a stale node assignment in the shim cache
(bind scenario, oracle O1)
- YUNIKORN-3374: Phantom allocation when a node with an assumed pod is removed
and re-added (node-flap scenario)
- YUNIKORN-3128: Yunikorn ignores pending pods after apiserver errors
(mechanism detail added there as a comment: whether a failed bind retries or
wedges is pure timing)
The "Found by" one-liners still belong on the other tickets — on 3375 and 3376,
now readable as "Found by: the fault-injection tests of YUNIKORN-3377 …".
was:
Companion to the invariant property fuzzer of YUNIKORN-3373, one layer up:
where the fuzzer checks the scheduler cache as a data structure, this drives
the REAL scheduling stack (Context, Application/Task state machines,
dispatcher, embedded core through the MockScheduler harness) while injecting
the faults a real cluster produces, and asserts end-state oracles once the
cluster settles.
Two scenarios:
# *Bind failures*: the mock kube client fails a seeded fraction (p=0.3) of Bind
calls; a kubelet simulator turns each successful bind into assigned/Running
informer updates, and a resync simulator keeps re-reporting never-bound pods as
unassigned -- the informer model is truthful to what a real cluster does.
# *Node flaps*: the same bind churn plus seeded node remove/re-add cycles
delivered through the informer path, modelling a Node object deleted and
re-created while its pods keep running ({{kubectl delete node}} + kubelet
self-re-registration, inside the pod-GC ~40s orphan quarantine).
Oracles (end-state invariants, valid under any interleaving):
- O1: a pod that never bound successfully has no node assignment in the shim
cache (the YUNIKORN-3355 phantom oracle)
- O2: a bound pod is in the cache on exactly the node it was bound to
- O3: no pod is left assumed once the cluster is idle
- O4: no pod's assignment points at a node absent from the cache
- O7: every pod the shim has placed on a node has a matching core allocation on
that node (shim/core consistency, read directly from the core partition state)
*These tests currently FAIL on master by design*:
- The bind-failure scenario fails via O1 -- the YUNIKORN-3355 cache poisoning
(a failed bind leaves a phantom assignment; on master 100% of never-bound pods
leak one).
- The node-flap scenario additionally fails via O7 -- a node delete/re-add
permanently desynchronizes shim and core allocation state and corrupts queue
accounting: YUNIKORN-3375.
and
Running the scenarios under {{-race}} also surfaced a data race on
{{Task.pod}}: YUNIKORN-3376.
The PR stays a draft until those fixes merge, after which the bind scenario
passes and the flap scenario passes with the re-report fix; the tests then
become permanent regression guards.
Issues found:
- [YUNIKORN-3375|https://issues.apache.org/jira/browse/YUNIKORN-3375]: Node
delete/re-add silently corrupts queue accounting: adopted pods are never
re-reported to the core
Found by: the fault-injection tests of <chaos JIRA key> (node-flap scenario,
oracle O7). Related: YUNIKORN-3373, YUNIKORN-3355, YUNIKORN-3374.
- [YUNIKORN-3376|https://issues.apache.org/jira/browse/YUNIKORN-3376]: Data
race on Task.pod: checkPodMetadataBeforeScheduling reads without the task lock
Found by: running the <chaos JIRA key> fault-injection scenarios under -race.
Issues re-discoveered:
- [YUNIKORN-3128|https://issues.apache.org/jira/browse/YUNIKORN-3128]: Data
race on Task.pod: checkPodMetadataBeforeScheduling reads without the task lock
> Add fault-injection tests with phantom-allocation oracles for the shim
> ----------------------------------------------------------------------
>
> Key: YUNIKORN-3377
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3377
> Project: Apache YuniKorn
> Issue Type: Test
> Components: shim - kubernetes
> Reporter: Dale Richardson
> Priority: Major
>
> Companion to the invariant property fuzzer of YUNIKORN-3373, one layer up:
> where the fuzzer checks the scheduler cache as a data structure, this drives
> the REAL scheduling stack (Context, Application/Task state machines,
> dispatcher, embedded core through the MockScheduler harness) while injecting
> the faults a real cluster produces, and asserts end-state oracles once the
> cluster settles.
> Two scenarios:
> # *Bind failures*: the mock kube client fails a seeded fraction (p=0.3) of
> Bind calls; a kubelet simulator turns each successful bind into
> assigned/Running informer updates, and a resync simulator keeps re-reporting
> never-bound pods as unassigned -- the informer model is truthful to what a
> real cluster does.
> # *Node flaps*: the same bind churn plus seeded node remove/re-add cycles
> delivered through the informer path, modelling a Node object deleted and
> re-created while its pods keep running ({{kubectl delete node}} + kubelet
> self-re-registration, inside the pod-GC ~40s orphan quarantine).
> Oracles (end-state invariants, valid under any interleaving):
> - O1: a pod that never bound successfully has no node assignment in the shim
> cache (the YUNIKORN-3355 phantom oracle)
> - O2: a bound pod is in the cache on exactly the node it was bound to
> - O3: no pod is left assumed once the cluster is idle
> - O4: no pod's assignment points at a node absent from the cache
> - O5: every pod the shim has placed on a node has a matching core allocation
> on that node (shim/core consistency, read directly from the core partition
> state)
> *These tests currently FAIL on master by design*:
> - The bind-failure scenario fails via O1 -- the YUNIKORN-3355 cache poisoning
> (a failed bind leaves a phantom assignment; on master 100% of never-bound
> pods leak one).
> - The node-flap scenario additionally fails via O5 -- a node delete/re-add
> permanently desynchronizes shim and core allocation state and corrupts queue
> accounting: YUNIKORN-3375.
> - Running the scenarios under {{-race}} also surfaced a data race on
> {{Task.pod}}: YUNIKORN-3376.
> The accompanying PR stays a draft until those fixes merge, after which the
> bind scenario passes and the flap scenario passes with the re-report fix; the
> tests then become permanent regression guards.
> *New issues found by these tests*:
> - YUNIKORN-3375: Node delete/re-add silently corrupts queue accounting:
> adopted pods are never re-reported to the core (node-flap scenario, oracle O5)
> - YUNIKORN-3376: Data race on Task.pod: checkPodMetadataBeforeScheduling
> reads without the task lock (any scenario under -race)
> *Pre-existing issues deterministically reproduced*:
> - YUNIKORN-3355: Failed bind leaves a stale node assignment in the shim cache
> (bind scenario, oracle O1)
> - YUNIKORN-3374: Phantom allocation when a node with an assumed pod is
> removed and re-added (node-flap scenario)
> - YUNIKORN-3128: Yunikorn ignores pending pods after apiserver errors
> (mechanism detail added there as a comment: whether a failed bind retries or
> wedges is pure timing)
> The "Found by" one-liners still belong on the other tickets — on 3375 and
> 3376, now readable as "Found by: the fault-injection tests of YUNIKORN-3377
> …".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
