[
https://issues.apache.org/jira/browse/YUNIKORN-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
weichen lai reassigned YUNIKORN-3354:
-------------------------------------
Assignee: weichen lai
> Recovery pod ordering is not reproducible despite intending to be
> -----------------------------------------------------------------
>
> Key: YUNIKORN-3354
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3354
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: shim - kubernetes
> Affects Versions: 1.9.0
> Reporter: Dale Richardson
> Assignee: weichen lai
> Priority: Minor
>
> {{Context.registerPods}} sorts the recovered pods before replaying them, with
> the stated goal of making the resulting queue order reproducible:
> {code:go}
> // sort pods by creation time so that overall queue ordering is consistent
> with prior runs
> sort.Slice(pods, func(i, j int) bool {
> return pods[i].CreationTimestamp.Unix() < pods[j].CreationTimestamp.Unix()
> })
> {code}
> Three things prevent that from holding:
> * The input comes from {{PodInformer.Lister().List()}}, which returns the
> informer's items in map iteration order, so the starting order is not
> deterministic between runs.
> * The sort key is {{CreationTimestamp.Unix()}} - whole seconds. Kubernetes
> creationTimestamp has second resolution at source, so every pod created in
> the same second collapses into one equal-key group. For a burst - a Job with
> parallelism N, a Deployment scale-up - that is the entire set.
> * {{sort.Slice}} is not stable, so equal-key elements are left in an
> arbitrary order rather than their input order.
> Combined, the order within any same-second group is arbitrary and can differ
> between runs of the same recovery.
> h3. Result
> Queue ordering after a scheduler restart is not reproducible, which is what
> the comment says it is trying to achieve. The same non-determinism reaches
> per-application task dispatch, which map-iterates and sorts on the same
> second-resolution creation time.
> h3. Notes
> Switching to {{sort.SliceStable}} alone does not fix it, because the input
> order is already non-deterministic - stability would preserve an order that
> carries no information. It needs a deterministic secondary key. Pod UID is
> the obvious candidate; namespace and name would also work and has the
> advantage of being readable in logs.
> Fixing {{registerPods}} alone is not sufficient either. Submission order to
> the core is re-derived on every scheduling pass by the shim's {{getTasks}},
> which map-iterates the task map and unstable-sorts on the same
> second-resolution creation time. The same secondary key needs to go there, or
> the stated goal still is not met.
> Out of scope but worth noting so the ticket is not read as a complete
> determinism fix: on the core side, applications tied on submission time are
> re-ordered by map iteration on each cycle as well. Within an application,
> same-priority same-second asks do keep their arrival order, which is what
> makes the replay order above matter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
