[
https://issues.apache.org/jira/browse/YUNIKORN-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17723929#comment-17723929
]
Peter Bacsko commented on YUNIKORN-1724:
----------------------------------------
Update: it turned out that the scheduling interval was set to 0, which
effectively resulted in a busy loop. With the default value 1 second, IMO this
is practically a non-issue.
I don't think the PR is relevant, but I have a different suggestion: get rid of
this cycle entirely. There's no reason to send new tasks (pods) to the core in
batches. It could potentially speed up request sorting if we send new pods, ie.
if we send 50 new pods in a batch, that could significantly reduce
{{Application.sortRequests()}} calls. But this would require changes in the
core (more complications) and we can speed it up in different ways.
Incoming tasks should be handled immediately, which simplifies the code: we can
get rid of the New/Pending state (either of the two), just run
{{postTaskPending()}} logic after the task is added to the application object.
cc [~wilfreds]
> Improve the performance of shim side scheduling cycle
> -----------------------------------------------------
>
> Key: YUNIKORN-1724
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1724
> Project: Apache YuniKorn
> Issue Type: Sub-task
> Components: shim - kubernetes
> Reporter: Peter Bacsko
> Assignee: Peter Bacsko
> Priority: Major
> Labels: pull-request-available
> Attachments: getNewTasks.png
>
>
> Performance testing of Yunikorn uncovered that a lot of time is spent in
> {{Application.Schedule()}} in the shim. The problem is related to the fact
> that we collect task objects based on their state which is maintained by
> {{{}fsm.FSM{}}}. Even though we run {{Application.Schedule()}} once per
> second, it's still an issue due to the large number of {{RWMutex.RLock()}}
> calls. With a lot of pods, this consumes significant amount of CPU time.
> Also, different code paths are affected:
> The first is inside the switch-case part in {{{}Schedule(){}}}. We want to
> know the number of tasks in "New" state and we end up scanning all task
> objects for their status.
> The second is retrieving the "New" tasks from {{taskMap}} structure. This is
> done by {{GetNewTasks()}} / {{{}getTasks(){}}}, copying tasks based on their
> respective state to a new slice.
> To speed things up, we have to track the "New" tasks in a new map which is
> dynamically maintained when a new task added and when it leaves the New state
> (or the task gets removed). Knowing how many tasks we have also becomes
> trivial and won't require slice iteration/filtering.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: issues-h...@yunikorn.apache.org
