GitHub user zhan849 opened a pull request:
https://github.com/apache/helix/pull/224
[HELIX-718] implement ThreadCountBasedTaskAssigner
In this RB, I implemented a thread count based task assigner that is
optimized for short-term use cases. It assumes:
- All tasks to assign have same quota type
- All tasks to assign requires only 1 thread
The algorithms did best effort that tasks with same type / same job are
spread out: i.e.
- if there are 3 nodes, each has 10 threads for each quota type A, B, and C
- node1 is empty, node2 and node3 each has 5 typeB tasks and 5 typeC tasks
running
=> when 3 typeA tasks are to be assigned, it will assign 1 typeA task to
each node rather than squeeze all 3 typeA tasks to node1.
Added tests for the assigner. Below is the profiling results, each result
takes average of 100 trails:
Assign 50K tasks onto 1K nodes:
testing batch size: 10000
Average time: 118ms
testing batch size: 5000
Average time: 114ms
testing batch size: 2000
Average time: 117ms
testing batch size: 1000
Average time: 119ms
testing batch size: 500
Average time: 123ms
testing batch size: 100
Average time: 182ms
Assign 10K tasks onto 1K nodes:
testing batch size: 10000
Average time: 25ms
testing batch size: 5000
Average time: 21ms
testing batch size: 2000
Average time: 22ms
testing batch size: 1000
Average time: 25ms
testing batch size: 500
Average time: 22ms
testing batch size: 100
Average time: 34ms
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhan849/helix harry/simple-assigner
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/helix/pull/224.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #224
----
commit 6cb574d5aea6ca9cb9e6b5184bc80cb5e05d53b8
Author: Harry Zhang <hrzhang@...>
Date: 2018-07-09T23:04:19Z
[HELIX-718] implement ThreadCountBasedTaskAssigner
----
---
