Clint Byrum wrote:
Excerpts from Boris Pavlovic's message of 2015-10-11 00:02:39 -0700:
2Everybody,
Just curios why we need such complexity.
Let's take a look from other side:
1) Information about all hosts (even in case of 100k hosts) will be less
then 1 GB
2) Usually servers that runs scheduler service have at least 64GB RAM and
more on the board
3) math.log(100000)< 12 (binary search per rule)
4) We have less then 20 rules for scheduling
5) Information about hosts is updated every 60 seconds (no updates host is
dead)
According to this information:
1) We can store everything in RAM of single server
2) We can use Python
3) Information about hosts is temporary data and shouldn't be stored in
persistence storage
Simplest architecture to cover this:
1) Single RPC service that has two methods: find_host(rules),
update_host(host, data)
2) Store information about hosts like a dict (host_name->data)
3) Create for each rule binary tree and update it on each host update
4) Make a algorithm that will use binary trees to find host based on rules
5) Each service like compute node, volume node, or neutron will send
updates about host
that they managed (cross service scheduling)
6) Make a algorithm that will sync host stats in memory between different
schedulers
I'm in, except I think this gets simpler with an intermediary service
like ZK/Consul to keep track of this 1GB of data and replace the need
for 6, and changes the implementation of 5 to "updates its record and
signals its presence".
What you've described is where I'd like to experiment, but I don't want
to reinvent ZK or Consul or etcd when they already exist and do such a
splendid job keeping observers informed of small changes in small data
sets. You still end up with the same in-memory performance, and this is
in line with some published white papers from Google around their use
of Chubby, which is their ZK/Consul.
+1 let's not recreate this; the code @ paste.openstack.org/show/475941/
basically does 1-6 and within about ~100 lines, it doesn't optimize
things into a binary tree, but thats easily doable... for all I care put
the information received into N trees (perhaps even using
http://docs.openstack.org/developer/taskflow/types.html#module-taskflow.types.tree)
and do searches across those as desired (and this is where u can get
into considering something like numpy to help).
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
