Ed Leafe wrote:
Wow, I seem to have unleashed a bunch of pent-up frustration in the
community! It's great to see everyone coming forward with their ideas
and insights for improving the way Nova (and, by extension, all of
OpenStack) can potentially scale.
I do have a few comments on the discussion:
1) This isn't a proposal to simply add some sort of DLM to Nova as a
magic cure-all. The concerns about Nova's ability to scale have to do
a lot more with the overall internal communication design.
2) I really liked the comment about "made-up numbers". It's so true:
we are all impressed by such examples of speed that we sometimes
forget whether speeding up X will improve the overall process to any
significant degree. The purpose of my original email back in July,
and the question I asked at the Nova midcycle, is if we could get
some numbers that would be a target to shoot for with any of these
experiments. Sure, I could come up with a test that shows a zillion
transactions per second, but if that doesn't result in a cloud being
able to schedule more efficiently, what's the point?
3) I like the idea of something like ZooKeeper, but my concern is how
to efficiently query the data. If, for example, we had records for
100K compute nodes, would it be possible to do the equivalent of
"SELECT * FROM resources WHERE resource_type = 'compute' AND
free_ram_mb>= 2048 AND …" - well, you get the idea. Are complex data
queries possible in ZK? I haven't been able to find that information
anywhere.
The idea is that u wouldn't do these queries against any remote source
in the first place. Instead a scheduler would get notified (via a
concept like
http://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#sc_zkDataMode_watches)
when a hypervisor updates its data in zookeeper (or other equivalent
system); when that notification happens the scheduler then reads the
data then updates some *local* data source with that information (this
could be a in-memory dict, or a local sqlite, or something else better
optimized for searching fast) and then from that point on that local
source is used to do queries on. This way a hypervisor (compute-node) is
performing *nearly* the equivalent of a push notification (like on your
phone) to schedulers.
4) It is true that even in a very large deployment, it is possible to
keep all the relevant data needed for scheduling in memory. My
concern is how to efficiently search that data, much like in the ZK
scenario.
See above.
5) Concerns about Cassandra running with OpenJDK instead of the
Oracle JVM are troubling. I sent an email about this to one of the
people I know at DataStax, but so far have not received a response.
And while it would be great to have people contribute to OpenJDK to
make it compatible, keep in mind that that would be an ongoing
commitment, not just a one-time effort.
6) I remember discussions back in the Austin-Bexar time frame about
what Thierry referred to as 'flavor-based schedulers', and they were
immediately discounted as not sophisticated enough to handle the sort
of complex scheduling requests that were expected. I'd be interested
in finding out from the big cloud providers what percentage of their
requests would fall into this simple structure, and what percent are
more complicated than that. Having hosts listening to queues that
they know they can satisfy removes the raciness from the process,
although it would require some additional handling for the situation
where no host accepts the request. Still, it has the advantage of
being dead simple. Unfortunately, this would probably require a
bigger architectural change than integrating Cassandra into the
Scheduler would.
Another discussion that also should get talked about, but is again much
larger in scope: https://review.openstack.org/#/c/210549/ (still WIP but
the idea/problem/issue hopefully is clear).
I hope that those of us who will be at the Tokyo Summit and are
interested in these ideas can get together for an informal
discussion, and come up with some ideas for grand experiments and
reality checks. ;-)
BTW, I started playing around with some ideas, and thought that if
anyone wanted to also try Cassandra, I'd write up a quick how-to for
setting up a small cluster:
http://blog.leafe.com/small-scale-cassandra/. Using docker images
makes it a breeze!
-- Ed Leafe
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
