On 5/5/2017 6:44 PM, Fox, Kevin M wrote:
Note, when I say OpenStack below, I'm talking about
nova/glance/cinder/neutron/horizon/heat/octavia/designate. No offence to the
other projects intended. just trying to constrain the conversation a bit...
Those parts are fairly comparable to what k8s provides.
I think part of your point is valid, that k8s isn't as feature rich in some
ways, (networking for example), and will get more complex in time. But it has a
huge amount of functionality for significantly less effort compared to an
OpenStack deployment with similar functionality today.
I think there are some major things different between the two projects that are
really paying off for k8s over OpenStack right now. We can use those as
learning opportunities moving forward or the gap will continue to widen, as
will the user migrations away from OpenStack. These are mostly architectural
things.
Versions:
* The real core of OpenStack is essentially version 1 + iterative changes.
* k8s is essentially the third version of Borg. Plenty of room to ditch bad
ideas/decisions.
That means OpenStack's architecture has essentially grown organically rather
then being as carefully thought out. The backwards compatibility has been a
good goal, but its so hard to upgrade most places burn it down and stand up
something new anyway so a lot of work with a lot less payoff then you would
think. Maybe it is time to consider OpenStack version 2...
I think OpenStack's greatest strength is its standardized api's. Thus far we've
been changing the api's over time and keeping the implementation mostly the
same... maybe we should consider keeping the api the same and switch some of
the implementations out... It might take a while to get back to where we are
now, but I suspect the overall solution would be much better now that we have
so much experience with building the first one.
k8s and OpenStack do largely the same thing. get in user request, schedule the
resource onto some machines and allow management/lifecycle of the thing.
Why then does k8s scalability goal target 5000 nodes and OpenStack really
struggle with more then 300 nodes without a huge amount of extra work? I think
its architecture. OpenStack really abuses rabbit, does a lot with relational
databases that maybe are better done elsewhere, and forces isolation between
projects that maybe is not the best solution.
Part of it I think is combined services. They don't have separate services for
cinder-api/nova-api,neutron-api/heat-api/etc. Just kube-apiserver. same with
the *-schedulers, just kube-scheduler. This means many fewer things to manage
for ops, and allows for faster communication times (lower latency). In theory
OpenStack could scale out much better with the finer grained services. but I'm
not sure thats really ever shown true in practice.
Layered/eating own dogfood:
* OpenStack assumes the operator will install "all the things".
* K8s uses k8s to deploy lots of itself.
You use kubelet with the same yaml file format normally used to deploy stuff to
deploy etcd/kube-apiserver/kube-scheduler/kube-controller-manager to get a
working base system.
You then use the base system to launch sdn, ingress, service-discovery, the ui,
etc.
This means the learning required is substantially less when it comes to
debugging problems, performing upgrades, etc, because its the same for the most
part for k8s as it is for any other app running on it. The learning costs is
way lower.
Versioning:
* With OpenStack, Upgrades are hard, mismatched version servers/agents are
hard/impossible.
* K8s, they support the controllers being 2 versions ahead of the clients.
Its hard to bolt this on after the fact, but its also harder when you have
multiple communications channels to do it with. Having to do it in http, sql,
and rabbit messages makes it so much harder. Having only one place talk to the
single datastore (etcd) makes that easier, as is only having one place
everything interacts with the servers kube-apiserver.
Some amount of distribution:
* OpenStack components are generally expected to come from distro's.
* K8s, Core pieces like kube-apiserver are distributed prebuilt and ready to
go in container images if you chose to use them.
Minimal silo's:
* the various OpenStack projects are very silo'ed.
* Most of the k8s subsystems currently are all tightly integrated with each
other and are managed by the same teams.
This has lead to architectural decisions that take more of the bigger picture
into account. Under OpenStack's current model, each project does their own
thing without too much design into how it all comes together in the end. A lot
of problems spring from this. I'm sure k8s will get more and more silo'ed as it
grows and matures. But right now, the lack of silo's really are speeding its
development.
Anyway, I think I'm done rambling for now... I do hope we can reflect on some
of the differences between the projects and see if we can figure out how to
pull in some of the good ideas from k8s.
Thanks,
Kevin
Am I stating the obvious when I say that in the beginning there were two
projects in OpenStack: Nova and Swift. Cinder and Glance sprang forth
from Nova. We had nova-network (we still do). Neutron didn't come from
Nova and it's taken us a few years to finally get to the place where we
deprecated nova-network and made Neutron the default networking backend
for Nova.
Yes things could move quick and dirty when it was essentially a single
project. We're still paying for a lot of quick and dirty technical debt
within Nova, and a lot of old code we've either deprecated or are
working on deprecating (like rxtx_factor in flavors and personality
files to name a couple but there are really a ton of examples - quotas,
bdm v1, cells v1, etc).
Splitting nova-volume out to Cinder still left a *ton* of tight-coupling
between projects. That's a major reason it's taken so long to get volume
multi-attach supported in Nova, because the interfaces between the
projects were just bad (lots of shared data and knowledge about the
internals of each project rather than coding to clean REST API
interfaces). We're working on correcting that now, but it takes time.
Frankly I'm fine with the projects doing their own thing as long as
there is a clean API (with microversions for changes) between them.
It's an interesting idea to think what it would be like if everything
was a couple of projects, but it also scares me. :)
It's also fun to think about just dropping backward compatibility on the
API and do a brand new major version, but we talked about that with Nova
v3 for how many releases? At least 3 before deciding that microversions
was the best way forward. We're going to be talking about raising
minimum microversions in the API in Boston and I'm eager to get feedback
on that idea.
--
Thanks,
Matt
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
