On 06/09/2017 03:09 PM, Monty Taylor wrote:
On 06/09/2017 02:35 PM, Clark Boylan wrote:
On Fri, Jun 9, 2017, at 09:22 AM, Monty Taylor wrote:
Hey all!
Tristan has recently pushed up some patches related to providing a Web
Dashboard for Zuul. We have a web app for nodepool. We already have the
Github webhook receiver which is inbound http. There have been folks who
have expressed interest in adding active-REST abilities for performing
actions. AND we have the new websocket-based log streaming.
We're currently using Paste for HTTP serving (which is effectively
dead), autobahn for websockets and WebOB for request/response
processing.
This means that before we get too far down the road, it's probably time
to pick how we're going to do those things in general. There are 2
questions on the table:
* HTTP serving
* REST framework
They may or may not be related, and one of the options on the table
implies an answer for both. I'm going to start with the answer I think
we should pick:
*** tl;dr ***
We should use aiohttp with no extra REST framework.
Meaning:
- aiohttp serving REST and websocket streaming in a scale-out tier
- talking RPC to the scheduler over gear or zk
- possible in-process aiohttp endpoints for k8s style health endpoints
Since we're talking about a web scale-out tier that we should just have
a single web tier for zuul and nodepool. This continues the thinking
that nodepool is a component of Zuul.
I'm not sure that this is a great idea. We've already seen that people
have wanted to use nodepool without a Zuul and even without performing
CI. IIRC paul wanted to use it to keep a set of asterisks floating
around for example. We've also seen that people want to use
subcomponents of nodepool to build and manage a set of images for clouds
without making instances.
Excellent point.
In the past we have been careful to keep logical tools separate which
has made it easy for us to add new tools and remove old ones.
Operationally this may be perceived as making things more difficult to a
newcomer, but it makes life much much better 3-6 months down the road.
In order to write zuul jobs, end-users must know what node labels are
available. A zuul client that says "please get me a list of available
node labels" could make sense to a user. As we get more non-OpenStack
users, those people may not have any concept that there is a separate
thing called "nodepool".
*** The MUCH more verbose version ***
I'm now going to outline all of the thoughts and options I've had or
have heard other people say. It's an extra complete list - there are
ideas in here you might find silly/bad. But since we're picking a
direction, I think it's important we consider the options in front of
us.
This will cover 3 http serving options:
- WSGI
- aiohttp
- gRPC
and 3 REST framework options:
- pecan
- flask-restplus
- apistar
** HTTP Serving **
WSGI
The WSGI approach is one we're all familiar with and it works with
pretty much every existing Python REST framework. For us I believe if we
go this route we'd want to serve it with something like uwsgi and
Apache. That adds the need for an Apache layer and/or management uwsgi
process. However, it means we can make use of normal tools we all likely
know at least to some degree.
FWIW I don't think Apache would be required. uWSGI is a fairly capable
http server aiui. You can also pip install uwsgi so the simple case
remains fairly simple I think.
Also good point.
A downside is that we'll need to continue to handle our Websockets work
independently (which is what we're doing now)
Because it's in a separate process, the API tier will need to make
requests of the scheduler over a bus, which could be either gearman or
zk.
Note that OpenStack has decided that this is a better solution than
using web servers in the python process. That doesn't necessarily mean
it is the best choice for Zuul, but it seems like there is a lot we can
learn from the choice to switch to WSGI in OpenStack.
Yah. I definitely more strongly lean towards external.
aiohttp
Zuul v3 is Python3, which means we can use aiohttp. aiohttp isn't
particularly compatible with the REST frameworks, but it has built-in
route support and helpers for receiving and returning JSON. We don't
need ORM mapping support, so the only thing we'd really be MISSING from
REST frameworks is auto-generated documentation.
aiohttp also supports websockets directly, so we could port the autobahn
work to use aiohttp.
aiohttp can be run in-process in a thread. However, websocket
log-streaming is already a separate process for scaling purposes, so if
we decide that one impl backend is a value, it probably makes sense to
just stick the web tier in the websocket scaleout process anyway.
However, we could probably write a facade layer with a gear backend and
an in-memory backend so that simple users could just run the in-process
version but scale-out was possible for larger installs (like us)
Since aiohttp can be in-process, it also allows us to easily add some
'/health' endpoints to all of our services directly, even if they aren't
intended to be publicly consumable. That's important for running richly
inside of things like kubernetes that like to check in on health status
of services to know about rescheduling them. This way we could add a
simple thread to the scheduler and the executors and the mergers and the
nodepool launchers and builders that adds a '/health' endpoint.
See above. OpenStack has decided this is the wrong route to take
(granted with eventlet and python2.7 not asyncio and python3.5). There
are scaling and debugging challenges faced when you try to run an in
process web server.
Well - two different things here.
Actual endpoint = external.
But for the k8s/prometheus request for /health endpoints, the endpoint
needs to actually be per service that is running in a k8s container,
because it's how k8s knows what to do. So for openstack, we'll be
starting with a top-level API service provided /health endpoint that'll
work like things work. But then eventually each of the services,
nova-compute, nova-scheduler, nova-conductor, etc - would each get their
own simple /health endpoint that the controlling service manager can hit
for status of the process in question.
That's what I'm mostly advocating we can support with the ability to run
"in process". It's an "in process as well" that's important.
gRPC / gRPC-REST gateway
This is a curve-ball. We could define our APIs using gRPC. That gets us
a story for an API that is easily consumable by all sorts of clients,
and that supports exciting things like bi-directional streaming
channels. gRPC isn't (yet) consumable directly in browsers, nor does
Github send gRPC webhooks. BUT - there is a REST Gateway for gRPC:
https://github.com/grpc-ecosystem/grpc-gateway
that generates HTTP/1.1+JSON interfaces from the gRPC descriptions and
translates between protobuf and json automatically. The "REST" interface
it produces does not support url-based parameters, so everything is done
in payload bodies, so it's:
GET /nodes
{
'id': 1234
}
rather than
GET /nodes/1234
but that's still totally fine - and totally works for both status.json
and GH webhooks.
The catch is - grpc-gateway is a grpc compiler plugin that generates
golang code. So we'd either have to write our own plugin that does the
same thing but for generating python code, or we'd have to write our
gRPC/REST layer in go. I betcha folks would appreciate if we implemented
the plugin for python, but that's a long tent-pole for this purpose so I
don't honestly think we should consider it. Therefore, we should
consider that using gRPC + gRPC-REST implies writing the web-tier in go.
That obviously implies an additional process that needs to talk over an
RPC bus.
There are clear complexity costs involved with adding a second language
component, especially WRT deployment. (pip install zuul would not be
sufficient) OTOH - it would open the door to using protobuf-based
objects for internal communication, and would open the door for rich
client apps without REST polling and also potentially nice Android apps
(gRPC works great for mobile apps) I think that makes it a hard sell.
THAT SAID - there are only 2 things that explicitly need REST over HTTP
1.1 - thats the github webhooks and status.json. We could write
everything in gRPC except those two. Browser support for gRPC is coming
soon (they've moved from "someone is working on it" to "contact us about
early access") so status.json could move to being pure gRPC as well ...
and the webhook endpoint is pretty simple, so just having it be an
in-process aiohttp handler isn't a terrible cost. So if we thought
"screw it, let's just gRPC and not have an HTTP/1.1 REST interface at
all" - we can stay all in python and gRPC isn't a huge cost at that
point.
gRPC doesn't handle websockets - but we could still run the gRPC serving
and the websocket serving out of the same scale-out web tier.
Another data point for gRPC is that the etcd3 work in OpenStack found
that the existing python lib(s) for grpc don't play nice with eventlet
or asyncio or anything that isn't Thread()
(https://github.com/grpc/grpc/issues/6046 is the bug tracking that I
think). This would potentially make the use of asyncio elsewhere
(websockets) more complicated.
Ah - thanks! I didn't realize it touched asyncio too - I thought it was
just eventlet related.
Nevermind then. :)
** Summary
Based on the three above, it seems like we need to think about separate
web-tier regardless of choice. The one option that doesn't strictly
require a separate tier is the one that lets us align on websockets, so
it seems that co-location there would be simple.
aiohttp seems like the cleanest forward path. It'll require reworking
the autobahn code (sorry Shrews) - but is nicely aligned with our
Python3 state. It's new - but it's not as totally new as gRPC is. And
since we'll already have some websockets stuff, we could also write
streaming websockets APIs for the things where we'd want that from gRPC.
* REST Framework
If we decide to go the WSGI route, then we need to talk REST frameworks
(and it's possible we decide to go WSGI because we want to use a REST
framework)
I'm not sure I understand why the WSGI and REST frameworks are being
conflated. You can do one or the other or both and whichever you choose
shouldn't affect the other too much aiui. There is even a flask-aiohttp
lib.
Well, because WSGI + gRPC is obviously out. :)
But from flask-aiohttp's readme:
"""
I made this project for testing compatability between WSGI & Async IO.
Since WSGI has no consideration of Async IO, Flask-aiohttp cannot be
perfect.
So, I don't recommend you to use this library for production. Libraries
that was made for Async IO would be better choice (Like gevent, Tornado
or AioHTTP).
"""
I mostly am considering that if we pick aiohttp, since it has routing
and request mashaling already baked in, there is no real value in adding
an additional lib on top of it, as least not at this point. Maybe in the
future someone will write a $something that layers on top of aiohttp and
makes it better - but aiohttp itself seems pretty complete.
So you're right - it's not strictly necessary to conflate, but the only
http-layer scenario where we'd be considering these seriously is the
WSGI choice.
The assumption in this case is that the websocket layer is a separate
entity.
There are three 'reasonable' options available:
- pecan
- flask-restplus
- apistar
pecan
pecan is used in a lot of OpenStack services and is also used by
Storyboard, so it's well known. Tristan's patches so far use Pecan, so
we've got example code.
On the other hand, Pecan seems to be mostly only used in OpenStack land
and hasn't gotten much adoption elsewhere.
flask-restplus
https://flask-restplus.readthedocs.io/en/stable/
flask is extremely popular for folks doing REST in Python.
flask-restplus is a flask extension that also produces Swagger Docs for
the REST api, and provides for serving an interactive swagger-ui based
browseable interface to the API. You can also define models using
JSONSchema. Those are not needed for simple cases like status.json, but
for fuller REST API might be nice.
Of course, in all cases we could simply document our API using swagger
and get the same thing - but that does involve maintaining model/api
descriptions and documentation separately.
It's worth noting (I just found it myself) that we can do swagger with
aiohttp too:
http://aiohttp-swagger.readthedocs.io/en/latest/
apistar
https://github.com/tomchristie/apistar
apistar is BRAND NEW and was announced at this year's PyCon. It's from
the Django folks and is aimed at writing REST separate from Django.
It's python3 from scratch - although it's SO python3 focused that it
requires python 3.6. This is because it makes use of type annotations:
Type hinting is in python 3.5 and apistar's trove identifer things
mention 3.5 support (not sure if actually the case though). But if so
3.5 is far easier to use since it is in more distros than Arch and
Tumbleweed (like with 3.6).
Ah - neat. Their code examples are all showing f'' strings which are 3.6
only, so I was assuming 3.6 required.
def show_request(request: http.Request):
return {
'method': request.method,
'url': request.url,
'headers': dict(request.headers)
}
def create_project() -> Response:
data = {'name': 'new project', 'id': 123}
headers = {'Location': 'http://example.com/project/123/'}
return Response(data, status=201, headers=headers)
and f'' strings:
def echo_username(username):
return {'message': f'Welcome, {username}!'}
Python folks seem to be excited about apistar so far - but I think
python 3.6 is a bridge too far - it honestly introduces more deployment
issues as doing a golang-gRPC layer.
** Summary
I don't think the REST frameworks offer enough benefit to justify their
use and adopting WSGI as our path forward.
Yesterday SpamapS mentioned wanting to be able to grow the Zuul
community. Just based on looking at the choices OpenStack is making
(moving TO wsgi) and the general populatity of Flask in the python
community I think that you may want to consider both wsgi and flask
simply because they are tools that are known to scale reasonably well
and many people are familiar with them.
If we decided to do WSGI I would totally strongly advocate we use Flask.
** Thoughts on RPC Bus **
gearman is a simple way to add RPC calls between an API tier and the
scheduler. However, we got rid of gear from nodepool already, and we
intend on getting rid of gearman in v4 anyway.
If we use zk, we'll have to do a little bit more thinking about how to
do the RPC calls which will make this take more work. BUT - it means we
can define one API that covers both Zuul and Nodepool and will be
forward compatible with a v4 no-gearman world.
We *could* use gearman in zuul and run an API in-process in nodepool.
Then we could take a page out of early Nova and do a proxy-layer in zuul
that makes requests of nodepool's API.
We could just assume that there's gonna be an Apache fronting this stuff
and suggest deployment with routing to zuul and nodepool apis with
mod_proxy rules.
Finally, as clarkb pointed out in response to the ingestors spec, we
could introduce MQTT and use it. I'm wary of doing that for this because
it introduces a totally required new tech stack at a late stage.
Mostly I was just pointing out that I think the vast majority of the
infastructure work to have something like a zuul ingestor is done. You
just have to read from an mqtt connection instead of a gerrit ssh
connection. Granted this does require running more services (mqtt server
and the event stream handler) and doesn't handle entities like Github.
That said MQTT unlike Gearman seems to be seeing quite a bit of
development activity due to the popularity of IoT. Gearman has worked
reasonably well for us though so I don't think we need to just replace
it to get in on the IoT bandwagon.
The concerns I have wrt ingestors aren't protocol based so much but
about missing events.
It's a pretty poor user experience when Zuul misses an event. Because
the events from gerrit come from the ssh event stream, it means
ultimately that you need to have more than one listener attached to the
stream and then for the events to go through a de-dup phase so that jobs
don't get triggered more than once. This is mostly thinking about
updating zuul - which is a scenario in which we miss events while we're
down.
If we use MQTT, I believe it gives us the ability to pick up somewhere
when we connect - but I'm not certain if ingesting the events into the
MQTT service itself is HA enough that we can restart the MQTT services.
But the same principles could be applied - more than one event stream
handler, potentially just using zk in a leader-election kind of manner
if needed, writing the events themselves to MQTT server. I'll definitely
be excited to learn more - but at some point in the future I'd like to
be able to frequently roll updates without missing events.
Since we're starting fresh, I like the idea of a single API service that
RPCs to zuul and nodepool, so I like the idea of using ZK for the RPC
layer. BUT - using gear and adding just gear worker threads back to
nodepol wouldn't be super-terrible maybe.
Nodepool hasn't had a Gearman less release yet so you don't have to
worry about backward compat at least.
:)
** Final Summary **
As I tl;dr'd earlier, I think aiohttp co-located with the scale-out
websocket tier talking to the scheduler over zk is the best bet for us.
I think it's both simple enough to adopt and gets us a rich set of
features. It also lets us implement in-process simple health endpoints
on each service with the same tech stack.
I'm wary of this simply because it looks a lot like repeating
OpenStack's (now failed) decision to stick web servers in a bunch of
python processes then do cooperative multithreading with them along with
all your application logic. It just gets complicated. I also think this
underestimates the value of using tools people are familiar with (wsgi
and flask) particularly if making it easy to jump in and building
community is a goal.
I must clearly have just said something incorrectly, as that's not what
I'm suggesting AT ALL.
I'm suggesting a zuul-api service that stateless/scale-out able and is
independent of the application. Then that service, like the OpenStack
services, would get its info from the application via $something. In the
followups jeblair points out that we can currently do this via a mix of
methods - gearman or zk depending on what makes the most sense.
I think that approach is what we should do regardless of whether we do
WSGI or aiohttp.
I'm advocating aiohttp because it has enough REST helpers built in and
it does websockets out of the box, so we can use one library rather than
two for our varied HTTP needs.
Sorry if I rambled too much and made that not clear.
_______________________________________________
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
_______________________________________________
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
