On 06/27/2017 01:03 PM, Monty Taylor wrote:
On 06/12/2017 03:17 PM, James E. Blair wrote:
Clint Byrum <[email protected]> writes:
Excerpts from corvus's message of 2017-06-09 13:11:00 -0700:
Clark Boylan <[email protected]> writes:
I'm wary of this simply because it looks a lot like repeating
OpenStack's (now failed) decision to stick web servers in a bunch of
python processes then do cooperative multithreading with them along
with
all your application logic. It just gets complicated. I also think
this
underestimates the value of using tools people are familiar with (wsgi
and flask) particularly if making it easy to jump in and building
community is a goal.
I agree that mixing an asyncio based httpserver with application logic
using cooperative multithreading is not a good idea. Happily that is
not the proposal. The proposal is that the webserver be a separate
process from the rest of Zuul, it would be an independently scaleable
component, and *only* the webserver would use asyncio.
I'm not totally convinced that having an HTTP service in the scheduler
that gets proxied to when appropriate is the worst idea in the short
term,
since we already have one and it already works reasonably well with
paste,
we just want to get rid of paste faster than we can refactor it out by
making a ZK backend.
Even if we remove paste and create a web tier aiohttp thing, we end up
writing most of what would be complex about doing it in-process in the
scheduler. So, to tack gearman on top of that, versus just letting the
reverse proxy do its job, seems like extra work.
What I'd like to get out of this conversation is a shared understanding
of what the web tier for Zuul should look like in the future, so that we
can know where we want to end up eventually, but *not* a set of
additional requirements for Zuul v3.0. In other words, I think this is
a long-term, rather than short-term conversation.
The way I see it is that we're adding a bunch of new functionality to an
area of Zuul that we've traditionally kept very simple. We're growing
from a simple JSON endpoint to support websockets, event injection via
hooks, and a full-blown API for historic data.
That last item in particular calls out for a real web framework. Since
it is new work and has substantial interaction with the web framework,
it would be good to know what our end state is, so that folks working on
it can go ahead and head in that direction.
The other aspects, which are largely already implemented, can be ported
over in the fullness of time.
We do not need to change how we are doing webhooks or log streaming for
Zuul v3.0.
In fact, I imagine that at least initially, we would implement something
in openstack-infra like what you describe, Clint. We will have an
Apache server which proxies status.json requests and webhooks to
zuul-scheduler, and proxies websocket requests to the streaming server.
As time permits, we can incorporate those into a comprehensive web
server with the framework we choose.
Does that sound like a good plan?
Does aiohttp alone fit the bill as Monty suggests, or do we need to
consider something else?
As a followup. We have discovered that autobahn is problematic with the
manner in which we want to run it, and that aiohttp works great - so the
websocket streaming will be switching to autobahn. It has also been
will be switching to aiohttp
I suck
verified that this works great running inside of a thread, so we're fine
with the use cases we're hoping for.
With that in mind, I believe the path should be for the new server being
written for the console-log streaming to be called "zuul-web" and that
it should serve the console-log websocket at /console-stream or
/console-log or something.
We can then use our Apache frontend to serve /status from the
zuul-scheduler and /console-stream from zuul-web - and in the fullness
of time we can potentially move the webhooks and the status page from
the scheduler to zuul-web should we choose.
Additionally, as we look at things like dashboards, they can go directly
into zuul-web and be written with aiohttp. As zuul-web is stateless,
it's a great candidate for a pure scaleout model.
Additionally, if a deployer would prefer to run it in a pre-fork WSGI
model, it comes with a uvloop enabled Gunicorn Worker:
aiohttp.GunicornUVLoopWebWorker
http://aiohttp.readthedocs.io/en/stable/deployment.html#nginx-gunicorn
There's a good model for:
- simple in-process - good for unit tests and also for AIO deploys
- in-process scale-out - deployer uses something like kubernetes and
gets scale by just running more than one container
- worker pool - running a pool of workers on a machine with Apache or
nginx with gunicorn managing
Which I believe means all of our bases are covered and there's no need
to introduce additional technologies.
Monty
_______________________________________________
OpenStack-Infra mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
_______________________________________________
OpenStack-Infra mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
