Re: [OpenStack-Infra] On the subject of HTTP interfaces and Zuul
Excerpts from corvus's message of 2017-06-09 13:11:00 -0700: > Clark Boylan writes: > > > I'm wary of this simply because it looks a lot like repeating > > OpenStack's (now failed) decision to stick web servers in a bunch of > > python processes then do cooperative multithreading with them along with > > all your application logic. It just gets complicated. I also think this > > underestimates the value of using tools people are familiar with (wsgi > > and flask) particularly if making it easy to jump in and building > > community is a goal. > > I agree that mixing an asyncio based httpserver with application logic > using cooperative multithreading is not a good idea. Happily that is > not the proposal. The proposal is that the webserver be a separate > process from the rest of Zuul, it would be an independently scaleable > component, and *only* the webserver would use asyncio. > I'm not totally convinced that having an HTTP service in the scheduler that gets proxied to when appropriate is the worst idea in the short term, since we already have one and it already works reasonably well with paste, we just want to get rid of paste faster than we can refactor it out by making a ZK backend. Even if we remove paste and create a web tier aiohttp thing, we end up writing most of what would be complex about doing it in-process in the scheduler. So, to tack gearman on top of that, versus just letting the reverse proxy do its job, seems like extra work. ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] About aarch64 third party CI
On Fri, Jun 09, 2017 at 07:58:44PM +, Jeremy Stanley wrote: > On 2017-06-07 14:26:10 +0800 (+0800), Xinliang Liu wrote: > [...] > > we already have our own pre-built debian cloud image, could I just > > use it and not use the one built by diskimage-builder? > [...] > > The short answer is that nodepool doesn't currently have support for > directly using an image provided independent of its own image build > process. Clark was suggesting[*] in IRC today that it might be > possible to inject records into Zookeeper (acting as a "fake" > nodepool-builder daemon basically) to accomplish this, but nobody > has yet implemented such a solution to our knowledge. > > Longer term, I think we do want a feature in nodepool to be able to > specify the ID of a prebuilt image for a label/provider (at least we > discussed that we wouldn't reject the idea if someone proposed a > suitable implementation). Just be aware that nodepool's use of > diskimage-builder to regularly rebuild images is intentional and > useful since it ensures images are updated with the latest packages, > kernels, warm caches and whatever else you specify in your elements > so reducing job runtimes as they spend less effort updating these > things on every run. > > [*] http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2017-06-09.log.html#t2017-06-09T15:32:27-2 > > > -- > Jeremy Stanley Actually, I think 458073[1] aims to fix this use case. I haven't tired it myself but it adds support for using images which are not built and managed by nodepool. This is currently only on feature/zuulv3 branch. [1] https://review.openstack.org/#/c/458073/ > ___ > OpenStack-Infra mailing list > OpenStack-Infra@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] On the subject of HTTP interfaces and Zuul
On 06/09/2017 03:09 PM, Monty Taylor wrote: On 06/09/2017 02:35 PM, Clark Boylan wrote: On Fri, Jun 9, 2017, at 09:22 AM, Monty Taylor wrote: Hey all! Tristan has recently pushed up some patches related to providing a Web Dashboard for Zuul. We have a web app for nodepool. We already have the Github webhook receiver which is inbound http. There have been folks who have expressed interest in adding active-REST abilities for performing actions. AND we have the new websocket-based log streaming. We're currently using Paste for HTTP serving (which is effectively dead), autobahn for websockets and WebOB for request/response processing. This means that before we get too far down the road, it's probably time to pick how we're going to do those things in general. There are 2 questions on the table: * HTTP serving * REST framework They may or may not be related, and one of the options on the table implies an answer for both. I'm going to start with the answer I think we should pick: *** tl;dr *** We should use aiohttp with no extra REST framework. Meaning: - aiohttp serving REST and websocket streaming in a scale-out tier - talking RPC to the scheduler over gear or zk - possible in-process aiohttp endpoints for k8s style health endpoints Since we're talking about a web scale-out tier that we should just have a single web tier for zuul and nodepool. This continues the thinking that nodepool is a component of Zuul. I'm not sure that this is a great idea. We've already seen that people have wanted to use nodepool without a Zuul and even without performing CI. IIRC paul wanted to use it to keep a set of asterisks floating around for example. We've also seen that people want to use subcomponents of nodepool to build and manage a set of images for clouds without making instances. Excellent point. In the past we have been careful to keep logical tools separate which has made it easy for us to add new tools and remove old ones. Operationally this may be perceived as making things more difficult to a newcomer, but it makes life much much better 3-6 months down the road. In order to write zuul jobs, end-users must know what node labels are available. A zuul client that says "please get me a list of available node labels" could make sense to a user. As we get more non-OpenStack users, those people may not have any concept that there is a separate thing called "nodepool". *** The MUCH more verbose version *** I'm now going to outline all of the thoughts and options I've had or have heard other people say. It's an extra complete list - there are ideas in here you might find silly/bad. But since we're picking a direction, I think it's important we consider the options in front of us. This will cover 3 http serving options: - WSGI - aiohttp - gRPC and 3 REST framework options: - pecan - flask-restplus - apistar ** HTTP Serving ** WSGI The WSGI approach is one we're all familiar with and it works with pretty much every existing Python REST framework. For us I believe if we go this route we'd want to serve it with something like uwsgi and Apache. That adds the need for an Apache layer and/or management uwsgi process. However, it means we can make use of normal tools we all likely know at least to some degree. FWIW I don't think Apache would be required. uWSGI is a fairly capable http server aiui. You can also pip install uwsgi so the simple case remains fairly simple I think. Also good point. A downside is that we'll need to continue to handle our Websockets work independently (which is what we're doing now) Because it's in a separate process, the API tier will need to make requests of the scheduler over a bus, which could be either gearman or zk. Note that OpenStack has decided that this is a better solution than using web servers in the python process. That doesn't necessarily mean it is the best choice for Zuul, but it seems like there is a lot we can learn from the choice to switch to WSGI in OpenStack. Yah. I definitely more strongly lean towards external. aiohttp Zuul v3 is Python3, which means we can use aiohttp. aiohttp isn't particularly compatible with the REST frameworks, but it has built-in route support and helpers for receiving and returning JSON. We don't need ORM mapping support, so the only thing we'd really be MISSING from REST frameworks is auto-generated documentation. aiohttp also supports websockets directly, so we could port the autobahn work to use aiohttp. aiohttp can be run in-process in a thread. However, websocket log-streaming is already a separate process for scaling purposes, so if we decide that one impl backend is a value, it probably makes sense to just stick the web tier in the websocket scaleout process anyway. However, we could probably write a facade layer with a gear backend and an in-memory backend so that simple users could just run the in-process version but scale-out was possible for larger installs (like us) Since aiohttp can be in-p
Re: [OpenStack-Infra] On the subject of HTTP interfaces and Zuul
Clark Boylan writes: > I'm wary of this simply because it looks a lot like repeating > OpenStack's (now failed) decision to stick web servers in a bunch of > python processes then do cooperative multithreading with them along with > all your application logic. It just gets complicated. I also think this > underestimates the value of using tools people are familiar with (wsgi > and flask) particularly if making it easy to jump in and building > community is a goal. I agree that mixing an asyncio based httpserver with application logic using cooperative multithreading is not a good idea. Happily that is not the proposal. The proposal is that the webserver be a separate process from the rest of Zuul, it would be an independently scaleable component, and *only* the webserver would use asyncio. -Jim ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] On the subject of HTTP interfaces and Zuul
On 06/09/2017 02:35 PM, Clark Boylan wrote: On Fri, Jun 9, 2017, at 09:22 AM, Monty Taylor wrote: Hey all! Tristan has recently pushed up some patches related to providing a Web Dashboard for Zuul. We have a web app for nodepool. We already have the Github webhook receiver which is inbound http. There have been folks who have expressed interest in adding active-REST abilities for performing actions. AND we have the new websocket-based log streaming. We're currently using Paste for HTTP serving (which is effectively dead), autobahn for websockets and WebOB for request/response processing. This means that before we get too far down the road, it's probably time to pick how we're going to do those things in general. There are 2 questions on the table: * HTTP serving * REST framework They may or may not be related, and one of the options on the table implies an answer for both. I'm going to start with the answer I think we should pick: *** tl;dr *** We should use aiohttp with no extra REST framework. Meaning: - aiohttp serving REST and websocket streaming in a scale-out tier - talking RPC to the scheduler over gear or zk - possible in-process aiohttp endpoints for k8s style health endpoints Since we're talking about a web scale-out tier that we should just have a single web tier for zuul and nodepool. This continues the thinking that nodepool is a component of Zuul. I'm not sure that this is a great idea. We've already seen that people have wanted to use nodepool without a Zuul and even without performing CI. IIRC paul wanted to use it to keep a set of asterisks floating around for example. We've also seen that people want to use subcomponents of nodepool to build and manage a set of images for clouds without making instances. Excellent point. In the past we have been careful to keep logical tools separate which has made it easy for us to add new tools and remove old ones. Operationally this may be perceived as making things more difficult to a newcomer, but it makes life much much better 3-6 months down the road. In order to write zuul jobs, end-users must know what node labels are available. A zuul client that says "please get me a list of available node labels" could make sense to a user. As we get more non-OpenStack users, those people may not have any concept that there is a separate thing called "nodepool". *** The MUCH more verbose version *** I'm now going to outline all of the thoughts and options I've had or have heard other people say. It's an extra complete list - there are ideas in here you might find silly/bad. But since we're picking a direction, I think it's important we consider the options in front of us. This will cover 3 http serving options: - WSGI - aiohttp - gRPC and 3 REST framework options: - pecan - flask-restplus - apistar ** HTTP Serving ** WSGI The WSGI approach is one we're all familiar with and it works with pretty much every existing Python REST framework. For us I believe if we go this route we'd want to serve it with something like uwsgi and Apache. That adds the need for an Apache layer and/or management uwsgi process. However, it means we can make use of normal tools we all likely know at least to some degree. FWIW I don't think Apache would be required. uWSGI is a fairly capable http server aiui. You can also pip install uwsgi so the simple case remains fairly simple I think. Also good point. A downside is that we'll need to continue to handle our Websockets work independently (which is what we're doing now) Because it's in a separate process, the API tier will need to make requests of the scheduler over a bus, which could be either gearman or zk. Note that OpenStack has decided that this is a better solution than using web servers in the python process. That doesn't necessarily mean it is the best choice for Zuul, but it seems like there is a lot we can learn from the choice to switch to WSGI in OpenStack. Yah. I definitely more strongly lean towards external. aiohttp Zuul v3 is Python3, which means we can use aiohttp. aiohttp isn't particularly compatible with the REST frameworks, but it has built-in route support and helpers for receiving and returning JSON. We don't need ORM mapping support, so the only thing we'd really be MISSING from REST frameworks is auto-generated documentation. aiohttp also supports websockets directly, so we could port the autobahn work to use aiohttp. aiohttp can be run in-process in a thread. However, websocket log-streaming is already a separate process for scaling purposes, so if we decide that one impl backend is a value, it probably makes sense to just stick the web tier in the websocket scaleout process anyway. However, we could probably write a facade layer with a gear backend and an in-memory backend so that simple users could just run the in-process version but scale-out was possible for larger installs (like us) Since aiohttp can be in-process, it also allows us to easily add some '/he
Re: [OpenStack-Infra] About aarch64 third party CI
On 2017-06-07 14:26:10 +0800 (+0800), Xinliang Liu wrote: [...] > we already have our own pre-built debian cloud image, could I just > use it and not use the one built by diskimage-builder? [...] The short answer is that nodepool doesn't currently have support for directly using an image provided independent of its own image build process. Clark was suggesting[*] in IRC today that it might be possible to inject records into Zookeeper (acting as a "fake" nodepool-builder daemon basically) to accomplish this, but nobody has yet implemented such a solution to our knowledge. Longer term, I think we do want a feature in nodepool to be able to specify the ID of a prebuilt image for a label/provider (at least we discussed that we wouldn't reject the idea if someone proposed a suitable implementation). Just be aware that nodepool's use of diskimage-builder to regularly rebuild images is intentional and useful since it ensures images are updated with the latest packages, kernels, warm caches and whatever else you specify in your elements so reducing job runtimes as they spend less effort updating these things on every run. [*] http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2017-06-09.log.html#t2017-06-09T15:32:27-2 > -- Jeremy Stanley signature.asc Description: Digital signature ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] On the subject of HTTP interfaces and Zuul
On Fri, Jun 09, 2017 at 12:35:59PM -0700, Clark Boylan wrote: > On Fri, Jun 9, 2017, at 09:22 AM, Monty Taylor wrote: > > Hey all! > > > > Tristan has recently pushed up some patches related to providing a Web > > Dashboard for Zuul. We have a web app for nodepool. We already have the > > Github webhook receiver which is inbound http. There have been folks who > > have expressed interest in adding active-REST abilities for performing > > actions. AND we have the new websocket-based log streaming. > > > > We're currently using Paste for HTTP serving (which is effectively > > dead), autobahn for websockets and WebOB for request/response processing. > > > > This means that before we get too far down the road, it's probably time > > to pick how we're going to do those things in general. There are 2 > > questions on the table: > > > > * HTTP serving > > * REST framework > > > > They may or may not be related, and one of the options on the table > > implies an answer for both. I'm going to start with the answer I think > > we should pick: > > > > *** tl;dr *** > > > > We should use aiohttp with no extra REST framework. > > > > Meaning: > > > > - aiohttp serving REST and websocket streaming in a scale-out tier > > - talking RPC to the scheduler over gear or zk > > - possible in-process aiohttp endpoints for k8s style health endpoints > > > > Since we're talking about a web scale-out tier that we should just have > > a single web tier for zuul and nodepool. This continues the thinking > > that nodepool is a component of Zuul. > > I'm not sure that this is a great idea. We've already seen that people > have wanted to use nodepool without a Zuul and even without performing > CI. IIRC paul wanted to use it to keep a set of asterisks floating > around for example. We've also seen that people want to use > subcomponents of nodepool to build and manage a set of images for clouds > without making instances. > Ya, asterisk use case aside, I think image build as a service is a prime example of something nodepool could be great at on its own. Especially now that nodepool-builder is scaling out very well with zookeeper. > In the past we have been careful to keep logical tools separate which > has made it easy for us to add new tools and remove old ones. > Operationally this may be perceived as making things more difficult to a > newcomer, but it makes life much much better 3-6 months down the road. > > > > > In order to write zuul jobs, end-users must know what node labels are > > available. A zuul client that says "please get me a list of available > > node labels" could make sense to a user. As we get more non-OpenStack > > users, those people may not have any concept that there is a separate > > thing called "nodepool". > > > > *** The MUCH more verbose version *** > > > > I'm now going to outline all of the thoughts and options I've had or > > have heard other people say. It's an extra complete list - there are > > ideas in here you might find silly/bad. But since we're picking a > > direction, I think it's important we consider the options in front of us. > > > > This will cover 3 http serving options: > > > > - WSGI > > - aiohttp > > - gRPC > > > > and 3 REST framework options: > > > > - pecan > > - flask-restplus > > - apistar > > > > ** HTTP Serving ** > > > > WSGI > > > > The WSGI approach is one we're all familiar with and it works with > > pretty much every existing Python REST framework. For us I believe if we > > go this route we'd want to serve it with something like uwsgi and > > Apache. That adds the need for an Apache layer and/or management uwsgi > > process. However, it means we can make use of normal tools we all likely > > know at least to some degree. > > FWIW I don't think Apache would be required. uWSGI is a fairly capable > http server aiui. You can also pip install uwsgi so the simple case > remains fairly simple I think. > > > > > A downside is that we'll need to continue to handle our Websockets work > > independently (which is what we're doing now) > > > > Because it's in a separate process, the API tier will need to make > > requests of the scheduler over a bus, which could be either gearman or > > zk. > > > > Note that OpenStack has decided that this is a better solution than > using web servers in the python process. That doesn't necessarily mean > it is the best choice for Zuul, but it seems like there is a lot we can > learn from the choice to switch to WSGI in OpenStack. > > > aiohttp > > > > Zuul v3 is Python3, which means we can use aiohttp. aiohttp isn't > > particularly compatible with the REST frameworks, but it has built-in > > route support and helpers for receiving and returning JSON. We don't > > need ORM mapping support, so the only thing we'd really be MISSING from > > REST frameworks is auto-generated documentation. > > > > aiohttp also supports websockets directly, so we could port the autobahn > > work to use aioh
Re: [OpenStack-Infra] On the subject of HTTP interfaces and Zuul
On Fri, Jun 9, 2017, at 09:22 AM, Monty Taylor wrote: > Hey all! > > Tristan has recently pushed up some patches related to providing a Web > Dashboard for Zuul. We have a web app for nodepool. We already have the > Github webhook receiver which is inbound http. There have been folks who > have expressed interest in adding active-REST abilities for performing > actions. AND we have the new websocket-based log streaming. > > We're currently using Paste for HTTP serving (which is effectively > dead), autobahn for websockets and WebOB for request/response processing. > > This means that before we get too far down the road, it's probably time > to pick how we're going to do those things in general. There are 2 > questions on the table: > > * HTTP serving > * REST framework > > They may or may not be related, and one of the options on the table > implies an answer for both. I'm going to start with the answer I think > we should pick: > > *** tl;dr *** > > We should use aiohttp with no extra REST framework. > > Meaning: > > - aiohttp serving REST and websocket streaming in a scale-out tier > - talking RPC to the scheduler over gear or zk > - possible in-process aiohttp endpoints for k8s style health endpoints > > Since we're talking about a web scale-out tier that we should just have > a single web tier for zuul and nodepool. This continues the thinking > that nodepool is a component of Zuul. I'm not sure that this is a great idea. We've already seen that people have wanted to use nodepool without a Zuul and even without performing CI. IIRC paul wanted to use it to keep a set of asterisks floating around for example. We've also seen that people want to use subcomponents of nodepool to build and manage a set of images for clouds without making instances. In the past we have been careful to keep logical tools separate which has made it easy for us to add new tools and remove old ones. Operationally this may be perceived as making things more difficult to a newcomer, but it makes life much much better 3-6 months down the road. > > In order to write zuul jobs, end-users must know what node labels are > available. A zuul client that says "please get me a list of available > node labels" could make sense to a user. As we get more non-OpenStack > users, those people may not have any concept that there is a separate > thing called "nodepool". > > *** The MUCH more verbose version *** > > I'm now going to outline all of the thoughts and options I've had or > have heard other people say. It's an extra complete list - there are > ideas in here you might find silly/bad. But since we're picking a > direction, I think it's important we consider the options in front of us. > > This will cover 3 http serving options: > > - WSGI > - aiohttp > - gRPC > > and 3 REST framework options: > > - pecan > - flask-restplus > - apistar > > ** HTTP Serving ** > > WSGI > > The WSGI approach is one we're all familiar with and it works with > pretty much every existing Python REST framework. For us I believe if we > go this route we'd want to serve it with something like uwsgi and > Apache. That adds the need for an Apache layer and/or management uwsgi > process. However, it means we can make use of normal tools we all likely > know at least to some degree. FWIW I don't think Apache would be required. uWSGI is a fairly capable http server aiui. You can also pip install uwsgi so the simple case remains fairly simple I think. > > A downside is that we'll need to continue to handle our Websockets work > independently (which is what we're doing now) > > Because it's in a separate process, the API tier will need to make > requests of the scheduler over a bus, which could be either gearman or > zk. > Note that OpenStack has decided that this is a better solution than using web servers in the python process. That doesn't necessarily mean it is the best choice for Zuul, but it seems like there is a lot we can learn from the choice to switch to WSGI in OpenStack. > aiohttp > > Zuul v3 is Python3, which means we can use aiohttp. aiohttp isn't > particularly compatible with the REST frameworks, but it has built-in > route support and helpers for receiving and returning JSON. We don't > need ORM mapping support, so the only thing we'd really be MISSING from > REST frameworks is auto-generated documentation. > > aiohttp also supports websockets directly, so we could port the autobahn > work to use aiohttp. > > aiohttp can be run in-process in a thread. However, websocket > log-streaming is already a separate process for scaling purposes, so if > we decide that one impl backend is a value, it probably makes sense to > just stick the web tier in the websocket scaleout process anyway. > > However, we could probably write a facade layer with a gear backend and > an in-memory backend so that simple users could just run the in-process > version but scale-out was possible for larger installs (li
Re: [OpenStack-Infra] Space for Sahara artifacts (disk images)
On 2017-05-30 12:49:37 +0200 (+0200), Luigi Toscano wrote: > On Friday, 26 May 2017 15:27:02 CEST Jeremy Stanley wrote: [...] > > We can see about extending the available free space for > > tarballs.openstack.org after we relocate it off the same server > > where we store our job logs, but we don't have a timeline for > > when that will be. I'm sorry I don't have better news on that > > front. > > It wouldn't be a problem to wait a bit. Even if you don't have a timeline, do > you know the approximate timing? Is it more 6 months, one year, or more? [...] Aside from it being something we've talked about probably wanting (putting tarballs content in AFS and being able to present a local cache to workers in each provider/region we use for our CI system), I don't think it's on anyone's radar yet from an execution perspective. We've had more recent discussions about relocating our logs.openstack.org site off that same server and into another service provider... doing that would also free up plenty of space for growing the current tarballs site and may well happen sooner (though again I have no solid timeline for that work, we need a spec for things like the 404->301 forwarding configuration on whichever end has the DNS record while waiting out the retention period). Rough guess is at least a few months given our current team throughput and priorities. > > If you need a periodic job to upload and replace pregenerated > > branch-tip snapshot images for consumption by other CI jobs, we > > should be able to work something out pretty easily (that's what > > the other projects I mentioned above have been doing). > > This is not the use case described in my original request. > Nevertheless, it could be useful for some of our scenario jobs. > But wouldn't this be constrained by the lack of space as well? Based on the numbers you gave, we could fairly confidently provide sufficient space for images built from the tips of supported branches since they would be replaced rather than accumulating new ones for each tag. Hosting image sets for every point release requires a fair amount more available space by comparison. -- Jeremy Stanley signature.asc Description: Digital signature ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] On the subject of HTTP interfaces and Zuul
Clint Byrum writes:
> Your words are more succinct than what I wrote, which is nice. I think
> we agree on the general direction for the time being.
>
> However, I don't think ZK will be a good choice for async event
> handling. I'd sooner expect MQTT to replace gear for that. It's worth
> noting that MQTT's protocol shares a lot in common with gearman and was
> created to do similar things.
You make some good points here, and in the other message, which have
both immediate and longer term aspects.
Your concern about using ZK for distributed ingestion is worth
considering as part of that discussion. We've shelved it for the moment
as potentially distracting for v3 work. But I think we can take your
point as being both that we should consider the potential issues there
when we discuss it, but also, for the moment, let's not presume in this
design that we're going to shove events into ZK in the future.
The other immediate aspect of this is that, if we are going to use this
framework for Github web hooks, we do need *some* answer of how to get
that info to the scheduler.
I'd say that since this is intended to be part of Zuul v3, and that we
have not taken any steps to reduce reliance on gearman in v3, that we
should go ahead and say that when the webhooks move to this framework,
they should submit their events to the scheduler via gearman. The
scheduler already accepts administrative events ("zuul enqueue") over
gearman; this is not a stretch.
I think that should clarify this aspect of this proposal for now, and
leaves us to consider the general question of distributed event
ingestion later.
-Jim
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] On the subject of HTTP interfaces and Zuul
On 06/09/2017 12:35 PM, Clint Byrum wrote: Excerpts from Monty Taylor's message of 2017-06-09 11:22:03 -0500: Hey all! Tristan has recently pushed up some patches related to providing a Web Dashboard for Zuul. We have a web app for nodepool. We already have the Github webhook receiver which is inbound http. There have been folks who have expressed interest in adding active-REST abilities for performing actions. AND we have the new websocket-based log streaming. We're currently using Paste for HTTP serving (which is effectively dead), autobahn for websockets and WebOB for request/response processing. This means that before we get too far down the road, it's probably time to pick how we're going to do those things in general. There are 2 questions on the table: * HTTP serving * REST framework They may or may not be related, and one of the options on the table implies an answer for both. I'm going to start with the answer I think we should pick: *** tl;dr *** We should use aiohttp with no extra REST framework. +1 for this. More inline.. Meaning: - aiohttp serving REST and websocket streaming in a scale-out tier - talking RPC to the scheduler over gear or zk - possible in-process aiohttp endpoints for k8s style health endpoints I think it's worth also discussing REST and gRPC for .. well.. RPC. ++ Since we're talking about a web scale-out tier that we should just have a single web tier for zuul and nodepool. This continues the thinking that nodepool is a component of Zuul. In order to write zuul jobs, end-users must know what node labels are available. A zuul client that says "please get me a list of available node labels" could make sense to a user. As we get more non-OpenStack users, those people may not have any concept that there is a separate thing called "nodepool". I have these two conflicting desires which is that I want Zuul to be useful as much on its own as possible, but I also don't want it to grow too rigid and be hard to adapt to new environments. I think infra has done a great job of loosely coupling nodepool and zuul without going ultra-pedantic about the interfaces, and that helps me satisfy both of these conflicting desires to some extent. I think for now, this status quo should continue and the thin line should still be respected. All of that said, if there was one web tier that had a /nodepool and a /zuul, I'd be fine with that. I'm can live with that, at least for now. However, I really see nodepool as an implementation detail of zuul, not a top-class citizen. We talk about "running zuul" and I imagine BonnyCI would expose the "Zuul" api to its users so that they could use Zuul API clients to talk to the api. So the /nodepool endpoint just feels - 'icky' to me. I may be totally wrong on that though. Maybe if there was no /zuul or /nodepol but instead there were things like: /nodes /status /jobs etc - and /nodes was the nodepool api served by nodepool potentially via an API proxy, and the others were all served by Zuul, that would be less icky feelling? ** Thoughts on RPC Bus ** gearman is a simple way to add RPC calls between an API tier and the scheduler. However, we got rid of gear from nodepool already, and we intend on getting rid of gearman in v4 anyway. The devil is in the details here IMO. We kind of abused gearman to do RPC in nodepool anyway. In particular, we were asking gearman about status to make scheduling decisions, which is actually a fairly expensive thing and not something gearman was ever designed to do. ZK is specifically designed to coordinate processes, which is exactly what we're using it for in this case. If we use zk, we'll have to do a little bit more thinking about how to do the RPC calls which will make this take more work. BUT - it means we can define one API that covers both Zuul and Nodepool and will be forward compatible with a v4 no-gearman world. I don't know if ZK is a great choice for the RPCs we know we have: * Github webhooks are async event streams, which will needlessly overtax a synchronous HA ZK cluster. Gearman is built to do this one well. Yah - I think github webhooks are their own beast in a few ways - but well suited to being handled in a few different ways. I don't think we should necessarily write zk off here - but I may be wrong and I'm ok with that. :) * status.json requests would be perfect if we had a ZK already under the scheduler (zuulv4). But that's a long way off. Yup. * log streaming is already basically working as a websocket HTTP endpoint that can be hit directly and is stateless and thus allows simple loadbalancing, right?. I don't feel compelled strongly to align other efforts with it since it is somewhat unique in its scope. Yah - I don't think it has to be hard-aligned as much as if we start using aiohttp for other things, it seems like maybe not a great thing to keep around the autobahn depend just for that one thing. We *could* use
Re: [OpenStack-Infra] On the subject of HTTP interfaces and Zuul
Excerpts from corvus's message of 2017-06-09 09:41:37 -0700: > Monty Taylor writes: > > > We should use aiohttp with no extra REST framework. > > > > Meaning: > > > > - aiohttp serving REST and websocket streaming in a scale-out tier > > - talking RPC to the scheduler over gear or zk > > - possible in-process aiohttp endpoints for k8s style health endpoints > > ... > > > Since we're starting fresh, I like the idea of a single API service > > that RPCs to zuul and nodepool, so I like the idea of using ZK for the > > RPC layer. BUT - using gear and adding just gear worker threads back > > to nodepol wouldn't be super-terrible maybe. > > Thanks for the thoughtful analysis. I think your argument is compelling > and I generally like the approach you suggest. > > On the RPC front, how about we accept that, for the moment, the > webserver will need to consult ZK for collecting some information > (current nodepool label/image status), and use gear for other things > (querying zuul about build status)? > > The rest of Zuul already uses both things, let's just have the webserver > do the same. Eventually gear functions will be replaced with ZK. > Your words are more succinct than what I wrote, which is nice. I think we agree on the general direction for the time being. However, I don't think ZK will be a good choice for async event handling. I'd sooner expect MQTT to replace gear for that. It's worth noting that MQTT's protocol shares a lot in common with gearman and was created to do similar things. ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] On the subject of HTTP interfaces and Zuul
Excerpts from Monty Taylor's message of 2017-06-09 11:22:03 -0500: > Hey all! > > Tristan has recently pushed up some patches related to providing a Web > Dashboard for Zuul. We have a web app for nodepool. We already have the > Github webhook receiver which is inbound http. There have been folks who > have expressed interest in adding active-REST abilities for performing > actions. AND we have the new websocket-based log streaming. > > We're currently using Paste for HTTP serving (which is effectively > dead), autobahn for websockets and WebOB for request/response processing. > > This means that before we get too far down the road, it's probably time > to pick how we're going to do those things in general. There are 2 > questions on the table: > > * HTTP serving > * REST framework > > They may or may not be related, and one of the options on the table > implies an answer for both. I'm going to start with the answer I think > we should pick: > > *** tl;dr *** > > We should use aiohttp with no extra REST framework. > +1 for this. More inline.. > Meaning: > > - aiohttp serving REST and websocket streaming in a scale-out tier > - talking RPC to the scheduler over gear or zk > - possible in-process aiohttp endpoints for k8s style health endpoints > I think it's worth also discussing REST and gRPC for .. well.. RPC. > Since we're talking about a web scale-out tier that we should just have > a single web tier for zuul and nodepool. This continues the thinking > that nodepool is a component of Zuul. > > In order to write zuul jobs, end-users must know what node labels are > available. A zuul client that says "please get me a list of available > node labels" could make sense to a user. As we get more non-OpenStack > users, those people may not have any concept that there is a separate > thing called "nodepool". > I have these two conflicting desires which is that I want Zuul to be useful as much on its own as possible, but I also don't want it to grow too rigid and be hard to adapt to new environments. I think infra has done a great job of loosely coupling nodepool and zuul without going ultra-pedantic about the interfaces, and that helps me satisfy both of these conflicting desires to some extent. I think for now, this status quo should continue and the thin line should still be respected. All of that said, if there was one web tier that had a /nodepool and a /zuul, I'd be fine with that. > ** Thoughts on RPC Bus ** > > gearman is a simple way to add RPC calls between an API tier and the > scheduler. However, we got rid of gear from nodepool already, and we > intend on getting rid of gearman in v4 anyway. > The devil is in the details here IMO. We kind of abused gearman to do RPC in nodepool anyway. In particular, we were asking gearman about status to make scheduling decisions, which is actually a fairly expensive thing and not something gearman was ever designed to do. ZK is specifically designed to coordinate processes, which is exactly what we're using it for in this case. > If we use zk, we'll have to do a little bit more thinking about how to > do the RPC calls which will make this take more work. BUT - it means we > can define one API that covers both Zuul and Nodepool and will be > forward compatible with a v4 no-gearman world. > I don't know if ZK is a great choice for the RPCs we know we have: * Github webhooks are async event streams, which will needlessly overtax a synchronous HA ZK cluster. Gearman is built to do this one well. * status.json requests would be perfect if we had a ZK already under the scheduler (zuulv4). But that's a long way off. * log streaming is already basically working as a websocket HTTP endpoint that can be hit directly and is stateless and thus allows simple loadbalancing, right?. I don't feel compelled strongly to align other efforts with it since it is somewhat unique in its scope. > We *could* use gearman in zuul and run an API in-process in nodepool. > Then we could take a page out of early Nova and do a proxy-layer in zuul > that makes requests of nodepool's API. > > We could just assume that there's gonna be an Apache fronting this stuff > and suggest deployment with routing to zuul and nodepool apis with > mod_proxy rules. > > Finally, as clarkb pointed out in response to the ingestors spec, we > could introduce MQTT and use it. I'm wary of doing that for this because > it introduces a totally required new tech stack at a late stage. > > Since we're starting fresh, I like the idea of a single API service that > RPCs to zuul and nodepool, so I like the idea of using ZK for the RPC > layer. BUT - using gear and adding just gear worker threads back to > nodepol wouldn't be super-terrible maybe. > I'm inclined to ask why we don't just have a routing HTTP proxy that sends stuff directly to the backends of nodepool, log streaming, or zuul-scheduler. That proxy and the backends can all use aiohttp to ser
Re: [OpenStack-Infra] On the subject of HTTP interfaces and Zuul
On 06/09/2017 11:41 AM, James E. Blair wrote: Monty Taylor writes: We should use aiohttp with no extra REST framework. Meaning: - aiohttp serving REST and websocket streaming in a scale-out tier - talking RPC to the scheduler over gear or zk - possible in-process aiohttp endpoints for k8s style health endpoints ... Since we're starting fresh, I like the idea of a single API service that RPCs to zuul and nodepool, so I like the idea of using ZK for the RPC layer. BUT - using gear and adding just gear worker threads back to nodepol wouldn't be super-terrible maybe. Thanks for the thoughtful analysis. I think your argument is compelling and I generally like the approach you suggest. On the RPC front, how about we accept that, for the moment, the webserver will need to consult ZK for collecting some information (current nodepool label/image status), and use gear for other things (querying zuul about build status)? The rest of Zuul already uses both things, let's just have the webserver do the same. Eventually gear functions will be replaced with ZK. I think that's a great idea! ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] On the subject of HTTP interfaces and Zuul
Monty Taylor writes: > We should use aiohttp with no extra REST framework. > > Meaning: > > - aiohttp serving REST and websocket streaming in a scale-out tier > - talking RPC to the scheduler over gear or zk > - possible in-process aiohttp endpoints for k8s style health endpoints ... > Since we're starting fresh, I like the idea of a single API service > that RPCs to zuul and nodepool, so I like the idea of using ZK for the > RPC layer. BUT - using gear and adding just gear worker threads back > to nodepol wouldn't be super-terrible maybe. Thanks for the thoughtful analysis. I think your argument is compelling and I generally like the approach you suggest. On the RPC front, how about we accept that, for the moment, the webserver will need to consult ZK for collecting some information (current nodepool label/image status), and use gear for other things (querying zuul about build status)? The rest of Zuul already uses both things, let's just have the webserver do the same. Eventually gear functions will be replaced with ZK. -Jim ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] On the subject of HTTP interfaces and Zuul
Hey all!
Tristan has recently pushed up some patches related to providing a Web
Dashboard for Zuul. We have a web app for nodepool. We already have the
Github webhook receiver which is inbound http. There have been folks who
have expressed interest in adding active-REST abilities for performing
actions. AND we have the new websocket-based log streaming.
We're currently using Paste for HTTP serving (which is effectively
dead), autobahn for websockets and WebOB for request/response processing.
This means that before we get too far down the road, it's probably time
to pick how we're going to do those things in general. There are 2
questions on the table:
* HTTP serving
* REST framework
They may or may not be related, and one of the options on the table
implies an answer for both. I'm going to start with the answer I think
we should pick:
*** tl;dr ***
We should use aiohttp with no extra REST framework.
Meaning:
- aiohttp serving REST and websocket streaming in a scale-out tier
- talking RPC to the scheduler over gear or zk
- possible in-process aiohttp endpoints for k8s style health endpoints
Since we're talking about a web scale-out tier that we should just have
a single web tier for zuul and nodepool. This continues the thinking
that nodepool is a component of Zuul.
In order to write zuul jobs, end-users must know what node labels are
available. A zuul client that says "please get me a list of available
node labels" could make sense to a user. As we get more non-OpenStack
users, those people may not have any concept that there is a separate
thing called "nodepool".
*** The MUCH more verbose version ***
I'm now going to outline all of the thoughts and options I've had or
have heard other people say. It's an extra complete list - there are
ideas in here you might find silly/bad. But since we're picking a
direction, I think it's important we consider the options in front of us.
This will cover 3 http serving options:
- WSGI
- aiohttp
- gRPC
and 3 REST framework options:
- pecan
- flask-restplus
- apistar
** HTTP Serving **
WSGI
The WSGI approach is one we're all familiar with and it works with
pretty much every existing Python REST framework. For us I believe if we
go this route we'd want to serve it with something like uwsgi and
Apache. That adds the need for an Apache layer and/or management uwsgi
process. However, it means we can make use of normal tools we all likely
know at least to some degree.
A downside is that we'll need to continue to handle our Websockets work
independently (which is what we're doing now)
Because it's in a separate process, the API tier will need to make
requests of the scheduler over a bus, which could be either gearman or zk.
aiohttp
Zuul v3 is Python3, which means we can use aiohttp. aiohttp isn't
particularly compatible with the REST frameworks, but it has built-in
route support and helpers for receiving and returning JSON. We don't
need ORM mapping support, so the only thing we'd really be MISSING from
REST frameworks is auto-generated documentation.
aiohttp also supports websockets directly, so we could port the autobahn
work to use aiohttp.
aiohttp can be run in-process in a thread. However, websocket
log-streaming is already a separate process for scaling purposes, so if
we decide that one impl backend is a value, it probably makes sense to
just stick the web tier in the websocket scaleout process anyway.
However, we could probably write a facade layer with a gear backend and
an in-memory backend so that simple users could just run the in-process
version but scale-out was possible for larger installs (like us)
Since aiohttp can be in-process, it also allows us to easily add some
'/health' endpoints to all of our services directly, even if they aren't
intended to be publicly consumable. That's important for running richly
inside of things like kubernetes that like to check in on health status
of services to know about rescheduling them. This way we could add a
simple thread to the scheduler and the executors and the mergers and the
nodepool launchers and builders that adds a '/health' endpoint.
gRPC / gRPC-REST gateway
This is a curve-ball. We could define our APIs using gRPC. That gets us
a story for an API that is easily consumable by all sorts of clients,
and that supports exciting things like bi-directional streaming
channels. gRPC isn't (yet) consumable directly in browsers, nor does
Github send gRPC webhooks. BUT - there is a REST Gateway for gRPC:
https://github.com/grpc-ecosystem/grpc-gateway
that generates HTTP/1.1+JSON interfaces from the gRPC descriptions and
translates between protobuf and json automatically. The "REST" interface
it produces does not support url-based parameters, so everything is done
in payload bodies, so it's:
GET /nodes
{
'id': 1234
}
rather than
GET /nodes/1234
but that's still totally fine - and totally works for both status.json
Re: [OpenStack-Infra] Disc space usage translate.openstack.org
Am 2017-06-09 15:44, schrieb Clark Boylan: On Thu, Jun 8, 2017, at 11:44 PM, Frank Kloeker wrote: Dear infra team, it seems there are still issues on translate.openstack.org machine. The disc is nearly full [1]. In the past it was caused by missing logrotate rules after 16.04 migration. Maybe it needs some adjustments. thx Frank [1] http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=2748&rra_id=3&view_type=&graph_start=1494311049&graph_end=1496989449 I've gone ahead and deleted all wildfly logs that are more than two week old. I've also pushed a change [2] to automate this process so should no longer be a problem once that is merged. Are you noticing functionality problems with the Zanata service on translate.openstack.org? It seems happy if I use it. Please let us know if there are functional problems. [2] https://review.openstack.org/472683 Hi Clark, don't worry. I've just looked at cacti stats in the morning because https://review.openstack.org was down and saw the increasing disc space usage on translate.openstack.org. Functional problems was not discovered. Thanks for fixing up the issue. kind regards Frank ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Disc space usage translate.openstack.org
On Thu, Jun 8, 2017, at 11:44 PM, Frank Kloeker wrote: > Dear infra team, > > it seems there are still issues on translate.openstack.org machine. The > disc is nearly full [1]. In the past it was caused by missing logrotate > rules after 16.04 migration. Maybe it needs some adjustments. > > thx > > Frank > > [1] > http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=2748&rra_id=3&view_type=&graph_start=1494311049&graph_end=1496989449 I've gone ahead and deleted all wildfly logs that are more than two week old. I've also pushed a change [2] to automate this process so should no longer be a problem once that is merged. Are you noticing functionality problems with the Zanata service on translate.openstack.org? It seems happy if I use it. Please let us know if there are functional problems. [2] https://review.openstack.org/472683 Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
