Re: [OpenStack-Infra] A tool for slurping gerrit changes in to bug updates
On 26/05/16 14:06, Jeremy Stanley wrote: On 2016-05-26 11:37:18 +0100 (+0100), Zara Zaimeche wrote: [...] Updating the bug/task-tracker from those changes, however, would be different for each tracker, so you wouldn't be able to do that part in a generic way. Plus, while analyzing the data from the stream could theoretically be the same, in practice different data might matter for each tracker, and one might be looking for a different regex in the commit message, etc. So I think the only part that can be generic will be getting the changes from gerrit, and from other comments on this thread, it sounds as though people have already covered that. I'd imagine things like regular expression matches and API methods could be extracted into configuration (as long as we make assumptions like the task tracker APIs are all RESTish and don't require fancy branching business logic). So a rules engine which can be configured to tokenize parts of specified Gerrit events and then expand corresponding variables in configured API calls would presumably fit the bill. I agree regexps should definitely be extractable. I'm less sure about API methods; some general things I've noticed in discussion of task trackers are: * The APIs seem to often be a muddy spot (I hear complaints about how they're not as RESTful as desired) * Trackers designed to work on a small scale have a very different architecture from those designed to span multiple repos So the first point there makes me wonder how safe we are in assuming tracker APIs are all RESTish. The second makes me wonder how far a generic implementation can go, or whether it'd be better to have different implementations for cross-project bug/task-trackers vs single project bug/task-trackers. (eg: the contrast between StoryBoard and Github issues) The latter tend not to, say, distinguish between stories and tasks, which makes sense when the tracker doesn't have to worry about coordinating cross-project work. But if someone writes middleware that assumes stories and tasks are the same thing in a tracker, it requires some fiddling to make it work for StoryBoard, possibly to the point where it's easier to implement a custom thing from scratch. From my perspective, there's a good chance that something designed initially for Github Enterprise won't have much I can use-- which is still fine; we were just asked if we'd find it useful. :) StoryBoard is the unusual one here, and I think a lot of teams with smaller trackers will find their architecture similar enough for it to work for them. And maybe it's possible not to make those assumptions, but I figure if I flag it up, it's easier for people to know what assumptions they're making, and choose an approach from the start. Best Wishes, Zara ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] A tool for slurping gerrit changes in to bug updates
On 2016-05-26 14:23:38 -0400 (-0400), Sean Dague wrote: [...] > It does run on a custom port... so the great firewall of China is still > probably an issue. [...] I had assumed websockets could be a solution there? http://jpmens.net/2014/07/03/the-mosquitto-mqtt-broker-gets-websockets-support/ > In my ideal world of awesomeness, there would be an MQTT server in infra > which was getting data from all the relevant change sources [...] This makes a good case for it running on a separate server then rather than directly on the Gerrit server. -- Jeremy Stanley ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] A tool for slurping gerrit changes in to bug updates
On 05/26/2016 02:11 PM, Matthew Treinish wrote: > On Thu, May 26, 2016 at 05:55:34PM +, Jeremy Stanley wrote: >> On 2016-05-26 12:54:49 -0400 (-0400), Matthew Treinish wrote: >>> Just a quick follow-up I started running this on a throwaway server at >>> 15.184.138.236. So you can subscribe to events from gerrit. >> [...] >> >> How resource-intensive is it? Curious whether it makes sense to run >> something like this directly on review.openstack.org. If zuul grew >> support for that mechanism, it might allow CI systems (third party >> or even our own) to wean off using SSH entirely since this is a >> problem in a lot of places from crazy enterprise firewall policies >> to systems running in mainland China. It does run on a custom port... so the great firewall of China is still probably an issue. > It's eating like nothing on my server right now. This is all running on a > single > cpu vm on a private cloud with a "Intel Xeon E312xx (Sandy Bridge)" (according > to /proc/cpuinfo) Mosquitto itself has a memory footprint < 1MB and I've seen > it > spike up to a whopping 1% cpu utilization. Although, this might increase a as > more > subscribers are added. This is the first time I've played with mosquitto and > mqtt > so I don't know what it's scaling is like. But, I imagine it should handle a > lot > of subscriptions well since it's supposed to be an IoT thing. germqtt is > eating > a bit more with consuming about 1.5M of RAM and it uses about the same CPU as > Mosquitto. In my ideal world of awesomeness, there would be an MQTT server in infra which was getting data from all the relevant change sources (gerrit changes, launchpad changes (which, yes requires something crazy like converting an email stream into events)), zuul enqueue, dequeue events). And then every time someone wanted to build some ad hoc web tool to take a slice of this they could consume the event stream for updates, instead of doing what everyone does and polls once an hour. As long as the topic trees are well structured, it should make for an easy way to get the slice they needed and only react to that. -Sean -- Sean Dague http://dague.net ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] A tool for slurping gerrit changes in to bug updates
On Thu, May 26, 2016 at 05:55:34PM +, Jeremy Stanley wrote: > On 2016-05-26 12:54:49 -0400 (-0400), Matthew Treinish wrote: > > Just a quick follow-up I started running this on a throwaway server at > > 15.184.138.236. So you can subscribe to events from gerrit. > [...] > > How resource-intensive is it? Curious whether it makes sense to run > something like this directly on review.openstack.org. If zuul grew > support for that mechanism, it might allow CI systems (third party > or even our own) to wean off using SSH entirely since this is a > problem in a lot of places from crazy enterprise firewall policies > to systems running in mainland China. It's eating like nothing on my server right now. This is all running on a single cpu vm on a private cloud with a "Intel Xeon E312xx (Sandy Bridge)" (according to /proc/cpuinfo) Mosquitto itself has a memory footprint < 1MB and I've seen it spike up to a whopping 1% cpu utilization. Although, this might increase a as more subscribers are added. This is the first time I've played with mosquitto and mqtt so I don't know what it's scaling is like. But, I imagine it should handle a lot of subscriptions well since it's supposed to be an IoT thing. germqtt is eating a bit more with consuming about 1.5M of RAM and it uses about the same CPU as Mosquitto. -Matt Treinish signature.asc Description: PGP signature ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] A tool for slurping gerrit changes in to bug updates
On 2016-05-26 12:54:49 -0400 (-0400), Matthew Treinish wrote: > Just a quick follow-up I started running this on a throwaway server at > 15.184.138.236. So you can subscribe to events from gerrit. [...] How resource-intensive is it? Curious whether it makes sense to run something like this directly on review.openstack.org. If zuul grew support for that mechanism, it might allow CI systems (third party or even our own) to wean off using SSH entirely since this is a problem in a lot of places from crazy enterprise firewall policies to systems running in mainland China. -- Jeremy Stanley ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] A tool for slurping gerrit changes in to bug updates
On 2016-05-26 09:57:58 -0700 (-0700), Zaro wrote: > Parts of jeepyb for gerrit and issue tracking functionality should > really be deprecated. Gerrit allows issue tracking integration with > its set of its-* based plugins. There's one for bugzilla, jira, and > even storyboard. [...] That's a very good point with respect to storyboard integration. I had completely forgotten you'd written that plugin a couple years ago. What are the up/down sides between that and a pull model for the review.openstack.org+storyboard.openstack.org use case? -- Jeremy Stanley ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] A tool for slurping gerrit changes in to bug updates
Parts of jeepyb for gerrit and issue tracking functionality should really be deprecated. Gerrit allows issue tracking integration with its set of its-* based plugins. There's one for bugzilla, jira, and even storyboard. However those probably won't help since it does need to connect directly to the issue tracker. I'm guessing what might help is the rabbitmq plugin. I'm not sure if it would be too heavyweight for your needs but you might want to take a look, https://gerrit.googlesource.com/plugins/rabbitmq/ -Khai On Wed, May 25, 2016 at 9:43 AM, Gregory Hayneswrote: > Hello -Infra folks, > > While setting up an OpenStack-infra style testing infrastructure we have > run in to the need for a tool to update our issue tracker in a different > manner than the current Gerrit -> jeepyb system used for OpenStack. Our > issue boils down to the fact that our bug tracker lives on a network our > Gerrit cannot initiate a connection in to. As a result we need something > to connect to Gerrit from within our bug tracker's network. We are > considering making a small project to connect to and read from the > Gerrit event stream and then update our bug tracker. > > > My hope with this email was to see if: > > Is there something (aside from not having crazy network requirements) > were missing that might make this project unnecessary? > > If we implemented this, would this be something the -infra project would > like to have live upstream? It seems easy enough to make this generally > useful to others with similar requirements. > > Any other thoughts/comments that might help :). > > Thanks, > Greg > > -- > Gregory Haynes > g...@greghaynes.net > > ___ > OpenStack-Infra mailing list > OpenStack-Infra@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] A tool for slurping gerrit changes in to bug updates
On Wed, May 25, 2016 at 10:08:57PM -0400, Matthew Treinish wrote: > On Wed, May 25, 2016 at 02:43:36PM -0400, Sean Dague wrote: > > On 05/25/2016 01:54 PM, Spencer Krum wrote: > > > > > > When working on a previous project, Nikki wrote this tool as a general > > > purpose hook into gerrit: > > > > > > https://github.com/notnownikki/zoidberg > > > > > > I don't think she is actively maintaining or using it right now though. > > > > One thing I've been thinking a bit about is whether the event stream > > could get into something like MQTT easily. In completely unrelated > > activities (https://home-assistant.io/) I've been playing with mosquitto > > (http://mosquitto.org/) quite a bit, and the ease of consumption of mqtt > > is quite nice. (You can even do it straight in javascript for web based > > things). > > It looked like this would be a fun little side project to experiment with, so > I > took a bit of time and wrote: > > https://github.com/mtreinish/germqtt > > It's still kinda rough and basic. For right now all it does is publishes a > json > blob of every event from the gerrit stream over mqtt. But, if it's something > we > wanted to continue with it's a good starting point. > Just a quick follow-up I started running this on a throwaway server at 15.184.138.236. So you can subscribe to events from gerrit. Using the mosquitto_sub command [1] is a simple way to play with it. For example: $ mosquitto_sub --topic 'gerrit/#' -h 15.184.138.236 will subscribe to all events $ mosquitto_sub --topic 'gerrit/openstack/nova/#' -h 15.184.138.236 will subscribe to all nova events $ mosquitto_sub --topic 'gerrit/+/+/comment-added' -h 15.184.138.236 will subscribe to all comment-added events. You can come up with any whatever subscriptions you want using the builtin wildcards on the protocol. [2] I'll probably keep the server running for a while, so feel free to play around with it. -Matt Treinish [1] http://mosquitto.org/man/mosquitto_sub-1.html [2] http://mosquitto.org/man/mqtt-7.html signature.asc Description: PGP signature ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] A tool for slurping gerrit changes in to bug updates
On 2016-05-26 11:37:18 +0100 (+0100), Zara Zaimeche wrote: [...] > Updating the bug/task-tracker from those changes, however, would > be different for each tracker, so you wouldn't be able to do that > part in a generic way. Plus, while analyzing the data from the > stream could theoretically be the same, in practice different data > might matter for each tracker, and one might be looking for a > different regex in the commit message, etc. So I think the only > part that can be generic will be getting the changes from gerrit, > and from other comments on this thread, it sounds as though people > have already covered that. I'd imagine things like regular expression matches and API methods could be extracted into configuration (as long as we make assumptions like the task tracker APIs are all RESTish and don't require fancy branching business logic). So a rules engine which can be configured to tokenize parts of specified Gerrit events and then expand corresponding variables in configured API calls would presumably fit the bill. -- Jeremy Stanley ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] A tool for slurping gerrit changes in to bug updates
On 25/05/16 18:44, Anita Kuno wrote: On 05/25/2016 01:13 PM, Jeremy Stanley wrote: On 2016-05-25 11:43:39 -0500 (-0500), Gregory Haynes wrote: [...] We are considering making a small project to connect to and read from the Gerrit event stream and then update our bug tracker. [...] If we implemented this, would this be something the -infra project would like to have live upstream? It seems easy enough to make this generally useful to others with similar requirements. Any other thoughts/comments that might help :). Please collaborate with the storyboard team on this. I believe the Gerrit integration plan there has always been to have an independent service which consumes the Gerrit event stream and then performs arbitrary callouts (likely over localhost on the storyboard server) to a task tracker API. Your goals seem closely aligned. I believe this is the patch that is in progress: https://review.openstack.org/#/c/302912/ I think Zara would welcome help on it if you want to talk to her in #storyboard. Thanks, Anita. ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra Hi, I'd add to this that I found getting gerrit changes pretty trivial using gerritlib (well, my control flow in that WIP is dodgy, but that's more a combination of my inexperience and my lack of time). Updating the bug/task-tracker from those changes, however, would be different for each tracker, so you wouldn't be able to do that part in a generic way. Plus, while analyzing the data from the stream could theoretically be the same, in practice different data might matter for each tracker, and one might be looking for a different regex in the commit message, etc. So I think the only part that can be generic will be getting the changes from gerrit, and from other comments on this thread, it sounds as though people have already covered that. Best Wishes, Zara ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] A tool for slurping gerrit changes in to bug updates
On Wed, May 25, 2016 at 02:43:36PM -0400, Sean Dague wrote: > On 05/25/2016 01:54 PM, Spencer Krum wrote: > > > > When working on a previous project, Nikki wrote this tool as a general > > purpose hook into gerrit: > > > > https://github.com/notnownikki/zoidberg > > > > I don't think she is actively maintaining or using it right now though. > > One thing I've been thinking a bit about is whether the event stream > could get into something like MQTT easily. In completely unrelated > activities (https://home-assistant.io/) I've been playing with mosquitto > (http://mosquitto.org/) quite a bit, and the ease of consumption of mqtt > is quite nice. (You can even do it straight in javascript for web based > things). It looked like this would be a fun little side project to experiment with, so I took a bit of time and wrote: https://github.com/mtreinish/germqtt It's still kinda rough and basic. For right now all it does is publishes a json blob of every event from the gerrit stream over mqtt. But, if it's something we wanted to continue with it's a good starting point. -Matt Treinish signature.asc Description: PGP signature ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] A tool for slurping gerrit changes in to bug updates
On 05/26/2016 04:43 AM, Sean Dague wrote: One thing I've been thinking a bit about is whether the event stream could get into something like MQTT easily. Although larger in scope than just gerrit, Fedora has something very similar to this with fedmsg [1] It is a pretty cool idea to have everything that's happening exposed in a common place with a documented format. -i [1] http://www.fedmsg.com/en/latest/ ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] A tool for slurping gerrit changes in to bug updates
On 05/25/2016 01:54 PM, Spencer Krum wrote: > > When working on a previous project, Nikki wrote this tool as a general > purpose hook into gerrit: > > https://github.com/notnownikki/zoidberg > > I don't think she is actively maintaining or using it right now though. One thing I've been thinking a bit about is whether the event stream could get into something like MQTT easily. In completely unrelated activities (https://home-assistant.io/) I've been playing with mosquitto (http://mosquitto.org/) quite a bit, and the ease of consumption of mqtt is quite nice. (You can even do it straight in javascript for web based things). -Sean -- Sean Dague http://dague.net ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] A tool for slurping gerrit changes in to bug updates
When working on a previous project, Nikki wrote this tool as a general purpose hook into gerrit: https://github.com/notnownikki/zoidberg I don't think she is actively maintaining or using it right now though. -- Spencer Krum n...@spencerkrum.com ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] A tool for slurping gerrit changes in to bug updates
On 05/25/2016 01:13 PM, Jeremy Stanley wrote: > On 2016-05-25 11:43:39 -0500 (-0500), Gregory Haynes wrote: > [...] >> We are considering making a small project to connect to and read >> from the Gerrit event stream and then update our bug tracker. > [...] >> If we implemented this, would this be something the -infra project >> would like to have live upstream? It seems easy enough to make >> this generally useful to others with similar requirements. >> >> Any other thoughts/comments that might help :). > > Please collaborate with the storyboard team on this. I believe the > Gerrit integration plan there has always been to have an independent > service which consumes the Gerrit event stream and then performs > arbitrary callouts (likely over localhost on the storyboard server) > to a task tracker API. Your goals seem closely aligned. > I believe this is the patch that is in progress: https://review.openstack.org/#/c/302912/ I think Zara would welcome help on it if you want to talk to her in #storyboard. Thanks, Anita. ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] A tool for slurping gerrit changes in to bug updates
On 2016-05-25 11:43:39 -0500 (-0500), Gregory Haynes wrote: [...] > We are considering making a small project to connect to and read > from the Gerrit event stream and then update our bug tracker. [...] > If we implemented this, would this be something the -infra project > would like to have live upstream? It seems easy enough to make > this generally useful to others with similar requirements. > > Any other thoughts/comments that might help :). Please collaborate with the storyboard team on this. I believe the Gerrit integration plan there has always been to have an independent service which consumes the Gerrit event stream and then performs arbitrary callouts (likely over localhost on the storyboard server) to a task tracker API. Your goals seem closely aligned. -- Jeremy Stanley ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] A tool for slurping gerrit changes in to bug updates
On Wed, May 25, 2016, at 11:55 AM, Jimmy Mcarthur wrote: > I'm not sure about Infra, but we're in the same boat with one of our bug > trackers. This would be awesome to have and I'm sure we would use it. > > Out of curiosity, what bug tracker are you currently using? GitHub Enterprise, although I think it would make sense (if we were to make this) to make the bug tracker a pluggable backend. I would expect this to be pretty simple to do and would be a requirement to live upstream since GHE certainly isn't a supported thing there. > > Cheers, > Jimmy > > Gregory Haynes wrote: > > Hello -Infra folks, > > > > While setting up an OpenStack-infra style testing infrastructure we have > > run in to the need for a tool to update our issue tracker in a different > > manner than the current Gerrit -> jeepyb system used for OpenStack. Our > > issue boils down to the fact that our bug tracker lives on a network our > > Gerrit cannot initiate a connection in to. As a result we need something > > to connect to Gerrit from within our bug tracker's network. We are > > considering making a small project to connect to and read from the > > Gerrit event stream and then update our bug tracker. > > > > > > My hope with this email was to see if: > > > > Is there something (aside from not having crazy network requirements) > > were missing that might make this project unnecessary? > > > > If we implemented this, would this be something the -infra project would > > like to have live upstream? It seems easy enough to make this generally > > useful to others with similar requirements. > > > > Any other thoughts/comments that might help :). > > > > Thanks, > > Greg > > > ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] A tool for slurping gerrit changes in to bug updates
I'm not sure about Infra, but we're in the same boat with one of our bug trackers. This would be awesome to have and I'm sure we would use it. Out of curiosity, what bug tracker are you currently using? Cheers, Jimmy Gregory Haynes wrote: Hello -Infra folks, While setting up an OpenStack-infra style testing infrastructure we have run in to the need for a tool to update our issue tracker in a different manner than the current Gerrit -> jeepyb system used for OpenStack. Our issue boils down to the fact that our bug tracker lives on a network our Gerrit cannot initiate a connection in to. As a result we need something to connect to Gerrit from within our bug tracker's network. We are considering making a small project to connect to and read from the Gerrit event stream and then update our bug tracker. My hope with this email was to see if: Is there something (aside from not having crazy network requirements) were missing that might make this project unnecessary? If we implemented this, would this be something the -infra project would like to have live upstream? It seems easy enough to make this generally useful to others with similar requirements. Any other thoughts/comments that might help :). Thanks, Greg ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra