On 8/15/19 11:03 AM, Ian Jackson wrote: > (off-list) > > Ian. You replied off-list, so I wont quote you, though I would like my answer to be public.
I probably should have mentioned that my remark was not so much related to the crash, but more to what I experienced using Salsa's CI. It's currently super slow, as apparently, it's using GCE using the smallest instance possible (maybe because we have a limited credit?). As a result, adding the salsa CI team's script just fails in some packages like for example OpenVSwitch, because of the slowness. In the extreme case of OpenVSwitch, during build, mostly all tests are timing-out. I found the overall experience frustrating, and the Salsa CI not very helpful because too slow. So, instead of what I experienced, I thought using Zuul + Nodepool like in the OpenStack CI would probably solve the slowness and resource starvation issues. What's done there, is that OpenStack based cloud providers are sponsoring a pool of VMs (. They are maintained using nodepool, that "pre-provision" the VMs. That's a bit like with apache prefork, where processes are spawned in advance to be ready for the next connections, except here, we're talking about VMs ready to accept CI jobs. Then Zuul does the job queue and scheduling. Overall, this is a very efficient CI, which is able to spawn thousands of jobs, on many different cloud providers. I'm not sure exactly what the Salsa issue was. Whatever it was, I strongly believe that Zuul + Nodepool would help: 1/ CI Jobs going faster 2/ Have a way more workers 3/ Don't have all our eggs on the same Google basket 4/ Use free software platforms instead of GCE 5/ Stop being bound to a single VM provider [1] Not only OpenStack uses Zuul, big companies like Netflix and MediaWiki foundations too. I've read that this year, there was some efforts to add Gitlab support to Zuul. I don't know what the status is though, but I know that it's possible to add Gerrit in front, and then have Gerrit plugged to Zuul. I've used Gerrit previously to do OpenStack packaging (the version of OpenStack in Stretch was done this way), and I was very happy of it. I'd love if I could have the opportunity to do this again, but with Salsa and the Debian infra this time. Cheers, Thomas Goirand (zigo) P.S: If you want an idea of what it is like with nodepool in action, go here: http://grafana.openstack.org then click on "Home" and select nodepool. There you'll see graphs of spawned instances. [1] The OpenStack CI is currently configured to use VMs from: Fortnebula, OVH (2 regions), Rackspace (2 regions), and Vexxhost, though it used to have a large amount of sponsors.
