Re: [PATCH 0/5] QEMU Gating CI
On Thu, Apr 23, 2020 at 01:04:13PM -0400, Cleber Rosa wrote: > > > - Original Message - > > From: "Peter Maydell" > > To: "Markus Armbruster" > > Cc: "Fam Zheng" , "Thomas Huth" , > > "Beraldo Leal" , "Erik > > Skultety" , "Alex Bennée" , > > "Wainer Moschetta" , > > "QEMU Developers" , "Wainer dos Santos Moschetta" > > , "Willian Rampazzo" > > , "Cleber Rosa" , "Philippe > > Mathieu-Daudé" , "Eduardo > > Habkost" > > Sent: Tuesday, April 21, 2020 8:53:49 AM > > Subject: Re: [PATCH 0/5] QEMU Gating CI > > > > On Thu, 19 Mar 2020 at 16:33, Markus Armbruster wrote: > > > Peter Maydell writes: > > > > I think we should start by getting the gitlab setup working > > > > for the basic "x86 configs" first. Then we can try adding > > > > a runner for s390 (that one's logistically easiest because > > > > it is a project machine, not one owned by me personally or > > > > by Linaro) once the basic framework is working, and expand > > > > from there. > > > > > > Makes sense to me. > > > > > > Next steps to get this off the ground: > > > > > > * Red Hat provides runner(s) for x86 stuff we care about. > > > > > > * If that doesn't cover 'basic "x86 configs" in your judgement, we > > > fill the gaps as described below under "Expand from there". > > > > > > * Add an s390 runner using the project machine you mentioned. > > > > > > * Expand from there: identify the remaining gaps, map them to people / > > > organizations interested in them, and solicit contributions from these > > > guys. > > > > > > A note on contributions: we need both hardware and people. By people I > > > mean maintainers for the infrastructure, the tools and all the runners. > > > Cleber & team are willing to serve for the infrastructure, the tools and > > > the Red Hat runners. > > > > So, with 5.0 nearly out the door it seems like a good time to check > > in on this thread again to ask where we are progress-wise with this. > > My impression is that this patchset provides most of the scripting > > and config side of the first step, so what we need is for RH to provide > > an x86 runner machine and tell the gitlab CI it exists. I appreciate > > that the whole coronavirus and working-from-home situation will have > > upended everybody's plans, especially when actual hardware might > > be involved, but how's it going ? > > > > Hi Peter, > > You hit the nail in the head here. We were affected indeed with our ability > to move some machines from one lab to another (across the country), but we're > actively working on it. > > From now on, I'll give you an update every time a significant event occurs on > our side. > Hi all, It took a while, but I finally have some updates, and they're pretty good news. Red Hat is sponsoring 3 x86_64 machines that will act as runners for the QEMU CI, and together with QEMU's own s390 and aarch64 machines (1 each), we have enough hardware now for a reasonable build and test coverage. The s390 and aarch64 machines are already up and running, and the x86_64 machines are being racked up, but should be up and running in the next few days. I'm working on an updated version of this series that takes into account this new scenario, and some fixes and improvements. And as a reminder, if you (individual or organization) would like to sponsor hardware or people to expand the QEMU build and test coverage, please reach out to us. Thanks, - Cleber. > > thanks > > -- PMM > > > > > > Thanks for checking in! > - Cleber. signature.asc Description: PGP signature
Re: [PATCH 0/5] QEMU Gating CI
On Mon, Apr 27, 2020 at 04:41:38PM +0200, Philippe Mathieu-Daudé wrote: > On 4/27/20 4:28 PM, Cleber Rosa wrote: > > On Mon, 27 Apr 2020 12:51:36 +0200 > > Philippe Mathieu-Daudé wrote: > > > > > On 4/27/20 7:12 AM, Cleber Rosa wrote: > > > > On Thu, 23 Apr 2020 23:28:21 +0200 > > > > Philippe Mathieu-Daudé wrote: > > > [...] > > > > > In some cases custom runners are acceptable. These runners won't be > > > > > "gating" but can post informative log and status. > > > > > > > > > > > > > Well, I have the feeling that some people maintaining those runners > > > > will *not* want to have them as "informational" only. If they > > > > invest a good amount of time on them, I believe they'll want to > > > > reap the benefits such as other not breaking the code they rely on. > > > > If their system is not gating, they lose that and may find > > > > breakage that CI did not catch. Again, I don't think "easily > > > > accessible" hardware should be the only criteria for > > > > gating/non-gating status. > > > > > > > > For instance, would you consider, say, a "Raspberry Pi 4 Model > > > > B", running KVM jobs to be a reproducible runner? Would you blame a > > > > developer that breaks a Gating CI job on such a platform and says > > > > that he can not reproduce it? > > > > > > I'm not sure I understood the problem, as I'd answer "yes" but I > > > guess you expect me to say "no"? > > > > > > > What I mean is: would you blame such a developer for *not* having a > > machine himself/herself that he/she can try to reproduce the failure? > > And would you consider a "Raspberry Pi 4 Model B" an easily available > > hardware? > > My view on this is if someone merged code in mainstream QEMU and maintains > it, and if it is not easy to reproduce the setup (for a bug reported by a CI > script), then it is the responsibility of the maintainer to resolve it. > Either by providing particular access to the hardware, or be ready to spend > a long debugging session over email and multiple time zones. > > If it is not possible, then this specific code/setup can not claim for > gating CI, and eventually mainstream isn't the best place for it. I'd caution to be wary about using gating CI as a big stick for hitting contributors with. The more rules we put in place whicih contributors have to follow before their work gets accepted for merge, the less likely someone is to have a positive experiance contributing to the project, or even be willing to try. This view of gating CI requirements was a negative aspect of contributing to the OpenStack project, which drove people away. There was pushback against contributing work because it lacked CI, but there was often no viable way for to actually provide CI in a feasible timeframe, especially for stuff only testable in physical hardware and not VMs. Even if you work for a big company, it doesn't make it easy to magic up money to spend on hardware & hosting to provide CI, as corporate beaurcracy & priorities will get in your way. I'd really encourage the more nuanced approach of thinking in terms of tiered support levels: - Tier 1: features that we have gating CI tests for. Will always work. - Tier 2: features that we have non-gating CI test for. Should work at time of release, but may be broken for periods in git master. - Tier 3: features that we don't have CI tests for. Compile tested only, relying on end user manual testing, so may or may not work at any time or in release. Obviously tier 1 is the gold standard that we would like everything to achieve but we'll never achieve that reality unless we cull 90% of QEMU's code. I don't think that's in the best interests of our users, because clearly stuff in Tier 2 and Tier 3 is still useful for a large portion of our end users - not least because Tier 3 is the level everything is at right now in QEMU unless using a downstream vendor's packages. The tier levels and CI are largely around setting reasonable quality expectations. Right now we often have a problem that poeople want to re-factor code but are afraid of breaking existing functionality that guests rely on. This causes delays in merging code or causes people to not even attempt the refactoring in the first place. This harms our forward progress in QEMU. With gating CI, we are declaring that contributors should feel free to refactor anything as long as it passes gating CI. IOW, contributors only have to care about Tier 1 features continuing to work. It would be nice if refactoring does not breaks stuff in Tier 2 / 3, but if it does, then that is acceptable collatoral damage. We would not block the merge on stuff that is Tier 2 / 3. Based on what I experianced in OpenStack the other big challenge is deciding when something can be promoted from Tier 2 to Tier 1. They had the official gating CI (for Tier 1) being maintained by the core project infrastructure team. Any CI provided by third party companies was
Re: [PATCH 0/5] QEMU Gating CI
On Mon, 27 Apr 2020 16:41:38 +0200 Philippe Mathieu-Daudé wrote: > On 4/27/20 4:28 PM, Cleber Rosa wrote: > > > > What I mean is: would you blame such a developer for *not* having a > > machine himself/herself that he/she can try to reproduce the > > failure? And would you consider a "Raspberry Pi 4 Model B" an > > easily available hardware? > > My view on this is if someone merged code in mainstream QEMU and > maintains it, and if it is not easy to reproduce the setup (for a bug > reported by a CI script), then it is the responsibility of the > maintainer to resolve it. Either by providing particular access to > the hardware, or be ready to spend a long debugging session over > email and multiple time zones. > Right, the "easy to reproduce" has a lot to with access to hardware, and a lot to do with access to the same or reproducible setup. And yes, if I maintain platform/job "foobar" that was once upgraded to gating status, has since then fallen behind and doesn't allow users to easily reproduce it, it all falls unto the maintainer to resolve issues. I'd even say that people having access to identical hardware could proactively challenge a given job status as gating if they fail to reproduce it with the provided documentation/scripts. > If it is not possible, then this specific code/setup can not claim > for gating CI, and eventually mainstream isn't the best place for it. > > >> [...] > IIUC, we're in agreement. :) Thanks, - CLeber.
Re: [PATCH 0/5] QEMU Gating CI
On 4/27/20 4:28 PM, Cleber Rosa wrote: On Mon, 27 Apr 2020 12:51:36 +0200 Philippe Mathieu-Daudé wrote: On 4/27/20 7:12 AM, Cleber Rosa wrote: On Thu, 23 Apr 2020 23:28:21 +0200 Philippe Mathieu-Daudé wrote: [...] In some cases custom runners are acceptable. These runners won't be "gating" but can post informative log and status. Well, I have the feeling that some people maintaining those runners will *not* want to have them as "informational" only. If they invest a good amount of time on them, I believe they'll want to reap the benefits such as other not breaking the code they rely on. If their system is not gating, they lose that and may find breakage that CI did not catch. Again, I don't think "easily accessible" hardware should be the only criteria for gating/non-gating status. For instance, would you consider, say, a "Raspberry Pi 4 Model B", running KVM jobs to be a reproducible runner? Would you blame a developer that breaks a Gating CI job on such a platform and says that he can not reproduce it? I'm not sure I understood the problem, as I'd answer "yes" but I guess you expect me to say "no"? What I mean is: would you blame such a developer for *not* having a machine himself/herself that he/she can try to reproduce the failure? And would you consider a "Raspberry Pi 4 Model B" an easily available hardware? My view on this is if someone merged code in mainstream QEMU and maintains it, and if it is not easy to reproduce the setup (for a bug reported by a CI script), then it is the responsibility of the maintainer to resolve it. Either by providing particular access to the hardware, or be ready to spend a long debugging session over email and multiple time zones. If it is not possible, then this specific code/setup can not claim for gating CI, and eventually mainstream isn't the best place for it. [...]
Re: [PATCH 0/5] QEMU Gating CI
On Mon, 27 Apr 2020 12:51:36 +0200 Philippe Mathieu-Daudé wrote: > On 4/27/20 7:12 AM, Cleber Rosa wrote: > > On Thu, 23 Apr 2020 23:28:21 +0200 > > Philippe Mathieu-Daudé wrote: > [...] > >> In some cases custom runners are acceptable. These runners won't be > >> "gating" but can post informative log and status. > >> > > > > Well, I have the feeling that some people maintaining those runners > > will *not* want to have them as "informational" only. If they > > invest a good amount of time on them, I believe they'll want to > > reap the benefits such as other not breaking the code they rely on. > > If their system is not gating, they lose that and may find > > breakage that CI did not catch. Again, I don't think "easily > > accessible" hardware should be the only criteria for > > gating/non-gating status. > > > > For instance, would you consider, say, a "Raspberry Pi 4 Model > > B", running KVM jobs to be a reproducible runner? Would you blame a > > developer that breaks a Gating CI job on such a platform and says > > that he can not reproduce it? > > I'm not sure I understood the problem, as I'd answer "yes" but I > guess you expect me to say "no"? > What I mean is: would you blame such a developer for *not* having a machine himself/herself that he/she can try to reproduce the failure? And would you consider a "Raspberry Pi 4 Model B" an easily available hardware? > [...] > >> Now the problem is GitLab runner is not natively available on the > >> architectures listed in this mail, so custom setup is required. A > >> dumb script running ssh to a machine also works (tested) but lot of > >> manual tuning/maintenance expected. > >> > > > > That's where I'm trying to help. I built and tested the > > gitlab-runner for a number of non-supported environments, and I > > expect to build further on that (say contributing code or feedback > > back to GitLab so they become official builds?). > > Good luck with that, it took more that 2 years to GitLab to > officially support AMD64: > https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/725 > You mean aarch64, sure. I'm not holding my breath, because we can always have our own binaries/ports (or other executors such as ssh) but I'm optimistic... > Hopefully the first non-x86 user was the hardest one who had to do > all the bad work, and next architecture might get supported quicker... > ... and this point is one of the reasons. The other is competition from Travis-CI (and others). Cheers, - Cleber.
Re: [PATCH 0/5] QEMU Gating CI
On 4/27/20 7:12 AM, Cleber Rosa wrote: On Thu, 23 Apr 2020 23:28:21 +0200 Philippe Mathieu-Daudé wrote: [...] In some cases custom runners are acceptable. These runners won't be "gating" but can post informative log and status. Well, I have the feeling that some people maintaining those runners will *not* want to have them as "informational" only. If they invest a good amount of time on them, I believe they'll want to reap the benefits such as other not breaking the code they rely on. If their system is not gating, they lose that and may find breakage that CI did not catch. Again, I don't think "easily accessible" hardware should be the only criteria for gating/non-gating status. For instance, would you consider, say, a "Raspberry Pi 4 Model B", running KVM jobs to be a reproducible runner? Would you blame a developer that breaks a Gating CI job on such a platform and says that he can not reproduce it? I'm not sure I understood the problem, as I'd answer "yes" but I guess you expect me to say "no"? [...] Now the problem is GitLab runner is not natively available on the architectures listed in this mail, so custom setup is required. A dumb script running ssh to a machine also works (tested) but lot of manual tuning/maintenance expected. That's where I'm trying to help. I built and tested the gitlab-runner for a number of non-supported environments, and I expect to build further on that (say contributing code or feedback back to GitLab so they become official builds?). Good luck with that, it took more that 2 years to GitLab to officially support AMD64: https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/725 Hopefully the first non-x86 user was the hardest one who had to do all the bad work, and next architecture might get supported quicker... Cheers, - Cleber.
Re: [PATCH 0/5] QEMU Gating CI
On Mon, 2020-04-27 at 01:24 -0400, Cleber Rosa wrote: > On Fri, 24 Apr 2020 08:57:46 +0200 > Erik Skultety wrote: > > On Thu, Apr 23, 2020 at 11:28:21PM +0200, Philippe Mathieu-Daudé > > wrote: > > > Thanks to insist with that point Daniel. I'd rather see every > > > configuration reproducible, so if we loose a hardware sponsor, we > > > can find another one and start another runner. > > > Also note, if it is not easy to reproduce a runner, it will be very > > > hard to debug a reported build/test error. > > > > (Thanks for bringing ^this point up Philippe) > > > > ...However, what we've been actively working on in libvirt is to > > extend the lcitool we have (which can spawn local test VMs) to the > > point where we're able to generate machines that would be the > > reproducible. Right now I'm playing with cloud-init integration with > > lcitool (patches coming soon) that would allow us to use the same > > machines locally as we'd want to have in, say, OpenStack and share > > them as compressed images, so even when updated/managed by lcitool > > locally, you'd get the same environment. > > This is great, and it aligns with my previous point that reproducibility > it's not *just* about the hardware, but about diligently documenting > and automating the environments, be them mundane or super specialized. > And IMO that should score some points when it comes to being > promoted/demoted as a Gating machine/job. I think there's room to extend and rework lcitool so that it can be used for QEMU CI as well, and we should definitely look into that. Right now it only really covers VMs and containers, but there's already one situation (FreeBSD) where the expectation is that you'd import an existing VM image and then apply CI-related customizations on top, so it's not too much of a stretch to imagine doing the same for a bare metal machine. We use Ansible, so as long as we can connect to the machine via ssh we're pretty much good to go. Installation of VMs we already perform in an unattended fashion using preseed/kickstart, and it should be relatively straighforward to adapt those configurations to also work on real hardware. This way we'd both be able to rely on having a sane OS as the base, and relieve the administrator from the duty of manually installing the machines. -- Andrea Bolognani / Red Hat / Virtualization
Re: [PATCH 0/5] QEMU Gating CI
On Fri, 24 Apr 2020 10:30:23 +0100 Daniel P. Berrangé wrote: > On Thu, Apr 23, 2020 at 01:36:48PM -0400, Cleber Rosa wrote: > > > > > > - Original Message - > > > From: "Daniel P. Berrangé" > > > To: "Cleber Rosa" > > > Cc: "Peter Maydell" , "Fam Zheng" > > > , "Thomas Huth" , "Beraldo > > > Leal" , "Erik Skultety" , > > > "Philippe Mathieu-Daudé" , "Wainer Moschetta" > > > , "Markus Armbruster" , > > > "Wainer dos Santos Moschetta" , "QEMU > > > Developers" , "Willian Rampazzo" > > > , "Alex Bennée" , > > > "Eduardo Habkost" Sent: Thursday, April 23, > > > 2020 1:13:22 PM Subject: Re: [PATCH 0/5] QEMU Gating CI > > > > > > On Thu, Apr 23, 2020 at 01:04:13PM -0400, Cleber Rosa wrote: > > > > > > > > > > > > - Original Message - > > > > > From: "Peter Maydell" > > > > > To: "Markus Armbruster" > > > > > Cc: "Fam Zheng" , "Thomas Huth" > > > > > , "Beraldo Leal" , "Erik > > > > > Skultety" , "Alex Bennée" > > > > > , "Wainer Moschetta" > > > > > , "QEMU Developers" > > > > > , "Wainer dos Santos Moschetta" > > > > > , "Willian Rampazzo" > > > > > , "Cleber Rosa" , > > > > > "Philippe Mathieu-Daudé" , "Eduardo > > > > > Habkost" Sent: Tuesday, April 21, 2020 > > > > > 8:53:49 AM Subject: Re: [PATCH 0/5] QEMU Gating CI > > > > > > > > > > On Thu, 19 Mar 2020 at 16:33, Markus Armbruster > > > > > wrote: > > > > > > Peter Maydell writes: > > > > > > > I think we should start by getting the gitlab setup > > > > > > > working for the basic "x86 configs" first. Then we can > > > > > > > try adding a runner for s390 (that one's logistically > > > > > > > easiest because it is a project machine, not one owned by > > > > > > > me personally or by Linaro) once the basic framework is > > > > > > > working, and expand from there. > > > > > > > > > > > > Makes sense to me. > > > > > > > > > > > > Next steps to get this off the ground: > > > > > > > > > > > > * Red Hat provides runner(s) for x86 stuff we care about. > > > > > > > > > > > > * If that doesn't cover 'basic "x86 configs" in your > > > > > > judgement, we fill the gaps as described below under > > > > > > "Expand from there". > > > > > > > > > > > > * Add an s390 runner using the project machine you > > > > > > mentioned. > > > > > > > > > > > > * Expand from there: identify the remaining gaps, map them > > > > > > to people / organizations interested in them, and solicit > > > > > > contributions from these > > > > > > guys. > > > > > > > > > > > > A note on contributions: we need both hardware and people. > > > > > > By people I mean maintainers for the infrastructure, the > > > > > > tools and all the runners. Cleber & team are willing to > > > > > > serve for the infrastructure, the tools and > > > > > > the Red Hat runners. > > > > > > > > > > So, with 5.0 nearly out the door it seems like a good time to > > > > > check in on this thread again to ask where we are > > > > > progress-wise with this. My impression is that this patchset > > > > > provides most of the scripting and config side of the first > > > > > step, so what we need is for RH to provide an x86 runner > > > > > machine and tell the gitlab CI it exists. I appreciate that > > > > > the whole coronavirus and working-from-home situation will > > > > > have upended everybody's plans, especially when actual > > > > > hardware might be involved, but how's it going ? > > > > > > > > > > > > > Hi Peter, > > > > > > > > You hit the nail in the head here.
Re: [PATCH 0/5] QEMU Gating CI
On Fri, 24 Apr 2020 08:57:46 +0200 Erik Skultety wrote: > On Thu, Apr 23, 2020 at 11:28:21PM +0200, Philippe Mathieu-Daudé > wrote: > > On 4/23/20 7:13 PM, Daniel P. Berrangé wrote: > > > On Thu, Apr 23, 2020 at 01:04:13PM -0400, Cleber Rosa wrote: > > > > - Original Message - > > > > > From: "Peter Maydell" > > > > > To: "Markus Armbruster" > > > > > Cc: "Fam Zheng" , "Thomas Huth" > > > > > , "Beraldo Leal" , "Erik > > > > > Skultety" , "Alex Bennée" > > > > > , "Wainer Moschetta" > > > > > , "QEMU Developers" > > > > > , "Wainer dos Santos Moschetta" > > > > > , "Willian Rampazzo" > > > > > , "Cleber Rosa" , > > > > > "Philippe Mathieu-Daudé" , "Eduardo > > > > > Habkost" Sent: Tuesday, April 21, 2020 > > > > > 8:53:49 AM Subject: Re: [PATCH 0/5] QEMU Gating CI > > > > > > > > > > On Thu, 19 Mar 2020 at 16:33, Markus Armbruster > > > > > wrote: > > > > > > Peter Maydell writes: > > > > > > > I think we should start by getting the gitlab setup > > > > > > > working for the basic "x86 configs" first. Then we can > > > > > > > try adding a runner for s390 (that one's logistically > > > > > > > easiest because it is a project machine, not one owned by > > > > > > > me personally or by Linaro) once the basic framework is > > > > > > > working, and expand from there. > > > > > > > > > > > > Makes sense to me. > > > > > > > > > > > > Next steps to get this off the ground: > > > > > > > > > > > > * Red Hat provides runner(s) for x86 stuff we care about. > > > > > > > > > > > > * If that doesn't cover 'basic "x86 configs" in your > > > > > > judgement, we fill the gaps as described below under > > > > > > "Expand from there". > > > > > > > > > > > > * Add an s390 runner using the project machine you > > > > > > mentioned. > > > > > > > > > > > > * Expand from there: identify the remaining gaps, map them > > > > > > to people / organizations interested in them, and solicit > > > > > > contributions from these guys. > > > > > > > > > > > > A note on contributions: we need both hardware and people. > > > > > > By people I mean maintainers for the infrastructure, the > > > > > > tools and all the runners. Cleber & team are willing to > > > > > > serve for the infrastructure, the tools and the Red Hat > > > > > > runners. > > > > > > > > > > So, with 5.0 nearly out the door it seems like a good time to > > > > > check in on this thread again to ask where we are > > > > > progress-wise with this. My impression is that this patchset > > > > > provides most of the scripting and config side of the first > > > > > step, so what we need is for RH to provide an x86 runner > > > > > machine and tell the gitlab CI it exists. I appreciate that > > > > > the whole coronavirus and working-from-home situation will > > > > > have upended everybody's plans, especially when actual > > > > > hardware might be involved, but how's it going ? > > > > > > > > > > > > > Hi Peter, > > > > > > > > You hit the nail in the head here. We were affected indeed > > > > with our ability to move some machines from one lab to another > > > > (across the country), but we're actively working on it. > > > > > > For x86, do we really need to be using custom runners ? > > > > > > With GitLab if someone forks the repo to their personal > > > namespace, they cannot use any custom runners setup by the origin > > > project. So if we use custom runners for x86, people forking > > > won't be able to run the GitLab CI jobs. > > > > > > As a sub-system maintainer I wouldn't like this, because I > > > ideally want to be able to run the same jobs on my staging tree, > > > that Peter will run at merge time for the
Re: [PATCH 0/5] QEMU Gating CI
On Thu, 23 Apr 2020 23:28:21 +0200 Philippe Mathieu-Daudé wrote: > On 4/23/20 7:13 PM, Daniel P. Berrangé wrote: > > On Thu, Apr 23, 2020 at 01:04:13PM -0400, Cleber Rosa wrote: > >> - Original Message - > >>> From: "Peter Maydell" > >>> To: "Markus Armbruster" > >>> Cc: "Fam Zheng" , "Thomas Huth" > >>> , "Beraldo Leal" , "Erik > >>> Skultety" , "Alex Bennée" > >>> , "Wainer Moschetta" > >>> , "QEMU Developers" , > >>> "Wainer dos Santos Moschetta" , "Willian > >>> Rampazzo" , "Cleber Rosa" > >>> , "Philippe Mathieu-Daudé" , > >>> "Eduardo Habkost" Sent: Tuesday, April 21, > >>> 2020 8:53:49 AM Subject: Re: [PATCH 0/5] QEMU Gating CI > >>> > >>> On Thu, 19 Mar 2020 at 16:33, Markus Armbruster > >>> wrote: > >>>> Peter Maydell writes: > >>>>> I think we should start by getting the gitlab setup working > >>>>> for the basic "x86 configs" first. Then we can try adding > >>>>> a runner for s390 (that one's logistically easiest because > >>>>> it is a project machine, not one owned by me personally or > >>>>> by Linaro) once the basic framework is working, and expand > >>>>> from there. > >>>> > >>>> Makes sense to me. > >>>> > >>>> Next steps to get this off the ground: > >>>> > >>>> * Red Hat provides runner(s) for x86 stuff we care about. > >>>> > >>>> * If that doesn't cover 'basic "x86 configs" in your judgement, > >>>> we fill the gaps as described below under "Expand from there". > >>>> > >>>> * Add an s390 runner using the project machine you mentioned. > >>>> > >>>> * Expand from there: identify the remaining gaps, map them to > >>>> people / organizations interested in them, and solicit > >>>> contributions from these guys. > >>>> > >>>> A note on contributions: we need both hardware and people. By > >>>> people I mean maintainers for the infrastructure, the tools and > >>>> all the runners. Cleber & team are willing to serve for the > >>>> infrastructure, the tools and the Red Hat runners. > >>> > >>> So, with 5.0 nearly out the door it seems like a good time to > >>> check in on this thread again to ask where we are progress-wise > >>> with this. My impression is that this patchset provides most of > >>> the scripting and config side of the first step, so what we need > >>> is for RH to provide an x86 runner machine and tell the gitlab CI > >>> it exists. I appreciate that the whole coronavirus and > >>> working-from-home situation will have upended everybody's plans, > >>> especially when actual hardware might be involved, but how's it > >>> going ? > >>> > >> > >> Hi Peter, > >> > >> You hit the nail in the head here. We were affected indeed with > >> our ability to move some machines from one lab to another (across > >> the country), but we're actively working on it. > > > > For x86, do we really need to be using custom runners ? > > > > With GitLab if someone forks the repo to their personal namespace, > > they cannot use any custom runners setup by the origin project. So > > if we use custom runners for x86, people forking won't be able to > > run the GitLab CI jobs. > > > > As a sub-system maintainer I wouldn't like this, because I ideally > > want to be able to run the same jobs on my staging tree, that Peter > > will run at merge time for the PULL request I send. > > > > Thus my strong preference would be to use the GitLab runners in > > every scenario where they are viable to use. Only use custom > > runners in the cases where GitLab runners are clearly inadequate > > for our needs. > > > > Based on what we've setup in GitLab for libvirt, the shared runners > > they have work fine for x86. Just need the environments you are > > testing to be provided as Docker containers (you can actually build > > and cache the container images during your CI job too). IOW, any > > Linux distro build and test jobs should be able to use shared > > runners on x86, and likewise min
Re: [PATCH 0/5] QEMU Gating CI
On Thu, 23 Apr 2020 18:50:47 +0100 Peter Maydell wrote: > On Thu, 23 Apr 2020 at 18:37, Cleber Rosa wrote: > > We're already using the shared x86 runners, but with a different > > goal. The goal of the "Gating CI" is indeed to expand on non-x86 > > environments. We're in a "chicken and egg" kind of situation, > > because we'd like to prove that GitLab CI will allow QEMU to expand > > to very different runners and jobs, while not really having all > > that hardware setup and publicly available at this time. > > We do have the S390 machine that IBM kindly made available to > the project -- that is not a personal or Linaro machine, so > there are no issues with giving you a login on that so you > can set it up as a CI runner. Drop me an email if you want > access to it. > I sure do. I'll shoot you an email off-list. > thanks > -- PMM > Thanks, - Cleber.
Re: [PATCH 0/5] QEMU Gating CI
On Fri, Apr 24, 2020 at 11:30 AM Daniel P. Berrangé wrote: > > - Original Message - > > > From: "Daniel P. Berrangé" > > > To: "Cleber Rosa" [...] > > Hi Daniel, > > > > We're already using the shared x86 runners, but with a different goal. The > > goal of the "Gating CI" is indeed to expand on non-x86 environments. We're > > in a "chicken and egg" kind of situation, because we'd like to prove that > > GitLab CI will allow QEMU to expand to very different runners and jobs, > > while > > not really having all that hardware setup and publicly available at this > > time. > > > > My experiments were really around that point, I mean, confirming that we > > can grow > > the number of architectures/runners/jobs/configurations to provide a > > coverage > > equal or greater to what Peter already does. > > So IIUC, you're saying that for x86 gating, the intention is to use shared > runners in general. > > Your current work that you say is blocked on access to x86 hardware, is just > about demonstrating the concept of plugging in custom runners, while we wait > for access to non-x86 hardware ? > > > > With GitLab if someone forks the repo to their personal namespace, they > > > cannot use any custom runners setup by the origin project. So if we use > > > custom runners for x86, people forking won't be able to run the GitLab > > > CI jobs. > > > > > > > They will continue to be able use the jobs and runners already defined in > > the .gitlab-ci.yml file. This work will only affect people pushing to the/a > > "staging" branch. > > > > > As a sub-system maintainer I wouldn't like this, because I ideally want > > > to be able to run the same jobs on my staging tree, that Peter will run > > > at merge time for the PULL request I send. > > > > > > > If you're looking for symmetry between any PR and "merge time" jobs, the > > only solution is to allow any PR to access all the diverse set of non-shared > > machines we're hoping to have in the near future. This may be something > > we'll get to, but I doubt we can tackle it in the near future now. > > It occurred to me that we could do this if we grant selective access to > the Gitlab repos, to people who are official subsystem maintainers. > GitLab has a concept of "protected branches", so you can control who is > allowed to push changes on a per-branch granularity. > > So, for example, in the main qemu.git, we could create branches for each > subsystem tree eg > > staging-block > staging-qapi > staging-crypto > staging-migration > > > and for each of these branches, we can grant access to relevant subsystem > maintainer(s). The MAINTAINERS file could help us with that, we already have scripts to parse its sections. Maintainers should keep it up-to-date, then the merge script would check, i.e.: // section separator --- // ignored Trivial patches // description ignored M: Michael Tokarev M: Laurent Vivier // must match commit author T: git git://git.corpit.ru/qemu.git trivial-patches T: git https://github.com/vivier/qemu.git trivial-patches // must match MR source > > When they're ready to send a pull request to Peter, they can push their > tree to this branch. Since the branch is in the main gitlab.com/qemu/qemu > project namespace, this branch can run CI using the private QEMU runners. > The subsystem maintainer can thus see the full set of CI results across > all platforms required by Gating, before Peter even gets the pull request. > > So when Peter then looks at merging the pull request to master, the only > he's likely to see are the non-deterministic bugs, or issues caused by > semantic conflicts with other recently merged code. > > It would even be possible to do the final merge into master entirely from > GitLab, no need to go via email. When the source branch & target branch are > within the same git repo, GitLab has the ability to run CI jobs against the > resulting merge commit in a strict gating manner, before it hits master. > They call this "Merge trains" in their documentation. > > IOW, from Peter's POV, merging pull requests could be as simple as hitting > the merge button in GitLab merge request UI. Everything wrt CI would be > completely automated, and the subsystem maintainers would have the > responsibility to dealing with merge conflicts & CI failures, which is > more scalable for the person co-ordinating the merges into master. > > > Regards, > Daniel
Re: [PATCH 0/5] QEMU Gating CI
On Thu, Apr 23, 2020 at 01:36:48PM -0400, Cleber Rosa wrote: > > > - Original Message - > > From: "Daniel P. Berrangé" > > To: "Cleber Rosa" > > Cc: "Peter Maydell" , "Fam Zheng" > > , "Thomas Huth" , > > "Beraldo Leal" , "Erik Skultety" , > > "Philippe Mathieu-Daudé" > > , "Wainer Moschetta" , "Markus > > Armbruster" , "Wainer dos > > Santos Moschetta" , "QEMU Developers" > > , "Willian Rampazzo" > > , "Alex Bennée" , "Eduardo > > Habkost" > > Sent: Thursday, April 23, 2020 1:13:22 PM > > Subject: Re: [PATCH 0/5] QEMU Gating CI > > > > On Thu, Apr 23, 2020 at 01:04:13PM -0400, Cleber Rosa wrote: > > > > > > > > > - Original Message - > > > > From: "Peter Maydell" > > > > To: "Markus Armbruster" > > > > Cc: "Fam Zheng" , "Thomas Huth" , > > > > "Beraldo Leal" , "Erik > > > > Skultety" , "Alex Bennée" , > > > > "Wainer Moschetta" , > > > > "QEMU Developers" , "Wainer dos Santos Moschetta" > > > > , "Willian Rampazzo" > > > > , "Cleber Rosa" , "Philippe > > > > Mathieu-Daudé" , "Eduardo > > > > Habkost" > > > > Sent: Tuesday, April 21, 2020 8:53:49 AM > > > > Subject: Re: [PATCH 0/5] QEMU Gating CI > > > > > > > > On Thu, 19 Mar 2020 at 16:33, Markus Armbruster > > > > wrote: > > > > > Peter Maydell writes: > > > > > > I think we should start by getting the gitlab setup working > > > > > > for the basic "x86 configs" first. Then we can try adding > > > > > > a runner for s390 (that one's logistically easiest because > > > > > > it is a project machine, not one owned by me personally or > > > > > > by Linaro) once the basic framework is working, and expand > > > > > > from there. > > > > > > > > > > Makes sense to me. > > > > > > > > > > Next steps to get this off the ground: > > > > > > > > > > * Red Hat provides runner(s) for x86 stuff we care about. > > > > > > > > > > * If that doesn't cover 'basic "x86 configs" in your judgement, we > > > > > fill the gaps as described below under "Expand from there". > > > > > > > > > > * Add an s390 runner using the project machine you mentioned. > > > > > > > > > > * Expand from there: identify the remaining gaps, map them to people / > > > > > organizations interested in them, and solicit contributions from > > > > > these > > > > > guys. > > > > > > > > > > A note on contributions: we need both hardware and people. By people > > > > > I > > > > > mean maintainers for the infrastructure, the tools and all the > > > > > runners. > > > > > Cleber & team are willing to serve for the infrastructure, the tools > > > > > and > > > > > the Red Hat runners. > > > > > > > > So, with 5.0 nearly out the door it seems like a good time to check > > > > in on this thread again to ask where we are progress-wise with this. > > > > My impression is that this patchset provides most of the scripting > > > > and config side of the first step, so what we need is for RH to provide > > > > an x86 runner machine and tell the gitlab CI it exists. I appreciate > > > > that the whole coronavirus and working-from-home situation will have > > > > upended everybody's plans, especially when actual hardware might > > > > be involved, but how's it going ? > > > > > > > > > > Hi Peter, > > > > > > You hit the nail in the head here. We were affected indeed with our > > > ability > > > to move some machines from one lab to another (across the country), but > > > we're > > > actively working on it. > > > > For x86, do we really need to be using custom runners ? > > > > Hi Daniel, > > We're already using the shared x86 runners, but with a different goal. The > goal of the "Gating CI" is indeed to expand
Re: [PATCH 0/5] QEMU Gating CI
On Thu, Apr 23, 2020 at 11:28:21PM +0200, Philippe Mathieu-Daudé wrote: > On 4/23/20 7:13 PM, Daniel P. Berrangé wrote: > > On Thu, Apr 23, 2020 at 01:04:13PM -0400, Cleber Rosa wrote: > > > - Original Message - > > > > From: "Peter Maydell" > > > > To: "Markus Armbruster" > > > > Cc: "Fam Zheng" , "Thomas Huth" , > > > > "Beraldo Leal" , "Erik > > > > Skultety" , "Alex Bennée" > > > > , "Wainer Moschetta" , > > > > "QEMU Developers" , "Wainer dos Santos > > > > Moschetta" , "Willian Rampazzo" > > > > , "Cleber Rosa" , "Philippe > > > > Mathieu-Daudé" , "Eduardo > > > > Habkost" > > > > Sent: Tuesday, April 21, 2020 8:53:49 AM > > > > Subject: Re: [PATCH 0/5] QEMU Gating CI > > > > > > > > On Thu, 19 Mar 2020 at 16:33, Markus Armbruster > > > > wrote: > > > > > Peter Maydell writes: > > > > > > I think we should start by getting the gitlab setup working > > > > > > for the basic "x86 configs" first. Then we can try adding > > > > > > a runner for s390 (that one's logistically easiest because > > > > > > it is a project machine, not one owned by me personally or > > > > > > by Linaro) once the basic framework is working, and expand > > > > > > from there. > > > > > > > > > > Makes sense to me. > > > > > > > > > > Next steps to get this off the ground: > > > > > > > > > > * Red Hat provides runner(s) for x86 stuff we care about. > > > > > > > > > > * If that doesn't cover 'basic "x86 configs" in your judgement, we > > > > >fill the gaps as described below under "Expand from there". > > > > > > > > > > * Add an s390 runner using the project machine you mentioned. > > > > > > > > > > * Expand from there: identify the remaining gaps, map them to people / > > > > >organizations interested in them, and solicit contributions from > > > > > these > > > > >guys. > > > > > > > > > > A note on contributions: we need both hardware and people. By people > > > > > I > > > > > mean maintainers for the infrastructure, the tools and all the > > > > > runners. > > > > > Cleber & team are willing to serve for the infrastructure, the tools > > > > > and > > > > > the Red Hat runners. > > > > > > > > So, with 5.0 nearly out the door it seems like a good time to check > > > > in on this thread again to ask where we are progress-wise with this. > > > > My impression is that this patchset provides most of the scripting > > > > and config side of the first step, so what we need is for RH to provide > > > > an x86 runner machine and tell the gitlab CI it exists. I appreciate > > > > that the whole coronavirus and working-from-home situation will have > > > > upended everybody's plans, especially when actual hardware might > > > > be involved, but how's it going ? > > > > > > > > > > Hi Peter, > > > > > > You hit the nail in the head here. We were affected indeed with our > > > ability > > > to move some machines from one lab to another (across the country), but > > > we're > > > actively working on it. > > > > For x86, do we really need to be using custom runners ? > > > > With GitLab if someone forks the repo to their personal namespace, they > > cannot use any custom runners setup by the origin project. So if we use > > custom runners for x86, people forking won't be able to run the GitLab > > CI jobs. > > > > As a sub-system maintainer I wouldn't like this, because I ideally want > > to be able to run the same jobs on my staging tree, that Peter will run > > at merge time for the PULL request I send. > > > > Thus my strong preference would be to use the GitLab runners in every > > scenario where they are viable to use. Only use custom runners in the > > cases where GitLab runners are clearly inadequate for our needs. > > > > Based on what we've setup in GitLab for libvirt, the shared runners > > they have work fine for x8
Re: [PATCH 0/5] QEMU Gating CI
On 4/23/20 7:13 PM, Daniel P. Berrangé wrote: On Thu, Apr 23, 2020 at 01:04:13PM -0400, Cleber Rosa wrote: - Original Message - From: "Peter Maydell" To: "Markus Armbruster" Cc: "Fam Zheng" , "Thomas Huth" , "Beraldo Leal" , "Erik Skultety" , "Alex Bennée" , "Wainer Moschetta" , "QEMU Developers" , "Wainer dos Santos Moschetta" , "Willian Rampazzo" , "Cleber Rosa" , "Philippe Mathieu-Daudé" , "Eduardo Habkost" Sent: Tuesday, April 21, 2020 8:53:49 AM Subject: Re: [PATCH 0/5] QEMU Gating CI On Thu, 19 Mar 2020 at 16:33, Markus Armbruster wrote: Peter Maydell writes: I think we should start by getting the gitlab setup working for the basic "x86 configs" first. Then we can try adding a runner for s390 (that one's logistically easiest because it is a project machine, not one owned by me personally or by Linaro) once the basic framework is working, and expand from there. Makes sense to me. Next steps to get this off the ground: * Red Hat provides runner(s) for x86 stuff we care about. * If that doesn't cover 'basic "x86 configs" in your judgement, we fill the gaps as described below under "Expand from there". * Add an s390 runner using the project machine you mentioned. * Expand from there: identify the remaining gaps, map them to people / organizations interested in them, and solicit contributions from these guys. A note on contributions: we need both hardware and people. By people I mean maintainers for the infrastructure, the tools and all the runners. Cleber & team are willing to serve for the infrastructure, the tools and the Red Hat runners. So, with 5.0 nearly out the door it seems like a good time to check in on this thread again to ask where we are progress-wise with this. My impression is that this patchset provides most of the scripting and config side of the first step, so what we need is for RH to provide an x86 runner machine and tell the gitlab CI it exists. I appreciate that the whole coronavirus and working-from-home situation will have upended everybody's plans, especially when actual hardware might be involved, but how's it going ? Hi Peter, You hit the nail in the head here. We were affected indeed with our ability to move some machines from one lab to another (across the country), but we're actively working on it. For x86, do we really need to be using custom runners ? With GitLab if someone forks the repo to their personal namespace, they cannot use any custom runners setup by the origin project. So if we use custom runners for x86, people forking won't be able to run the GitLab CI jobs. As a sub-system maintainer I wouldn't like this, because I ideally want to be able to run the same jobs on my staging tree, that Peter will run at merge time for the PULL request I send. Thus my strong preference would be to use the GitLab runners in every scenario where they are viable to use. Only use custom runners in the cases where GitLab runners are clearly inadequate for our needs. Based on what we've setup in GitLab for libvirt, the shared runners they have work fine for x86. Just need the environments you are testing to be provided as Docker containers (you can actually build and cache the container images during your CI job too). IOW, any Linux distro build and test jobs should be able to use shared runners on x86, and likewise mingw builds. Custom runners should only be needed if the jobs need todo *BSD / macOS builds, and/or have access to specific hardware devices for some reason. Thanks to insist with that point Daniel. I'd rather see every configuration reproducible, so if we loose a hardware sponsor, we can find another one and start another runner. Also note, if it is not easy to reproduce a runner, it will be very hard to debug a reported build/test error. A non-reproducible runner can not be used as gating, because if they fail it is not acceptable to lock the project development process. In some cases custom runners are acceptable. These runners won't be "gating" but can post informative log and status. [*] Specific hardware that is not easily available. - Alistair at last KVM forum talked about a RISCV board (to test host TCG) - Aleksandar said at last KVM forum Wavecomp could plug a CI20 MIPS (to test host TCG) - Lemote seems interested to setup some Loongson MIPSr6 board (to test interaction with KVM) [*] To run code requiring accepting License Agreements [*] To run non Free / Open Source code Owner of these runners take the responsibility to provide enough time/information about reported bugs, or to debug them themselves. Now the problem is GitLab runner is not natively available on the architectures listed in this mail, so custom setup is required. A dumb script running ssh to a machine also works (tested) but lot of manual tuning/maintenance expected.
Re: [PATCH 0/5] QEMU Gating CI
On Thu, 23 Apr 2020 at 18:37, Cleber Rosa wrote: > We're already using the shared x86 runners, but with a different goal. The > goal of the "Gating CI" is indeed to expand on non-x86 environments. We're > in a "chicken and egg" kind of situation, because we'd like to prove that > GitLab CI will allow QEMU to expand to very different runners and jobs, while > not really having all that hardware setup and publicly available at this time. We do have the S390 machine that IBM kindly made available to the project -- that is not a personal or Linaro machine, so there are no issues with giving you a login on that so you can set it up as a CI runner. Drop me an email if you want access to it. thanks -- PMM
Re: [PATCH 0/5] QEMU Gating CI
- Original Message - > From: "Daniel P. Berrangé" > To: "Cleber Rosa" > Cc: "Peter Maydell" , "Fam Zheng" > , "Thomas Huth" , > "Beraldo Leal" , "Erik Skultety" , > "Philippe Mathieu-Daudé" > , "Wainer Moschetta" , "Markus > Armbruster" , "Wainer dos > Santos Moschetta" , "QEMU Developers" > , "Willian Rampazzo" > , "Alex Bennée" , "Eduardo > Habkost" > Sent: Thursday, April 23, 2020 1:13:22 PM > Subject: Re: [PATCH 0/5] QEMU Gating CI > > On Thu, Apr 23, 2020 at 01:04:13PM -0400, Cleber Rosa wrote: > > > > > > - Original Message - > > > From: "Peter Maydell" > > > To: "Markus Armbruster" > > > Cc: "Fam Zheng" , "Thomas Huth" , > > > "Beraldo Leal" , "Erik > > > Skultety" , "Alex Bennée" , > > > "Wainer Moschetta" , > > > "QEMU Developers" , "Wainer dos Santos Moschetta" > > > , "Willian Rampazzo" > > > , "Cleber Rosa" , "Philippe > > > Mathieu-Daudé" , "Eduardo > > > Habkost" > > > Sent: Tuesday, April 21, 2020 8:53:49 AM > > > Subject: Re: [PATCH 0/5] QEMU Gating CI > > > > > > On Thu, 19 Mar 2020 at 16:33, Markus Armbruster > > > wrote: > > > > Peter Maydell writes: > > > > > I think we should start by getting the gitlab setup working > > > > > for the basic "x86 configs" first. Then we can try adding > > > > > a runner for s390 (that one's logistically easiest because > > > > > it is a project machine, not one owned by me personally or > > > > > by Linaro) once the basic framework is working, and expand > > > > > from there. > > > > > > > > Makes sense to me. > > > > > > > > Next steps to get this off the ground: > > > > > > > > * Red Hat provides runner(s) for x86 stuff we care about. > > > > > > > > * If that doesn't cover 'basic "x86 configs" in your judgement, we > > > > fill the gaps as described below under "Expand from there". > > > > > > > > * Add an s390 runner using the project machine you mentioned. > > > > > > > > * Expand from there: identify the remaining gaps, map them to people / > > > > organizations interested in them, and solicit contributions from > > > > these > > > > guys. > > > > > > > > A note on contributions: we need both hardware and people. By people I > > > > mean maintainers for the infrastructure, the tools and all the runners. > > > > Cleber & team are willing to serve for the infrastructure, the tools > > > > and > > > > the Red Hat runners. > > > > > > So, with 5.0 nearly out the door it seems like a good time to check > > > in on this thread again to ask where we are progress-wise with this. > > > My impression is that this patchset provides most of the scripting > > > and config side of the first step, so what we need is for RH to provide > > > an x86 runner machine and tell the gitlab CI it exists. I appreciate > > > that the whole coronavirus and working-from-home situation will have > > > upended everybody's plans, especially when actual hardware might > > > be involved, but how's it going ? > > > > > > > Hi Peter, > > > > You hit the nail in the head here. We were affected indeed with our > > ability > > to move some machines from one lab to another (across the country), but > > we're > > actively working on it. > > For x86, do we really need to be using custom runners ? > Hi Daniel, We're already using the shared x86 runners, but with a different goal. The goal of the "Gating CI" is indeed to expand on non-x86 environments. We're in a "chicken and egg" kind of situation, because we'd like to prove that GitLab CI will allow QEMU to expand to very different runners and jobs, while not really having all that hardware setup and publicly available at this time. My experiments were really around that point, I mean, confirming that we can grow the number of architectures/runners/jobs/configurations to provide a coverage equal or greater to what Peter already does. > With GitLab if someone forks the repo to their personal namespace, they > cannot u
Re: [PATCH 0/5] QEMU Gating CI
On Thu, Apr 23, 2020 at 01:04:13PM -0400, Cleber Rosa wrote: > > > - Original Message - > > From: "Peter Maydell" > > To: "Markus Armbruster" > > Cc: "Fam Zheng" , "Thomas Huth" , > > "Beraldo Leal" , "Erik > > Skultety" , "Alex Bennée" , > > "Wainer Moschetta" , > > "QEMU Developers" , "Wainer dos Santos Moschetta" > > , "Willian Rampazzo" > > , "Cleber Rosa" , "Philippe > > Mathieu-Daudé" , "Eduardo > > Habkost" > > Sent: Tuesday, April 21, 2020 8:53:49 AM > > Subject: Re: [PATCH 0/5] QEMU Gating CI > > > > On Thu, 19 Mar 2020 at 16:33, Markus Armbruster wrote: > > > Peter Maydell writes: > > > > I think we should start by getting the gitlab setup working > > > > for the basic "x86 configs" first. Then we can try adding > > > > a runner for s390 (that one's logistically easiest because > > > > it is a project machine, not one owned by me personally or > > > > by Linaro) once the basic framework is working, and expand > > > > from there. > > > > > > Makes sense to me. > > > > > > Next steps to get this off the ground: > > > > > > * Red Hat provides runner(s) for x86 stuff we care about. > > > > > > * If that doesn't cover 'basic "x86 configs" in your judgement, we > > > fill the gaps as described below under "Expand from there". > > > > > > * Add an s390 runner using the project machine you mentioned. > > > > > > * Expand from there: identify the remaining gaps, map them to people / > > > organizations interested in them, and solicit contributions from these > > > guys. > > > > > > A note on contributions: we need both hardware and people. By people I > > > mean maintainers for the infrastructure, the tools and all the runners. > > > Cleber & team are willing to serve for the infrastructure, the tools and > > > the Red Hat runners. > > > > So, with 5.0 nearly out the door it seems like a good time to check > > in on this thread again to ask where we are progress-wise with this. > > My impression is that this patchset provides most of the scripting > > and config side of the first step, so what we need is for RH to provide > > an x86 runner machine and tell the gitlab CI it exists. I appreciate > > that the whole coronavirus and working-from-home situation will have > > upended everybody's plans, especially when actual hardware might > > be involved, but how's it going ? > > > > Hi Peter, > > You hit the nail in the head here. We were affected indeed with our ability > to move some machines from one lab to another (across the country), but we're > actively working on it. For x86, do we really need to be using custom runners ? With GitLab if someone forks the repo to their personal namespace, they cannot use any custom runners setup by the origin project. So if we use custom runners for x86, people forking won't be able to run the GitLab CI jobs. As a sub-system maintainer I wouldn't like this, because I ideally want to be able to run the same jobs on my staging tree, that Peter will run at merge time for the PULL request I send. Thus my strong preference would be to use the GitLab runners in every scenario where they are viable to use. Only use custom runners in the cases where GitLab runners are clearly inadequate for our needs. Based on what we've setup in GitLab for libvirt, the shared runners they have work fine for x86. Just need the environments you are testing to be provided as Docker containers (you can actually build and cache the container images during your CI job too). IOW, any Linux distro build and test jobs should be able to use shared runners on x86, and likewise mingw builds. Custom runners should only be needed if the jobs need todo *BSD / macOS builds, and/or have access to specific hardware devices for some reason. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: [PATCH 0/5] QEMU Gating CI
- Original Message - > From: "Peter Maydell" > To: "Markus Armbruster" > Cc: "Fam Zheng" , "Thomas Huth" , "Beraldo > Leal" , "Erik > Skultety" , "Alex Bennée" , > "Wainer Moschetta" , > "QEMU Developers" , "Wainer dos Santos Moschetta" > , "Willian Rampazzo" > , "Cleber Rosa" , "Philippe > Mathieu-Daudé" , "Eduardo > Habkost" > Sent: Tuesday, April 21, 2020 8:53:49 AM > Subject: Re: [PATCH 0/5] QEMU Gating CI > > On Thu, 19 Mar 2020 at 16:33, Markus Armbruster wrote: > > Peter Maydell writes: > > > I think we should start by getting the gitlab setup working > > > for the basic "x86 configs" first. Then we can try adding > > > a runner for s390 (that one's logistically easiest because > > > it is a project machine, not one owned by me personally or > > > by Linaro) once the basic framework is working, and expand > > > from there. > > > > Makes sense to me. > > > > Next steps to get this off the ground: > > > > * Red Hat provides runner(s) for x86 stuff we care about. > > > > * If that doesn't cover 'basic "x86 configs" in your judgement, we > > fill the gaps as described below under "Expand from there". > > > > * Add an s390 runner using the project machine you mentioned. > > > > * Expand from there: identify the remaining gaps, map them to people / > > organizations interested in them, and solicit contributions from these > > guys. > > > > A note on contributions: we need both hardware and people. By people I > > mean maintainers for the infrastructure, the tools and all the runners. > > Cleber & team are willing to serve for the infrastructure, the tools and > > the Red Hat runners. > > So, with 5.0 nearly out the door it seems like a good time to check > in on this thread again to ask where we are progress-wise with this. > My impression is that this patchset provides most of the scripting > and config side of the first step, so what we need is for RH to provide > an x86 runner machine and tell the gitlab CI it exists. I appreciate > that the whole coronavirus and working-from-home situation will have > upended everybody's plans, especially when actual hardware might > be involved, but how's it going ? > Hi Peter, You hit the nail in the head here. We were affected indeed with our ability to move some machines from one lab to another (across the country), but we're actively working on it. >From now on, I'll give you an update every time a significant event occurs on our side. > thanks > -- PMM > > Thanks for checking in! - Cleber.
Re: [PATCH 0/5] QEMU Gating CI
On Thu, 19 Mar 2020 at 16:33, Markus Armbruster wrote: > Peter Maydell writes: > > I think we should start by getting the gitlab setup working > > for the basic "x86 configs" first. Then we can try adding > > a runner for s390 (that one's logistically easiest because > > it is a project machine, not one owned by me personally or > > by Linaro) once the basic framework is working, and expand > > from there. > > Makes sense to me. > > Next steps to get this off the ground: > > * Red Hat provides runner(s) for x86 stuff we care about. > > * If that doesn't cover 'basic "x86 configs" in your judgement, we > fill the gaps as described below under "Expand from there". > > * Add an s390 runner using the project machine you mentioned. > > * Expand from there: identify the remaining gaps, map them to people / > organizations interested in them, and solicit contributions from these > guys. > > A note on contributions: we need both hardware and people. By people I > mean maintainers for the infrastructure, the tools and all the runners. > Cleber & team are willing to serve for the infrastructure, the tools and > the Red Hat runners. So, with 5.0 nearly out the door it seems like a good time to check in on this thread again to ask where we are progress-wise with this. My impression is that this patchset provides most of the scripting and config side of the first step, so what we need is for RH to provide an x86 runner machine and tell the gitlab CI it exists. I appreciate that the whole coronavirus and working-from-home situation will have upended everybody's plans, especially when actual hardware might be involved, but how's it going ? thanks -- PMM
Re: [PATCH 0/5] QEMU Gating CI
On Thu, Mar 19, 2020 at 05:33:01PM +0100, Markus Armbruster wrote: > Peter Maydell writes: > > > On Tue, 17 Mar 2020 at 14:13, Cleber Rosa wrote: > >> > >> On Tue, Mar 17, 2020 at 09:29:32AM +, Peter Maydell wrote: > >> > Ah, I see. My assumption was that this was all stuff that you were > >> > working on, so that I would then be able to test that it worked > >> > correctly, > >> > not that I would need to do configuration of the gitlab.com setup. > > > >> So, I had to use temporary hardware resources to set this up (and set > >> it up countless times TBH). I had the understanding based on the list > >> of machines you documented[1] that at least some of them would be used > >> for the permanent setup. > > > > Well, some of them will be (eg the s390 box), but some of them > > are my personal ones that can't be reused easily. I'd assumed > > in any case that gitlab would have at least support for x86 hosts: > > we are definitely not going to continue to use my desktop machine > > for running CI builds! Also IIRC RedHat said they'd be able to > > provide some machines for runners. While GitLab let's you run x86 code for free with the "Linux Shared Runners"[1], I don't think it would be suitable to what we're trying to achieve. It's limited to a single OS (CoreOS), single architecture and really geared towards running containers. BTW, if it isn't clear, this is the approach being used today for the jobs defined on ".gitlab-ci.yml". IMO we can leverage and still expand on the use of the "Linux Shared Runners", but to really get a grasp oh how well this model can work for QEMU, we'll need "Specific Runners", because we're validating how/if we can depend on it for OS/architectures they don't support on shared runners (and sometimes not even for the gitlab-runner agent). > > Correct! As discussed at the QEMU summit, we'll gladly chip in runners > to test the stuff we care about, but to match the coverage of your > private zoo of machines, others will have to chip in, too. > I'm sorry I missed the original discussions, and I'm even more sorry if that led to any misunderstandings here. > >> OK, I see it, now it makes more sense. So we're "only" missing the > >> setup for the machines we'll use for the more permanent setup. Would > >> you like to do a staged setup/migration using one or some of the > >> machines you documented? I'm 100% onboard to help with this, meaning > >> that I can assist you with instructions, or do "pair setup" of the > >> machines if needed. I think a good part of the evaluation here comes > >> down to how manageable/reproducible the setup is, so it'd make sense > >> for one to be part of the setup itself. > > > > I think we should start by getting the gitlab setup working > > for the basic "x86 configs" first. Then we can try adding > > a runner for s390 (that one's logistically easiest because > > it is a project machine, not one owned by me personally or > > by Linaro) once the basic framework is working, and expand > > from there. > > Makes sense to me. > > Next steps to get this off the ground: > > * Red Hat provides runner(s) for x86 stuff we care about. > > * If that doesn't cover 'basic "x86 configs" in your judgement, we > fill the gaps as described below under "Expand from there". > > * Add an s390 runner using the project machine you mentioned. > > * Expand from there: identify the remaining gaps, map them to people / > organizations interested in them, and solicit contributions from these > guys. > > A note on contributions: we need both hardware and people. By people I > mean maintainers for the infrastructure, the tools and all the runners. > Cleber & team are willing to serve for the infrastructure, the tools and > the Red Hat runners. Right, while we've tried to streamline the process of setting up the machines, there will be plenty of changes to improve the automation. More importantly, maintaining the machines is very important to the super important goal of catching code regressions only, and not facing other failures. Mundane tasks such as making sure enough disk space is always available can be completely change the perception of the usefulness of a CI environment. And for this maintenance, we need help from people "owning" those machines. > > Does this sound workable? > > > But to a large degree I really don't want to have to get > > into the details of how gitlab works or setting up runners > > myself if I can avoid it. We're going through this migration > > because I want to be able to hand off the CI stuff to other > > people, not to retain control of it. > > Understand. We need contributions to gating CI, but the whole point of > this exercise is to make people other than *you* contribute to our > gating CI :) > > Let me use this opportunity to say thank you for all your integration > work! > > ^ THIS. I have to say that I'm still amazed as to how Peter has managed to automate, integrate and run all those tests in
Re: [PATCH 0/5] QEMU Gating CI
Peter Maydell writes: > On Tue, 17 Mar 2020 at 14:13, Cleber Rosa wrote: >> >> On Tue, Mar 17, 2020 at 09:29:32AM +, Peter Maydell wrote: >> > Ah, I see. My assumption was that this was all stuff that you were >> > working on, so that I would then be able to test that it worked correctly, >> > not that I would need to do configuration of the gitlab.com setup. > >> So, I had to use temporary hardware resources to set this up (and set >> it up countless times TBH). I had the understanding based on the list >> of machines you documented[1] that at least some of them would be used >> for the permanent setup. > > Well, some of them will be (eg the s390 box), but some of them > are my personal ones that can't be reused easily. I'd assumed > in any case that gitlab would have at least support for x86 hosts: > we are definitely not going to continue to use my desktop machine > for running CI builds! Also IIRC RedHat said they'd be able to > provide some machines for runners. Correct! As discussed at the QEMU summit, we'll gladly chip in runners to test the stuff we care about, but to match the coverage of your private zoo of machines, others will have to chip in, too. >> OK, I see it, now it makes more sense. So we're "only" missing the >> setup for the machines we'll use for the more permanent setup. Would >> you like to do a staged setup/migration using one or some of the >> machines you documented? I'm 100% onboard to help with this, meaning >> that I can assist you with instructions, or do "pair setup" of the >> machines if needed. I think a good part of the evaluation here comes >> down to how manageable/reproducible the setup is, so it'd make sense >> for one to be part of the setup itself. > > I think we should start by getting the gitlab setup working > for the basic "x86 configs" first. Then we can try adding > a runner for s390 (that one's logistically easiest because > it is a project machine, not one owned by me personally or > by Linaro) once the basic framework is working, and expand > from there. Makes sense to me. Next steps to get this off the ground: * Red Hat provides runner(s) for x86 stuff we care about. * If that doesn't cover 'basic "x86 configs" in your judgement, we fill the gaps as described below under "Expand from there". * Add an s390 runner using the project machine you mentioned. * Expand from there: identify the remaining gaps, map them to people / organizations interested in them, and solicit contributions from these guys. A note on contributions: we need both hardware and people. By people I mean maintainers for the infrastructure, the tools and all the runners. Cleber & team are willing to serve for the infrastructure, the tools and the Red Hat runners. Does this sound workable? > But to a large degree I really don't want to have to get > into the details of how gitlab works or setting up runners > myself if I can avoid it. We're going through this migration > because I want to be able to hand off the CI stuff to other > people, not to retain control of it. Understand. We need contributions to gating CI, but the whole point of this exercise is to make people other than *you* contribute to our gating CI :) Let me use this opportunity to say thank you for all your integration work!
Re: [PATCH 0/5] QEMU Gating CI
On Tue, 17 Mar 2020 at 14:13, Cleber Rosa wrote: > > On Tue, Mar 17, 2020 at 09:29:32AM +, Peter Maydell wrote: > > Ah, I see. My assumption was that this was all stuff that you were > > working on, so that I would then be able to test that it worked correctly, > > not that I would need to do configuration of the gitlab.com setup. > So, I had to use temporary hardware resources to set this up (and set > it up countless times TBH). I had the understanding based on the list > of machines you documented[1] that at least some of them would be used > for the permanent setup. Well, some of them will be (eg the s390 box), but some of them are my personal ones that can't be reused easily. I'd assumed in any case that gitlab would have at least support for x86 hosts: we are definitely not going to continue to use my desktop machine for running CI builds! Also IIRC RedHat said they'd be able to provide some machines for runners. > OK, I see it, now it makes more sense. So we're "only" missing the > setup for the machines we'll use for the more permanent setup. Would > you like to do a staged setup/migration using one or some of the > machines you documented? I'm 100% onboard to help with this, meaning > that I can assist you with instructions, or do "pair setup" of the > machines if needed. I think a good part of the evaluation here comes > down to how manageable/reproducible the setup is, so it'd make sense > for one to be part of the setup itself. I think we should start by getting the gitlab setup working for the basic "x86 configs" first. Then we can try adding a runner for s390 (that one's logistically easiest because it is a project machine, not one owned by me personally or by Linaro) once the basic framework is working, and expand from there. But to a large degree I really don't want to have to get into the details of how gitlab works or setting up runners myself if I can avoid it. We're going through this migration because I want to be able to hand off the CI stuff to other people, not to retain control of it. thanks -- PMM
Re: [PATCH 0/5] QEMU Gating CI
On Tue, Mar 17, 2020 at 09:29:32AM +, Peter Maydell wrote: > On Tue, 17 Mar 2020 at 04:59, Cleber Rosa wrote: > > Yes, that did the trick and I can now see the configuration. What I can > > *not* see is any "Specific Runner" configured. So maybe: > > > > 1) The documentation I included is not clear enough about the fact that > > setup steps need to be done on a machine so that it becomes a "Runner" > > > > 2) The (Ansible) playbooks (especially > > contrib/ci/orgs/qemu/gitlab-runner.yml) > > is not working as intended > > > > 3) Some expectations misalignment on machines that would be available to run > > those jobs > > > > In any case, none of those should be big problems. Please let me know what > > you did/experienced/expected up to this point, and we can continue from > > there. > > Ah, I see. My assumption was that this was all stuff that you were > working on, so that I would then be able to test that it worked correctly, > not that I would need to do configuration of the gitlab.com setup. Hi Peter, So, I had to use temporary hardware resources to set this up (and set it up countless times TBH). I had the understanding based on the list of machines you documented[1] that at least some of them would be used for the permanent setup. > I thought all the stuff about "how to set up runners" was only for > people who wanted to set up some 3rd-party CI for non-official > forks or for when we wanted to add extra runners in future (eg for > architectures not yet covered). So the only thing I did was follow > your "just push to staging" instructions. > OK, I see it, now it makes more sense. So we're "only" missing the setup for the machines we'll use for the more permanent setup. Would you like to do a staged setup/migration using one or some of the machines you documented? I'm 100% onboard to help with this, meaning that I can assist you with instructions, or do "pair setup" of the machines if needed. I think a good part of the evaluation here comes down to how manageable/reproducible the setup is, so it'd make sense for one to be part of the setup itself. FIY there's also the possibility of grabbing some free VMs on GCP, Azure, etc and setting them up as GitLab runners in a temporary way (because of the temporarily free and VM nature). I have a few problems with this approach, including the fact that it doesn't yield the complete experience wrt using hardware one owns and will have to manage, besides the hardware limitations themselves. Please let me know how you want to move on from here. Cheers, - Cleber. > thanks > -- PMM > [1] https://wiki.qemu.org/Requirements/GatingCI#Current_Tests signature.asc Description: PGP signature
Re: [PATCH 0/5] QEMU Gating CI
On Tue, 17 Mar 2020 at 04:59, Cleber Rosa wrote: > Yes, that did the trick and I can now see the configuration. What I can > *not* see is any "Specific Runner" configured. So maybe: > > 1) The documentation I included is not clear enough about the fact that > setup steps need to be done on a machine so that it becomes a "Runner" > > 2) The (Ansible) playbooks (especially contrib/ci/orgs/qemu/gitlab-runner.yml) > is not working as intended > > 3) Some expectations misalignment on machines that would be available to run > those jobs > > In any case, none of those should be big problems. Please let me know what > you did/experienced/expected up to this point, and we can continue from there. Ah, I see. My assumption was that this was all stuff that you were working on, so that I would then be able to test that it worked correctly, not that I would need to do configuration of the gitlab.com setup. I thought all the stuff about "how to set up runners" was only for people who wanted to set up some 3rd-party CI for non-official forks or for when we wanted to add extra runners in future (eg for architectures not yet covered). So the only thing I did was follow your "just push to staging" instructions. thanks -- PMM
Re: [PATCH 0/5] QEMU Gating CI
- Original Message - > From: "Peter Maydell" > To: "Cleber Rosa" > Cc: "Fam Zheng" , "Thomas Huth" , "Beraldo > Leal" , "Erik > Skultety" , "Alex Bennée" , > "Wainer Moschetta" , > "QEMU Developers" , "Wainer dos Santos Moschetta" > , "Willian Rampazzo" > , "Philippe Mathieu-Daudé" , "Eduardo > Habkost" > Sent: Monday, March 16, 2020 10:57:30 AM > Subject: Re: [PATCH 0/5] QEMU Gating CI > > On Mon, 16 Mar 2020 at 12:26, Cleber Rosa wrote: > > About the runners and the fact that the job is stuck without them, > > the message seems straightforward enough, but I can't get to the > > project configuration to look at the registered runners with my > > current permissions (set as "developer"). > > I've moved you up to 'maintainer' status, hopefully that is > sufficient to look at the relevant config ? > > thanks > -- PMM > > Hi Peter, Yes, that did the trick and I can now see the configuration. What I can *not* see is any "Specific Runner" configured. So maybe: 1) The documentation I included is not clear enough about the fact that setup steps need to be done on a machine so that it becomes a "Runner" 2) The (Ansible) playbooks (especially contrib/ci/orgs/qemu/gitlab-runner.yml) is not working as intended 3) Some expectations misalignment on machines that would be available to run those jobs In any case, none of those should be big problems. Please let me know what you did/experienced/expected up to this point, and we can continue from there. Regards, - Cleber.
Re: [PATCH 0/5] QEMU Gating CI
On Mon, Mar 16, 2020 at 4:24 PM Alex Bennée wrote: > > > Daniel P. Berrangé writes: > > > On Thu, Mar 12, 2020 at 03:36:11PM -0400, Cleber Rosa wrote: > >> The idea about a public facing Gating CI for QEMU was lastly > >> summarized in an RFC[1]. Since then, it was decided that a > >> simpler version should be attempted first. > >> > >> Changes from the RFC patches[2] accompanying the RFC document: > >> > >> - Moved gating job definitions to .gitlab-ci-gating.yml > >> - Added info on "--disable-libssh" build option requirement > >> (https://bugs.launchpad.net/qemu/+bug/1838763) to Ubuntu 18.04 jobs > >> - Added info on "--disable-glusterfs" build option requirement > >> (there's no static version of those libs in distro supplied > >> packages) to one > >> - Dropped ubuntu-18.04.3-x86_64-notools job definition, because it > >> doesn't fall into the general scope of gating job described by PMM > >> (and it did not run any test) > >> - Added w32 and w64 cross builds based on Fedora 30 > >> - Added a FreeBSD based job that builds all targets and runs `make > >> check` > >> - Added "-j`nproc`" and "-j`sysctl -n hw.ncpu`" options to make as a > >> simple but effective way of speeding up the builds and tests by > >> using a number of make jobs matching the number of CPUs > >> - Because the Ansible playbooks reference the content on Dockerfiles, > >> some fixes to some Dockerfiles caught in the process were included > >> - New patch with script to check or wait on a pipeline execution > >> > >> [1] - https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg00231.html > >> [2] - https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg00154.html > >> > >> Cleber Rosa (5): > >> tests/docker: add CentOS 8 Dockerfile > >> tests/docker: make "buildah bud" output similar to "docker build" > >> GitLab CI: avoid calling before_scripts on unintended jobs > >> GitLab Gating CI: introduce pipeline-status contrib script > >> GitLab Gating CI: initial set of jobs, documentation and scripts > >> > >> .gitlab-ci-gating.yml | 111 ++ > >> .gitlab-ci.yml| 32 ++- > >> contrib/ci/orgs/qemu/build-environment.yml| 208 ++ > >> contrib/ci/orgs/qemu/gitlab-runner.yml| 65 ++ > >> contrib/ci/orgs/qemu/inventory| 2 + > >> contrib/ci/orgs/qemu/vars.yml | 13 ++ > >> contrib/ci/scripts/gitlab-pipeline-status | 148 + > > > > FYI, the contrib/ directory is generally a place for arbitrary / adhoc > > but interesting user contributed files/sources that are not officially > > supported deliverables of the project. > > > > IOW, this is not a good home for the official CI scripts. > > > > We already have a .gitlab-ci.d/ directory that looks like it would > > be good for this. Or if that's not suitable, then scripts/ci/ is > > a second choice. > > I'd vote for scripts/ci/ or scripts/gitlab/ as the .gitlab-ci.d might be > a little hidden. > I vote for scripts/ci/ or scripts/gitlab/ too. With a little preference to scripts/ci/. Aleksandar > -- > Alex Bennée >
Re: [PATCH 0/5] QEMU Gating CI
On Mon, 16 Mar 2020 at 12:26, Cleber Rosa wrote: > About the runners and the fact that the job is stuck without them, > the message seems straightforward enough, but I can't get to the > project configuration to look at the registered runners with my > current permissions (set as "developer"). I've moved you up to 'maintainer' status, hopefully that is sufficient to look at the relevant config ? thanks -- PMM
Re: [PATCH 0/5] QEMU Gating CI
Daniel P. Berrangé writes: > On Thu, Mar 12, 2020 at 03:36:11PM -0400, Cleber Rosa wrote: >> The idea about a public facing Gating CI for QEMU was lastly >> summarized in an RFC[1]. Since then, it was decided that a >> simpler version should be attempted first. >> >> Changes from the RFC patches[2] accompanying the RFC document: >> >> - Moved gating job definitions to .gitlab-ci-gating.yml >> - Added info on "--disable-libssh" build option requirement >> (https://bugs.launchpad.net/qemu/+bug/1838763) to Ubuntu 18.04 jobs >> - Added info on "--disable-glusterfs" build option requirement >> (there's no static version of those libs in distro supplied >> packages) to one >> - Dropped ubuntu-18.04.3-x86_64-notools job definition, because it >> doesn't fall into the general scope of gating job described by PMM >> (and it did not run any test) >> - Added w32 and w64 cross builds based on Fedora 30 >> - Added a FreeBSD based job that builds all targets and runs `make >> check` >> - Added "-j`nproc`" and "-j`sysctl -n hw.ncpu`" options to make as a >> simple but effective way of speeding up the builds and tests by >> using a number of make jobs matching the number of CPUs >> - Because the Ansible playbooks reference the content on Dockerfiles, >> some fixes to some Dockerfiles caught in the process were included >> - New patch with script to check or wait on a pipeline execution >> >> [1] - https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg00231.html >> [2] - https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg00154.html >> >> Cleber Rosa (5): >> tests/docker: add CentOS 8 Dockerfile >> tests/docker: make "buildah bud" output similar to "docker build" >> GitLab CI: avoid calling before_scripts on unintended jobs >> GitLab Gating CI: introduce pipeline-status contrib script >> GitLab Gating CI: initial set of jobs, documentation and scripts >> >> .gitlab-ci-gating.yml | 111 ++ >> .gitlab-ci.yml| 32 ++- >> contrib/ci/orgs/qemu/build-environment.yml| 208 ++ >> contrib/ci/orgs/qemu/gitlab-runner.yml| 65 ++ >> contrib/ci/orgs/qemu/inventory| 2 + >> contrib/ci/orgs/qemu/vars.yml | 13 ++ >> contrib/ci/scripts/gitlab-pipeline-status | 148 + > > FYI, the contrib/ directory is generally a place for arbitrary / adhoc > but interesting user contributed files/sources that are not officially > supported deliverables of the project. > > IOW, this is not a good home for the official CI scripts. > > We already have a .gitlab-ci.d/ directory that looks like it would > be good for this. Or if that's not suitable, then scripts/ci/ is > a second choice. I'd vote for scripts/ci/ or scripts/gitlab/ as the .gitlab-ci.d might be a little hidden. -- Alex Bennée
Re: [PATCH 0/5] QEMU Gating CI
- Original Message - > From: "Peter Maydell" > To: "Cleber Rosa" > Cc: "Alex Bennée" , "QEMU Developers" > , "Fam Zheng" , > "Eduardo Habkost" , "Beraldo Leal" , > "Philippe Mathieu-Daudé" > , "Thomas Huth" , "Wainer dos Santos > Moschetta" , "Erik > Skultety" , "Willian Rampazzo" , > "Wainer Moschetta" > Sent: Monday, March 16, 2020 8:12:16 AM > Subject: Re: [PATCH 0/5] QEMU Gating CI > > On Mon, 16 Mar 2020 at 12:04, Cleber Rosa wrote: > > A few possible reasons come to my mind: > > > > 1) It usually takes a few seconds after the push for the pipeline to > > > > 2) If you've pushed to a repo different than gitlab.com/qemu-project/qemu, > >you'd have to tweak the project ID (-p|--project-id). > > > > 3) The local branch is not called "staging", so the script can not find the > >commit ID, in that case you can use -c|--commit. > > Yes, the local branch is something else for the purposes of > testing this series. But using --commit doesn't work either: > > $ ./contrib/ci/scripts/gitlab-pipeline-status --verbose --commit > 81beaaab0851fe8c4db971 --wait > ERROR: No pipeline found > failure > This may be a bug in the script because I was not expecting two pipelines for the same commit. Checking... > On the web UI: > https://gitlab.com/qemu-project/qemu/pipelines > the pipelines are marked "stuck" (I don't know why there > are two of them for the same commit); drilling down, > the build part has completed but all the test parts are > pending with "This job is stuck because you don't have > any active runners online with any of these tags assigned > to them" type messages. > I had also not come across the duplicate pipelines, so I'm trying to understand what's that about. About the runners and the fact that the job is stuck without them, the message seems straightforward enough, but I can't get to the project configuration to look at the registered runners with my current permissions (set as "developer"). > thanks > -- PMM > > Thanks, - Cleber.
Re: [PATCH 0/5] QEMU Gating CI
- Original Message - > From: "Daniel P. Berrangé" > To: "Cleber Rosa" > Cc: "Alex Bennée" , "Peter Maydell" > , qemu-devel@nongnu.org, "Fam > Zheng" , "Thomas Huth" , "Eduardo Habkost" > , "Erik Skultety" > , "Wainer Moschetta" , "Wainer dos > Santos Moschetta" > , "Willian Rampazzo" , "Philippe > Mathieu-Daudé" , > "Beraldo Leal" > Sent: Monday, March 16, 2020 8:38:07 AM > Subject: Re: [PATCH 0/5] QEMU Gating CI > > On Thu, Mar 12, 2020 at 03:36:11PM -0400, Cleber Rosa wrote: > > The idea about a public facing Gating CI for QEMU was lastly > > summarized in an RFC[1]. Since then, it was decided that a > > simpler version should be attempted first. > > > > Changes from the RFC patches[2] accompanying the RFC document: > > > > - Moved gating job definitions to .gitlab-ci-gating.yml > > - Added info on "--disable-libssh" build option requirement > > (https://bugs.launchpad.net/qemu/+bug/1838763) to Ubuntu 18.04 jobs > > - Added info on "--disable-glusterfs" build option requirement > > (there's no static version of those libs in distro supplied > > packages) to one > > - Dropped ubuntu-18.04.3-x86_64-notools job definition, because it > > doesn't fall into the general scope of gating job described by PMM > > (and it did not run any test) > > - Added w32 and w64 cross builds based on Fedora 30 > > - Added a FreeBSD based job that builds all targets and runs `make > > check` > > - Added "-j`nproc`" and "-j`sysctl -n hw.ncpu`" options to make as a > > simple but effective way of speeding up the builds and tests by > > using a number of make jobs matching the number of CPUs > > - Because the Ansible playbooks reference the content on Dockerfiles, > > some fixes to some Dockerfiles caught in the process were included > > - New patch with script to check or wait on a pipeline execution > > > > [1] - https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg00231.html > > [2] - https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg00154.html > > > > Cleber Rosa (5): > > tests/docker: add CentOS 8 Dockerfile > > tests/docker: make "buildah bud" output similar to "docker build" > > GitLab CI: avoid calling before_scripts on unintended jobs > > GitLab Gating CI: introduce pipeline-status contrib script > > GitLab Gating CI: initial set of jobs, documentation and scripts > > > > .gitlab-ci-gating.yml | 111 ++ > > .gitlab-ci.yml| 32 ++- > > contrib/ci/orgs/qemu/build-environment.yml| 208 ++ > > contrib/ci/orgs/qemu/gitlab-runner.yml| 65 ++ > > contrib/ci/orgs/qemu/inventory| 2 + > > contrib/ci/orgs/qemu/vars.yml | 13 ++ > > contrib/ci/scripts/gitlab-pipeline-status | 148 + > > FYI, the contrib/ directory is generally a place for arbitrary / adhoc > but interesting user contributed files/sources that are not officially > supported deliverables of the project. > > IOW, this is not a good home for the official CI scripts. > Good point, reason is that I wasn't/ain't sure this script is going to be "official" and really that useful. I'm happy to move it somewhere else though. > We already have a .gitlab-ci.d/ directory that looks like it would > be good for this. Or if that's not suitable, then scripts/ci/ is > a second choice. > Ack. Thanks! - Cleber. > > docs/devel/testing.rst| 142 > > tests/docker/dockerfiles/centos8.docker | 32 +++ > > .../dockerfiles/debian-win32-cross.docker | 2 +- > > 10 files changed, 751 insertions(+), 4 deletions(-) > > create mode 100644 .gitlab-ci-gating.yml > > create mode 100644 contrib/ci/orgs/qemu/build-environment.yml > > create mode 100644 contrib/ci/orgs/qemu/gitlab-runner.yml > > create mode 100644 contrib/ci/orgs/qemu/inventory > > create mode 100644 contrib/ci/orgs/qemu/vars.yml > > create mode 100755 contrib/ci/scripts/gitlab-pipeline-status > > create mode 100644 tests/docker/dockerfiles/centos8.docker > > Regards, > Daniel > -- > |: https://berrange.com -o-https://www.flickr.com/photos/dberrange > |:| > |: https://libvirt.org -o-https://fstop138.berrange.com > |:| > |: https://entangle-photo.org-o-https://www.instagram.com/dberrange > |:| >
Re: [PATCH 0/5] QEMU Gating CI
On Thu, Mar 12, 2020 at 03:36:11PM -0400, Cleber Rosa wrote: > The idea about a public facing Gating CI for QEMU was lastly > summarized in an RFC[1]. Since then, it was decided that a > simpler version should be attempted first. > > Changes from the RFC patches[2] accompanying the RFC document: > > - Moved gating job definitions to .gitlab-ci-gating.yml > - Added info on "--disable-libssh" build option requirement > (https://bugs.launchpad.net/qemu/+bug/1838763) to Ubuntu 18.04 jobs > - Added info on "--disable-glusterfs" build option requirement > (there's no static version of those libs in distro supplied > packages) to one > - Dropped ubuntu-18.04.3-x86_64-notools job definition, because it > doesn't fall into the general scope of gating job described by PMM > (and it did not run any test) > - Added w32 and w64 cross builds based on Fedora 30 > - Added a FreeBSD based job that builds all targets and runs `make > check` > - Added "-j`nproc`" and "-j`sysctl -n hw.ncpu`" options to make as a > simple but effective way of speeding up the builds and tests by > using a number of make jobs matching the number of CPUs > - Because the Ansible playbooks reference the content on Dockerfiles, > some fixes to some Dockerfiles caught in the process were included > - New patch with script to check or wait on a pipeline execution > > [1] - https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg00231.html > [2] - https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg00154.html > > Cleber Rosa (5): > tests/docker: add CentOS 8 Dockerfile > tests/docker: make "buildah bud" output similar to "docker build" > GitLab CI: avoid calling before_scripts on unintended jobs > GitLab Gating CI: introduce pipeline-status contrib script > GitLab Gating CI: initial set of jobs, documentation and scripts > > .gitlab-ci-gating.yml | 111 ++ > .gitlab-ci.yml| 32 ++- > contrib/ci/orgs/qemu/build-environment.yml| 208 ++ > contrib/ci/orgs/qemu/gitlab-runner.yml| 65 ++ > contrib/ci/orgs/qemu/inventory| 2 + > contrib/ci/orgs/qemu/vars.yml | 13 ++ > contrib/ci/scripts/gitlab-pipeline-status | 148 + FYI, the contrib/ directory is generally a place for arbitrary / adhoc but interesting user contributed files/sources that are not officially supported deliverables of the project. IOW, this is not a good home for the official CI scripts. We already have a .gitlab-ci.d/ directory that looks like it would be good for this. Or if that's not suitable, then scripts/ci/ is a second choice. > docs/devel/testing.rst| 142 > tests/docker/dockerfiles/centos8.docker | 32 +++ > .../dockerfiles/debian-win32-cross.docker | 2 +- > 10 files changed, 751 insertions(+), 4 deletions(-) > create mode 100644 .gitlab-ci-gating.yml > create mode 100644 contrib/ci/orgs/qemu/build-environment.yml > create mode 100644 contrib/ci/orgs/qemu/gitlab-runner.yml > create mode 100644 contrib/ci/orgs/qemu/inventory > create mode 100644 contrib/ci/orgs/qemu/vars.yml > create mode 100755 contrib/ci/scripts/gitlab-pipeline-status > create mode 100644 tests/docker/dockerfiles/centos8.docker Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: [PATCH 0/5] QEMU Gating CI
On Mon, 16 Mar 2020 at 12:04, Cleber Rosa wrote: > A few possible reasons come to my mind: > > 1) It usually takes a few seconds after the push for the pipeline to > > 2) If you've pushed to a repo different than gitlab.com/qemu-project/qemu, >you'd have to tweak the project ID (-p|--project-id). > > 3) The local branch is not called "staging", so the script can not find the >commit ID, in that case you can use -c|--commit. Yes, the local branch is something else for the purposes of testing this series. But using --commit doesn't work either: $ ./contrib/ci/scripts/gitlab-pipeline-status --verbose --commit 81beaaab0851fe8c4db971 --wait ERROR: No pipeline found failure On the web UI: https://gitlab.com/qemu-project/qemu/pipelines the pipelines are marked "stuck" (I don't know why there are two of them for the same commit); drilling down, the build part has completed but all the test parts are pending with "This job is stuck because you don't have any active runners online with any of these tags assigned to them" type messages. thanks -- PMM
Re: [PATCH 0/5] QEMU Gating CI
- Original Message - > From: "Cleber Rosa" > To: "Peter Maydell" > Cc: "Alex Bennée" , "QEMU Developers" > , "Fam Zheng" , > "Eduardo Habkost" , "Beraldo Leal" , > "Philippe Mathieu-Daudé" > , "Thomas Huth" , "Wainer dos Santos > Moschetta" , "Erik > Skultety" , "Willian Rampazzo" , > "Wainer Moschetta" > Sent: Monday, March 16, 2020 8:26:46 AM > Subject: Re: [PATCH 0/5] QEMU Gating CI > > > > - Original Message - > > From: "Peter Maydell" > > To: "Cleber Rosa" > > Cc: "Alex Bennée" , "QEMU Developers" > > , "Fam Zheng" , > > "Eduardo Habkost" , "Beraldo Leal" , > > "Philippe Mathieu-Daudé" > > , "Thomas Huth" , "Wainer dos Santos > > Moschetta" , "Erik > > Skultety" , "Willian Rampazzo" , > > "Wainer Moschetta" > > Sent: Monday, March 16, 2020 8:12:16 AM > > Subject: Re: [PATCH 0/5] QEMU Gating CI > > > > On Mon, 16 Mar 2020 at 12:04, Cleber Rosa wrote: > > > A few possible reasons come to my mind: > > > > > > 1) It usually takes a few seconds after the push for the pipeline to > > > > > > 2) If you've pushed to a repo different than > > > gitlab.com/qemu-project/qemu, > > >you'd have to tweak the project ID (-p|--project-id). > > > > > > 3) The local branch is not called "staging", so the script can not find > > > the > > >commit ID, in that case you can use -c|--commit. > > > > Yes, the local branch is something else for the purposes of > > testing this series. But using --commit doesn't work either: > > > > $ ./contrib/ci/scripts/gitlab-pipeline-status --verbose --commit > > 81beaaab0851fe8c4db971 --wait > > ERROR: No pipeline found > > failure > > > > This may be a bug in the script because I was not expecting two > pipelines for the same commit. Checking... > Looks like the GitLab API requires a 40 char commit ID, so this should work: $ ./contrib/ci/scripts/gitlab-pipeline-status --verbose --commit 81beaaab0851fe8c4db971df555600152bb83a6c --wait It'll stay silent at first, but then will print a message every 60 seconds. Thanks, - Cleber.
Re: [PATCH 0/5] QEMU Gating CI
On Thu, 12 Mar 2020 at 22:16, Cleber Rosa wrote: > The quick answer is: > > $ git push g...@gitlab.com:qemu-project/qemu.git my-branch:staging So I did this bit... > Once that push happens, you could use: > > $ contrib/ci/scripts/gitlab-pipeline-status --verbose --wait ...but this script just says: $ ./contrib/ci/scripts/gitlab-pipeline-status --verbose --wait ERROR: No pipeline found failure thanks -- PMM
Re: [PATCH 0/5] QEMU Gating CI
- Original Message - > From: "Peter Maydell" > To: "Cleber Rosa" > Cc: "Alex Bennée" , "QEMU Developers" > , "Fam Zheng" , > "Eduardo Habkost" , "Beraldo Leal" , > "Philippe Mathieu-Daudé" > , "Thomas Huth" , "Wainer dos Santos > Moschetta" , "Erik > Skultety" , "Willian Rampazzo" , > "Wainer Moschetta" > Sent: Monday, March 16, 2020 7:57:33 AM > Subject: Re: [PATCH 0/5] QEMU Gating CI > > On Thu, 12 Mar 2020 at 22:16, Cleber Rosa wrote: > > The quick answer is: > > > > $ git push g...@gitlab.com:qemu-project/qemu.git my-branch:staging > > So I did this bit... > > > Once that push happens, you could use: > > > > $ contrib/ci/scripts/gitlab-pipeline-status --verbose --wait > > ...but this script just says: > > $ ./contrib/ci/scripts/gitlab-pipeline-status --verbose --wait > ERROR: No pipeline found > failure > Hi Peter, A few possible reasons come to my mind: 1) It usually takes a few seconds after the push for the pipeline to 2) If you've pushed to a repo different than gitlab.com/qemu-project/qemu, you'd have to tweak the project ID (-p|--project-id). 3) The local branch is not called "staging", so the script can not find the commit ID, in that case you can use -c|--commit. > thanks > -- PMM > > Please let me know if any of these points helps. Cheers, - Cleber.
Re: [PATCH 0/5] QEMU Gating CI
On Fri, Mar 13, 2020 at 01:55:49PM +, Peter Maydell wrote: > On Thu, 12 Mar 2020 at 22:16, Cleber Rosa wrote: > > > > On Thu, Mar 12, 2020 at 10:00:42PM +, Peter Maydell wrote: > > > OK, so my question here is: > > > * what are the instructions that I have to follow to be > > > able to say "ok, here's my branch, run it through these tests, > > > please" ? > > > > The quick answer is: > > > > $ git push g...@gitlab.com:qemu-project/qemu.git my-branch:staging > > > > The longer explanation is that these jobs are limited to a "staging" > > branch, so all you'd have to do is to push something to a branch > > called "staging". If that branch happens to be from the > > "gitlab.com/qemu-project/qemu" repo, than the runners setup there > > would be used. The documentation an ansible playbooks are supposed > > to help with this setup. > > Great, thanks. Could I do that for testing purposes with a > staging branch that includes these patches, or would we have > to wait for them to be in master before it works? > You can definitely do that with a staging branch that includes these patches. And with regards to setting up machines with runners and other requirements, I tried to make it easy to replicate, but YMMV. So, please let me know if you find any issues whatsoever. > -- PMM > Thanks, - Cleber. signature.asc Description: PGP signature
Re: [PATCH 0/5] QEMU Gating CI
On Thu, 12 Mar 2020 at 22:16, Cleber Rosa wrote: > > On Thu, Mar 12, 2020 at 10:00:42PM +, Peter Maydell wrote: > > OK, so my question here is: > > * what are the instructions that I have to follow to be > > able to say "ok, here's my branch, run it through these tests, > > please" ? > > The quick answer is: > > $ git push g...@gitlab.com:qemu-project/qemu.git my-branch:staging > > The longer explanation is that these jobs are limited to a "staging" > branch, so all you'd have to do is to push something to a branch > called "staging". If that branch happens to be from the > "gitlab.com/qemu-project/qemu" repo, than the runners setup there > would be used. The documentation an ansible playbooks are supposed > to help with this setup. Great, thanks. Could I do that for testing purposes with a staging branch that includes these patches, or would we have to wait for them to be in master before it works? -- PMM
Re: [PATCH 0/5] QEMU Gating CI
On Thu, Mar 12, 2020 at 10:00:42PM +, Peter Maydell wrote: > On Thu, 12 Mar 2020 at 19:36, Cleber Rosa wrote: > > > > The idea about a public facing Gating CI for QEMU was lastly > > summarized in an RFC[1]. Since then, it was decided that a > > simpler version should be attempted first. > > OK, so my question here is: > * what are the instructions that I have to follow to be > able to say "ok, here's my branch, run it through these tests, > please" ? The quick answer is: $ git push g...@gitlab.com:qemu-project/qemu.git my-branch:staging The longer explanation is that these jobs are limited to a "staging" branch, so all you'd have to do is to push something to a branch called "staging". If that branch happens to be from the "gitlab.com/qemu-project/qemu" repo, than the runners setup there would be used. The documentation an ansible playbooks are supposed to help with this setup. Once that push happens, you could use: $ contrib/ci/scripts/gitlab-pipeline-status --verbose --wait Before doing something like: $ git push g...@gitlab.com:qemu-project/qemu.git my-branch:master > > thanks > -- PMM > Let me know if that makes sense. Cheers, - Cleber. signature.asc Description: PGP signature
Re: [PATCH 0/5] QEMU Gating CI
On Thu, 12 Mar 2020 at 19:36, Cleber Rosa wrote: > > The idea about a public facing Gating CI for QEMU was lastly > summarized in an RFC[1]. Since then, it was decided that a > simpler version should be attempted first. OK, so my question here is: * what are the instructions that I have to follow to be able to say "ok, here's my branch, run it through these tests, please" ? thanks -- PMM
[PATCH 0/5] QEMU Gating CI
The idea about a public facing Gating CI for QEMU was lastly summarized in an RFC[1]. Since then, it was decided that a simpler version should be attempted first. Changes from the RFC patches[2] accompanying the RFC document: - Moved gating job definitions to .gitlab-ci-gating.yml - Added info on "--disable-libssh" build option requirement (https://bugs.launchpad.net/qemu/+bug/1838763) to Ubuntu 18.04 jobs - Added info on "--disable-glusterfs" build option requirement (there's no static version of those libs in distro supplied packages) to one - Dropped ubuntu-18.04.3-x86_64-notools job definition, because it doesn't fall into the general scope of gating job described by PMM (and it did not run any test) - Added w32 and w64 cross builds based on Fedora 30 - Added a FreeBSD based job that builds all targets and runs `make check` - Added "-j`nproc`" and "-j`sysctl -n hw.ncpu`" options to make as a simple but effective way of speeding up the builds and tests by using a number of make jobs matching the number of CPUs - Because the Ansible playbooks reference the content on Dockerfiles, some fixes to some Dockerfiles caught in the process were included - New patch with script to check or wait on a pipeline execution [1] - https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg00231.html [2] - https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg00154.html Cleber Rosa (5): tests/docker: add CentOS 8 Dockerfile tests/docker: make "buildah bud" output similar to "docker build" GitLab CI: avoid calling before_scripts on unintended jobs GitLab Gating CI: introduce pipeline-status contrib script GitLab Gating CI: initial set of jobs, documentation and scripts .gitlab-ci-gating.yml | 111 ++ .gitlab-ci.yml| 32 ++- contrib/ci/orgs/qemu/build-environment.yml| 208 ++ contrib/ci/orgs/qemu/gitlab-runner.yml| 65 ++ contrib/ci/orgs/qemu/inventory| 2 + contrib/ci/orgs/qemu/vars.yml | 13 ++ contrib/ci/scripts/gitlab-pipeline-status | 148 + docs/devel/testing.rst| 142 tests/docker/dockerfiles/centos8.docker | 32 +++ .../dockerfiles/debian-win32-cross.docker | 2 +- 10 files changed, 751 insertions(+), 4 deletions(-) create mode 100644 .gitlab-ci-gating.yml create mode 100644 contrib/ci/orgs/qemu/build-environment.yml create mode 100644 contrib/ci/orgs/qemu/gitlab-runner.yml create mode 100644 contrib/ci/orgs/qemu/inventory create mode 100644 contrib/ci/orgs/qemu/vars.yml create mode 100755 contrib/ci/scripts/gitlab-pipeline-status create mode 100644 tests/docker/dockerfiles/centos8.docker -- 2.24.1
