Re: Fedora CI for large RPMs?
Hi, Regarding Zuul, your are right the cleanup is not configured and canceled jobs (eg. due to a PR being rebased/updated/closed) don't trigger a cancel scratch build on Koji. However Zuul provides a hook for this [1], then that would be a matter of adding the proper playbook to run in the job definition for the scratch build cleanup [2]. Fabien [1] https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.cleanup-run [2] https://pagure.io/fedora-zuul-jobs-config/blob/master/f/zuul.d/jobs.yaml#_81 On Wed, Nov 13, 2024 at 11:37 AM Nikita Popov wrote: > Hi! > > The llvm rpm (https://src.fedoraproject.org/rpms/llvm) has recently been > struggling with Fedora CI, by which I mean the CI that produces scratch > builds and runs dist-git tests on merge requests. > > The llvm rpm has a combination of long build times (typically 3-8 hours on > koji, depending on arch) combined with many merge requests, which is where > things break down. We've received multiple complaints that scratch builds > for our MRs occasionally end up clogging s390x koji, due to a number of > problems with Fedora CI. > > * Both Zuul and Fedora CI produce their own independent scratch builds, > increasing load by 2x. I think this is tracked as part of > https://pagure.io/fedora-ci/general/issue/476. This is the only problem > we were able to address ourselves, by disabling Zuul. > > * Fedora CI does not cancel old scratch builds when a new commit is pushed > or the MR is rebased (https://pagure.io/fedora-ci/general/issue/493). > This means that if some changes are pushed in response to MR feedback, you > end up with an extra set of scratch builds running in parallel. This is > further exacerbated by Pagure not having proper support for rebase merges, > so if you hit Rebase and then Merge you also get a bonus scratch build. > (I'm not sure whether Zuul properly cancels scratch builds, or whether it > produces zombies as well.) > > * Fedora CI has no configurability. For example, we can't disable just the > s390x scratch builds (https://pagure.io/fedora-ci/general/issue/494), > which tend to be more than twice as slow as other builds. > > * As far as I know, it's not even possible to disable Fedora CI entirely > to e.g. only use Zuul instead. Similarly, we can't stop automatically > triggering Fedora CI and requiring manual [citest] instead. > > * For scratch builds longer than 4 hours, Fedora CI will never report back > the result (https://pagure.io/fedora-ci/general/issue/485), even though > the scratch build continues running. It will stay in the pending state > forever. For llvm all scratch builds take more than 4 hours, so we never > get results. This also means that dist-git tests never run. I submitted a > PR to raise this timeout ( > https://github.com/fedora-ci/dist-git-build-pipeline/pull/41) but wasn't > able to get a response. > > It's not really necessary to solve *all* of these problems -- I think the > MVP to make MRs usable for llvm without negatively affecting other people > would probably be to increase the timeout and either a) implement > auto-cancellation for scratch builds or b) allow preventing auto-start of > CI, requiring manual [citest]. (Naively, I assume the latter is easier to > implement.) > > However, I haven't been able to get any response from maintainers on > Fedora CI issues or PRs, so I'm not really sure what to do here anymore, > thus this mail to fedora-devel. I'd appreciate any pointers on how to move > forward. > > Regards, > Nikita > -- > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue > -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Non-responsive maintainer check for fbo (Fabien Boucher)
Hi folks, I've orphaned python-gear and python-ws4py. As suggested in [1] I propose this change [2] to retire python-fb-re2. [1] https://bugzilla.redhat.com/show_bug.cgi?id=2260410 [2] https://src.fedoraproject.org/rpms/python-fb-re2/pull-request/2 -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Changes to Zuul CI - rpm-linter job (RPMLint 2.2.0)
Hello, Note related to packages attached to the Zuul CI. This change [1], when merged, will set the container image to run rpmlint on Fedora 36. RPMLint version will be 2.2.0. Currently, the rpm-linter job uses a Fedora 34 container image with rpmlint 1.11. This change might impact your rpmlint configuration. I plan to merge this change [1] next Friday. This lets some time to validate/update your rpmlint configuration. To do so, add the following line into the "initial" comment of a new PR to validate speculatively the CI for your package: Depends-on: https://pagure.io/fedora-zuul-jobs/pull-request/157 If the rpm-linter job is "success" then that's fine you can close the PR. In case of failure this PR could be used to update your rpmlint configuration (such as rpmlintrc files). Regards [1]: https://pagure.io/fedora-zuul-jobs/pull-request/157 ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Orphaned Zuul package
Hi, Just to inform you that I've just orphaned the Zuul package. Feel free to take over it if you'd like. Deploying Zuul via container images seems the best solution for the moment. Upstream project provides container images: https://hub.docker.com/u/zuul A k8s operator is also available: https://opendev.org/zuul/zuul-operator Regards, Fabien ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: More distgit attached to Fedora Zuul CI
On Fri, Feb 19, 2021 at 8:56 AM Ondrej Mosnacek wrote: > On Tue, Feb 9, 2021 at 4:29 PM Pierre - Yves Chibon > wrote: > > On Tue, Feb 09, 2021 at 03:35:11PM +0100, Miro Hrončok wrote: > > > On 09. 02. 21 15:27, Tom Stellard wrote: > > > > Has there been any more discussion about disabling simple-koji-ci on > > > > packages with Zuul enabled? > > > > > > It's generally disabled AFAIK. Or broken. Same outcome. > > > > AFAIK it's still running. Happy to decommission it though. > > I think in one pull request, I saw simple-koji-ci being able to > generate the source tarball (presumably from the Source: URL) -> SRPM > -> start the build even when the tarball hadn't yet been uploaded to > the lookaside cache. Fedora CI wasn't able to do that at that time > (probably still can't) and Zuul CI wasn't enabled on that repo back > then. > > Was I just imagining things or does simple-koji-ci really have that > feature? If yes, does/will Zuul CI provide the same? > > I can't say for simple-koji-ci, but regarding Zuul the default jobs workflow is: 1: rpm-scratch-build: build a srpm from the PR, submit the srpm to Koji to run a scratch build, finally store the built rpms in a public storage Once 1 is finished, and in parallel: 2: rpmlint: run rpmlint command on built rpms 3: rpminspect: run rpminspect command on built rpms 4: rpm-install-test: install the built rpms on the target system 5: rpm-test (if STI tests inside the PR/distgit): run the STI tests In addition, if the package produces architecture specific binaries, then Zuul runs additional scratch-build jobs such as rpm-scratch-build-[s390x, ...], that are skipped if fully noarch. This can be seen here [1] and here [2]. [1]: https://src.fedoraproject.org/rpms/python-gear/pull-request/41#comment-68067 [2]: https://src.fedoraproject.org/rpms/java-latest-openjdk/pull-request/45#comment-68054 ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
More distgit attached to Fedora Zuul CI
Hi Folks, We are going to add a new bundle of distgits to Fedora Zuul CI. This bundle [1] has been computed from datagrapper to get the list of distgits that received at least 2 PR updates since the last 3 months. Here [2] is the Pull-Request we are going to merge, in the coming days, into the Zuul configuration. Like for the previous bundles of distgits we added, we will notify, by email, maintainers that a CI has been attached to their packages. More information about FZCI in the wiki page [3]. Previous bundles we added: [4]. Please let us know if you have any questions or concerns. Regards, Fabien [1]: https://softwarefactory-project.io/paste/show/1942/ [2]: https://pagure.io/fedora-project-config/pull-request/130 [3]: https://fedoraproject.org/wiki/Zuul-based-ci [4]: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/K6US7R6GUGGIID6S4PYMQGREUKL45QAW/ ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: More distgit attached to Fedora Zuul CI ?
Hi Folks, We are going to add a new list of distgit to Fedora Zuul CI. The openstack-sig [1] group would like to benefit from this CI for the packages they maintain. Here [2] is the Pull-Request we are going to merge with these new distgits [3]. Like for the previous bundle of distgits we added, we will notify, by email, maintainers that a CI has been attached to their packages. Regards, Fabien [1]: https://src.fedoraproject.org/group/openstack-sig [2]: https://pagure.io/fedora-project-config/pull-request/119 [3]: https://softwarefactory-project.io/paste/show/1899/ On Mon, Nov 9, 2020 at 9:40 AM Fabien Boucher wrote: > Hi folks, > > I'm forwarding a mail I sent to the Fedora CI mailing list last week, as > it might interest some folks on that mailing list too. > > https://lists.fedoraproject.org/archives/list/c...@lists.fedoraproject.org/thread/7CK45NW5IDXU646AO5QPN2G7FEFOR5Y7/ > > --- > > As you may know we provide Zuul CI [1] for some Fedora projects and > packages [2],[3]. We are running this CI For Fedora for more than one year > and we see that the platform and jobs are pretty stable. This CI is part of > a wider CI platform that the Software Factory project is maintaining for > various projects like RDO [4]. We have some free capacity and we would like > to provide more CI to Fedora especially for distgit repos. > > Since a previous improvement [5], attaching a distgit repository to Fedora > Zuul CI is easier and does not require any project maintainer action. > > A Pull-Request is proposed here [6] to activate the CI on 148 distgit > repos (the sorted list of the distgit names is here [7]). Each distgit part > of this list has received at least 2 new PRs during the last two months > (source datagrepper). > > So we would like to know if Fedora maintainers and contributors will be > happy to get Zuul CI feedback on those repositories ? > Here is an example of a run of the default distgit CI jobs for > rpms/python3.10 [8]. > > Let us know if you think we can activate the CI for those projects ? > > We will be glad to include more repositories to that list regularly. > > Regards, > Fabien > > 1 - https://zuul-ci.org/ > 2 - https://fedoraproject.org/wiki/Zuul-based-ci > 3 - https://fedora.softwarefactory-project.io/zuul/status > 4 - https://review.rdoproject.org/zuul/status > 5 - > https://lists.fedoraproject.org/archives/list/c...@lists.fedoraproject.org/thread/7SZGJCIFOATZJ3KJYOBQMUG4SMAYAGGU/ > 6 - https://pagure.io/fedora-project-config/pull-request/116#request_diff > 7 - https://softwarefactory-project.io/paste/show/1892/ > 8 - https://src.fedoraproject.org/rpms/python3.10/pull-request/9 > ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
More distgit attached to Fedora Zuul CI ?
Hi folks, I'm forwarding a mail I sent to the Fedora CI mailing list last week, as it might interest some folks on that mailing list too. https://lists.fedoraproject.org/archives/list/c...@lists.fedoraproject.org/thread/7CK45NW5IDXU646AO5QPN2G7FEFOR5Y7/ --- As you may know we provide Zuul CI [1] for some Fedora projects and packages [2],[3]. We are running this CI For Fedora for more than one year and we see that the platform and jobs are pretty stable. This CI is part of a wider CI platform that the Software Factory project is maintaining for various projects like RDO [4]. We have some free capacity and we would like to provide more CI to Fedora especially for distgit repos. Since a previous improvement [5], attaching a distgit repository to Fedora Zuul CI is easier and does not require any project maintainer action. A Pull-Request is proposed here [6] to activate the CI on 148 distgit repos (the sorted list of the distgit names is here [7]). Each distgit part of this list has received at least 2 new PRs during the last two months (source datagrepper). So we would like to know if Fedora maintainers and contributors will be happy to get Zuul CI feedback on those repositories ? Here is an example of a run of the default distgit CI jobs for rpms/python3.10 [8]. Let us know if you think we can activate the CI for those projects ? We will be glad to include more repositories to that list regularly. Regards, Fabien 1 - https://zuul-ci.org/ 2 - https://fedoraproject.org/wiki/Zuul-based-ci 3 - https://fedora.softwarefactory-project.io/zuul/status 4 - https://review.rdoproject.org/zuul/status 5 - https://lists.fedoraproject.org/archives/list/c...@lists.fedoraproject.org/thread/7SZGJCIFOATZJ3KJYOBQMUG4SMAYAGGU/ 6 - https://pagure.io/fedora-project-config/pull-request/116#request_diff 7 - https://softwarefactory-project.io/paste/show/1892/ 8 - https://src.fedoraproject.org/rpms/python3.10/pull-request/9 ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: CPE Git Forge Decision
On Fri, Apr 3, 2020 at 9:11 PM Adam Williamson wrote: > > I also hope there will be an opportunity for discussion (with input > from the community) of whether those requirements can be fulfilled in > some way *other* than using a non-free Gitlab product. To take the > 'merge train' example - as Neal and I have mentioned, Pagure now in > fact *does* have some impressive capabilities in this area, thanks to > the Pagure<->Zuul integration that the Software Factory folks have been > working on, and presented at Devconf. I am personally using this for > multiple projects, and blogged about it here: > > https://www.happyassassin.net/2020/02/12/using-zuul-ci-with-pagure-io/ > As far as I understand the "merge train" feature from Gitlab, it seems to be similar to Zuul's "speculative merging" feature, except that Zuul is multi-repository aware by design, and thus Zuul is able to gate changes in their order of approval across a logical (user defined) set of repositories. When a project's code is spread across multiple repositories then Zuul's approach is invaluable. A striking example of a project spanning multiple repositories is of course Fedora where all packages are in their own dist-git repository. Another feature offered by Zuul that might be relevant in the dist-git context is the ability to have Zuul test PRs that require other (still unmerged) PRs. This allows creating CI jobs capable of handling build and runtime dependencies of RPMs. > an obvious avenue to explore here would be "can we work with Software > Factory folks to do a similar integration between Gitlab Core and > Zuul"? On the face of it, that would offer a way to achieve these > capabilities in a fully F/OSS way. > As for Gitlab integration, a driver is available for Zuul, but it is not as feature complete compared to the other drivers like the Pagure one. It might take some time to cover the missing bits; it might also be necessary to submit feature requests to gitlab's APIs themselves. I have been involved in writing those drivers and I fear it might not be as "easy" as it was for the Pagure driver. I might be wrong, but I don't expect any support from the gitlab side, for an obvious reason: offering "free" integration with Zuul would effectively kill their effort on monetizing "merge train", a similar in-house feature. On the other hand, I'd like to point out that working with Pagure maintainers to discuss missing features (mainly API endpoints) was really productive. Pagure folks are welcoming and provide guidance so it was easy for me to implement most of the missing pieces in the code to integrate with Zuul. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Defining the future of the packager workflow in Fedora
On Wed, Oct 2, 2019 at 9:20 PM Neal Gompa wrote: > > Unfortunately, it doesn't scale for the large number of packages we > have. Pull requests would work if we had mergify[1] working on > Dist-Git, otherwise I can't see how it'd work. > > [1]: https://github.com/Mergifyio/mergify-engine > > Since some months we (the team that maintain softwarefactory-project.io and review.rdoproject.org) are working on making Zuul [1] compatible with Pagure in order to implement a packager workflow through Zuul's jobs around the Pull Request model. Zuul is a powerful gating system, scalable, that support cross-repository jobs and artifacts sharing between jobs out of the box. Zuul's jobs are Ansible playbooks. We managed to do great progress that we shown at last Flock. We are tracking the progress in this EPIC [2]. Here is how a project is configured: [3], the template is defined like so: [4], and here the job definition of rawhide-rpm-scratch-build: [5] Zuul's jobs configuration is flexible and we can provide templates that fit different needs, whether the packager want a full workflow (from the scratch build, various validation jobs such as linting/rpminspect to the regular build) or just the scratch build. We will soon propose this as an opt-in service for the interested packagers. CI resources capacity will come from softwarefactory-project.io. [1]: https://zuul-ci.org/ [2]: https://teams.fedoraproject.org/project/ci/epic/14 [3]: https://pagure.io/fedora-zuul-jobs-config/blob/master/f/zuul.d/projects.yaml#_5 [4]: https://pagure.io/fedora-zuul-jobs/blob/master/f/zuul.d/templates.yaml [5]: https://pagure.io/fedora-zuul-jobs/blob/master/f/zuul.d/jobs.yaml#_1 ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Defining the future of the packager workflow in Fedora
On Fri, Sep 27, 2019 at 11:56 AM Miro Hrončok wrote: > Yes. I've em-mailed you about the problem when it was happening, asking > you to > disable it, there was no reply and I managed to build it at the end. > > So, I apologize. I missed your email. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Defining the future of the packager workflow in Fedora
On Fri, Sep 27, 2019 at 12:03 AM Miro Hrončok wrote: > I remember that during the Python 3.8 rebuilds in the side tag, one > package had > this automated somehow already. I was bumping the release/changelog and > trying > to build it in the side tag at least 5 times, but I was building with > --background, so the automation always got the NEVR in Koji first. > > I guess it is python-gear ? Where I'm experimenting some automation based on Zuul CI. There is a job triggered to build in Koji when master ref change. This job can be trigger instead when a PR is closed/merged, that way direct push will not trigger a build. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Self introduction: Fabien Boucher
Hello Folks, I've submitted two requests for review here: - Zuul : https://bugzilla.redhat.com/show_bug.cgi?id=1220451 - python-gear : https://bugzilla.redhat.com/show_bug.cgi?id=1215046 As these are my two first packages I give you some quick information about me. I'm Fabien boucher, software engineer, from Paris. Today I'm working around the Openstack community. I'm mostly interested by python programming, the scalable systems and Continous Integration's practice. Since some months I contribute to a project [1] that try to ease the deployment of a CI stack based on Gerrit/Zuul/Jenkins and CentOS images. Zuul is an important component for such a stack but it is not packaged for Fedora neither for CentOS so I would love to have it embedded in Fedora. I guess having Zuul packaged will also benefit to Fedora users that want to build their own CI architecture. I guess I need some help from reviewers for packages mentioned above :) Thanks in advance for your help ! [1]: https://github.com/enovance/software-factory Cheers, Fabien Boucher -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
