Re: RFC: Banning bots from submitting automated koji builds
On Fri, 2021-06-25 at 15:42 -0400, Daniel Walsh wrote: > > And sometimes this breakage is caused by other parts of the system. For > example a kernel update caused breakage in Podman when it suddenly > enabled overlay mounts, which no one had tried. We quickly fixed the > container-selinux package to handle it, and got the fixes in F33 and F34 > before the kernel showed up. > > If we remove Podman updates from Rawhide other then when we prepare for > release. Their will be errors that do not get caught early. > > Forcing us to treat Rawhide like we do F34 makes Rawhide less > interesting to the container effort. I don't believe anyone suggested anything of the sort, though. The initial post in this thread proposed banning automated builds. The discussion has touched on the problems of automated dumps of random git snapshots with no manual involvement and the lack of any kind of attention to Bodhi comments. The actual issues raised in the thread can all be addressed without "treat[ing] Rawhide like we do F34". -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On Fri, Jun 25, 2021 at 9:43 PM Daniel Walsh wrote: > (snip) > > And sometimes this breakage is caused by other parts of the system. For > example a kernel update caused breakage in Podman when it suddenly > enabled overlay mounts, which no one had tried. We quickly fixed the > container-selinux package to handle it, and got the fixes in F33 and F34 > before the kernel showed up. > > If we remove Podman updates from Rawhide other then when we prepare for > release. Their will be errors that do not get caught early. > > Forcing us to treat Rawhide like we do F34 makes Rawhide less > interesting to the container effort. That's not what we're asking for. Pushing Betas or RCs into Rawhide makes sense. But *only if* anybody is actually noticing that, for example, the last few dozen podman builds didn't even get pushed into rawhide repos because they failed gating tests. Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On 6/25/21 16:13, Neal Gompa wrote: On Fri, Jun 25, 2021 at 3:43 PM Daniel Walsh wrote: On 6/25/21 10:25, Neal Gompa wrote: On Fri, Jun 25, 2021 at 10:15 AM Lokesh Mandvekar wrote: Hi list, I own the rhcontainerbot account. Apologies it took so long to respond to this thread. A number of legitimate concerns have been raised about the bot, so let me address those below on behalf of the Containers team. We have disabled all autobuilds for now. The podman RC build landing in updates a month ago was a one-off and it has been discussed at: https://lists.podman.io/archives/list/pod...@lists.podman.io/thread/WYNTH224D5MVBC2RFOG6RPIC52JZFKAB/ The fuse-overlayfs downgrade occurred unintendedly during the upstream branch rename from master to main. That has been fixed at: https://koji.fedoraproject.org/koji/buildinfo?buildID=1775442 Disabling autobuilds during the branch rename phase would’ve evidently avoided this issue. Going forward: We will only manually build upstream release tags for Fedora releases. We would prefer to send RC tags to Fedora rawhide as that will trigger gating tests and allow us to test podman with FCOS and toolbox CIs, so please let us know if that would be a deal-breaker. RCs and final releases are generally okay IMO even for stable releases, as long as you're prepared to address feedback brought up in bodhi updates. The problem here is nobody is paying attention to Bodhi at all. We may look at re-enabling the bot only for koji builds of upstream releases, while bodhi updates will still be manual. We’ll make sure to check for breakages / version downgrades before re-enablement. The bot has so far compared upstream tags, rpm installability, version number sanity, but evidently it has missed a lot of cases including git branch changes. If we re-enable the bot, we will mention the human’s name and email for every changelog entry in every relevant package as well as regularly monitor the bot’s email. Please let us know if there are any concerns with this approach. We will use openSUSE’s OBS for builds of the latest upstream commits for our users who need the latest packages. We would need this to check if the latest commits in podman work well with new kernel features and selinux. Team members will not add karma to containers’ packages, with the exception of our QE Engineer who owns our gating tests and is in charge of final testing of our builds. Currently Ed Santiago (FAS: @santiago) owns that responsibility. The important aspect isn't who is doing it, but that it's actually *tested* to work. Very serious breakages have happened in the past, and that's we want to avoid going forward. And sometimes this breakage is caused by other parts of the system. For example a kernel update caused breakage in Podman when it suddenly enabled overlay mounts, which no one had tried. We quickly fixed the container-selinux package to handle it, and got the fixes in F33 and F34 before the kernel showed up. If we remove Podman updates from Rawhide other then when we prepare for release. Their will be errors that do not get caught early. Forcing us to treat Rawhide like we do F34 makes Rawhide less interesting to the container effort. But none of you are paying attention to Rawhide anyway. As far as I know, none of you actively run Rawhide, none of you test it, and nobody is paying attention to when stuff is pushed into Rawhide. This is the difference between what your team is doing and what I do when I push snapshots into Rawhide. If you're going to push stuff into Rawhide, *care* about it. I ran rawhide for > 10 years so I know the joy and sometimes pain, but I will have the team focus more on it going forward. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On Fri, Jun 25, 2021 at 3:43 PM Daniel Walsh wrote: > > On 6/25/21 10:25, Neal Gompa wrote: > > On Fri, Jun 25, 2021 at 10:15 AM Lokesh Mandvekar > > wrote: > >> Hi list, > >> > >> > >> I own the rhcontainerbot account. Apologies it took so long to respond to > >> this thread. A number of legitimate concerns have been raised about the > >> bot, so let me address those below on behalf of the Containers team. > >> > >> > >> We have disabled all autobuilds for now. > >> > >> The podman RC build landing in updates a month ago was a one-off and it > >> has been discussed at: > >> https://lists.podman.io/archives/list/pod...@lists.podman.io/thread/WYNTH224D5MVBC2RFOG6RPIC52JZFKAB/ > >> > >> The fuse-overlayfs downgrade occurred unintendedly during the upstream > >> branch rename from master to main. That has been fixed at: > >> https://koji.fedoraproject.org/koji/buildinfo?buildID=1775442 > >> Disabling autobuilds during the branch rename phase would’ve evidently > >> avoided this issue. > >> > >> > >> Going forward: > >> > >> We will only manually build upstream release tags for Fedora releases. We > >> would prefer to send RC tags to Fedora rawhide as that will trigger gating > >> tests and allow us to test podman with FCOS and toolbox CIs, so please let > >> us know if that would be a deal-breaker. > >> > > RCs and final releases are generally okay IMO even for stable > > releases, as long as you're prepared to address feedback brought up in > > bodhi updates. The problem here is nobody is paying attention to Bodhi > > at all. > > > >> We may look at re-enabling the bot only for koji builds of upstream > >> releases, while bodhi updates will still be manual. We’ll make sure to > >> check for breakages / version downgrades before re-enablement. The bot has > >> so far compared upstream tags, rpm installability, version number sanity, > >> but evidently it has missed a lot of cases including git branch changes. > >> > >> If we re-enable the bot, we will mention the human’s name and email for > >> every changelog entry in every relevant package as well as regularly > >> monitor the bot’s email. Please let us know if there are any concerns with > >> this approach. > >> > >> We will use openSUSE’s OBS for builds of the latest upstream commits for > >> our users who need the latest packages. We would need this to check if the > >> latest commits in podman work well with new kernel features and selinux. > >> > >> > >> Team members will not add karma to containers’ packages, with the > >> exception of our QE Engineer who owns our gating tests and is in charge of > >> final testing of our builds. Currently Ed Santiago (FAS: @santiago) owns > >> that responsibility. > >> > > The important aspect isn't who is doing it, but that it's actually > > *tested* to work. Very serious breakages have happened in the past, > > and that's we want to avoid going forward. > > And sometimes this breakage is caused by other parts of the system. For > example a kernel update caused breakage in Podman when it suddenly > enabled overlay mounts, which no one had tried. We quickly fixed the > container-selinux package to handle it, and got the fixes in F33 and F34 > before the kernel showed up. > > If we remove Podman updates from Rawhide other then when we prepare for > release. Their will be errors that do not get caught early. > > Forcing us to treat Rawhide like we do F34 makes Rawhide less > interesting to the container effort. > But none of you are paying attention to Rawhide anyway. As far as I know, none of you actively run Rawhide, none of you test it, and nobody is paying attention to when stuff is pushed into Rawhide. This is the difference between what your team is doing and what I do when I push snapshots into Rawhide. If you're going to push stuff into Rawhide, *care* about it. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On 6/25/21 10:25, Neal Gompa wrote: On Fri, Jun 25, 2021 at 10:15 AM Lokesh Mandvekar wrote: Hi list, I own the rhcontainerbot account. Apologies it took so long to respond to this thread. A number of legitimate concerns have been raised about the bot, so let me address those below on behalf of the Containers team. We have disabled all autobuilds for now. The podman RC build landing in updates a month ago was a one-off and it has been discussed at: https://lists.podman.io/archives/list/pod...@lists.podman.io/thread/WYNTH224D5MVBC2RFOG6RPIC52JZFKAB/ The fuse-overlayfs downgrade occurred unintendedly during the upstream branch rename from master to main. That has been fixed at: https://koji.fedoraproject.org/koji/buildinfo?buildID=1775442 Disabling autobuilds during the branch rename phase would’ve evidently avoided this issue. Going forward: We will only manually build upstream release tags for Fedora releases. We would prefer to send RC tags to Fedora rawhide as that will trigger gating tests and allow us to test podman with FCOS and toolbox CIs, so please let us know if that would be a deal-breaker. RCs and final releases are generally okay IMO even for stable releases, as long as you're prepared to address feedback brought up in bodhi updates. The problem here is nobody is paying attention to Bodhi at all. We may look at re-enabling the bot only for koji builds of upstream releases, while bodhi updates will still be manual. We’ll make sure to check for breakages / version downgrades before re-enablement. The bot has so far compared upstream tags, rpm installability, version number sanity, but evidently it has missed a lot of cases including git branch changes. If we re-enable the bot, we will mention the human’s name and email for every changelog entry in every relevant package as well as regularly monitor the bot’s email. Please let us know if there are any concerns with this approach. We will use openSUSE’s OBS for builds of the latest upstream commits for our users who need the latest packages. We would need this to check if the latest commits in podman work well with new kernel features and selinux. Team members will not add karma to containers’ packages, with the exception of our QE Engineer who owns our gating tests and is in charge of final testing of our builds. Currently Ed Santiago (FAS: @santiago) owns that responsibility. The important aspect isn't who is doing it, but that it's actually *tested* to work. Very serious breakages have happened in the past, and that's we want to avoid going forward. And sometimes this breakage is caused by other parts of the system. For example a kernel update caused breakage in Podman when it suddenly enabled overlay mounts, which no one had tried. We quickly fixed the container-selinux package to handle it, and got the fixes in F33 and F34 before the kernel showed up. If we remove Podman updates from Rawhide other then when we prepare for release. Their will be errors that do not get caught early. Forcing us to treat Rawhide like we do F34 makes Rawhide less interesting to the container effort. We will also notify the containers’ communities that rawhide will no longer contain the latest builds as some of them are accustomed to using. Having a COPR would be nice for this. With tools like Packit and such able to continuously build in COPR for every PR and every commit, you can provide a fairly nice experience here. I do this with rpmlint, for example. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
Dne 25. 06. 21 v 16:25 Neal Gompa napsal(a): We will use openSUSE’s OBS for builds of the latest upstream commits for our users who need the latest packages. We would need this to check if the latest commits in podman work well with new kernel features and selinux. We will also notify the containers’ communities that rawhide will no longer contain the latest builds as some of them are accustomed to using. Having a COPR would be nice for this. With tools like Packit and such able to continuously build in COPR for every PR and every commit, you can provide a fairly nice experience here. I do this with rpmlint, for +1 If you hesitate to use Copr for some reason or you want to help with initial setup then you can contact me privately and I will do my best to help you. Miroslav ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On Fri, Jun 25, 2021 at 10:15 AM Lokesh Mandvekar wrote: > > Hi list, > > > I own the rhcontainerbot account. Apologies it took so long to respond to > this thread. A number of legitimate concerns have been raised about the bot, > so let me address those below on behalf of the Containers team. > > > We have disabled all autobuilds for now. > > The podman RC build landing in updates a month ago was a one-off and it has > been discussed at: > https://lists.podman.io/archives/list/pod...@lists.podman.io/thread/WYNTH224D5MVBC2RFOG6RPIC52JZFKAB/ > > The fuse-overlayfs downgrade occurred unintendedly during the upstream branch > rename from master to main. That has been fixed at: > https://koji.fedoraproject.org/koji/buildinfo?buildID=1775442 > Disabling autobuilds during the branch rename phase would’ve evidently > avoided this issue. > > > Going forward: > > We will only manually build upstream release tags for Fedora releases. We > would prefer to send RC tags to Fedora rawhide as that will trigger gating > tests and allow us to test podman with FCOS and toolbox CIs, so please let us > know if that would be a deal-breaker. > RCs and final releases are generally okay IMO even for stable releases, as long as you're prepared to address feedback brought up in bodhi updates. The problem here is nobody is paying attention to Bodhi at all. > We may look at re-enabling the bot only for koji builds of upstream releases, > while bodhi updates will still be manual. We’ll make sure to check for > breakages / version downgrades before re-enablement. The bot has so far > compared upstream tags, rpm installability, version number sanity, but > evidently it has missed a lot of cases including git branch changes. > > If we re-enable the bot, we will mention the human’s name and email for every > changelog entry in every relevant package as well as regularly monitor the > bot’s email. Please let us know if there are any concerns with this approach. > > We will use openSUSE’s OBS for builds of the latest upstream commits for our > users who need the latest packages. We would need this to check if the latest > commits in podman work well with new kernel features and selinux. > > > Team members will not add karma to containers’ packages, with the exception > of our QE Engineer who owns our gating tests and is in charge of final > testing of our builds. Currently Ed Santiago (FAS: @santiago) owns that > responsibility. > The important aspect isn't who is doing it, but that it's actually *tested* to work. Very serious breakages have happened in the past, and that's we want to avoid going forward. > > We will also notify the containers’ communities that rawhide will no longer > contain the latest builds as some of them are accustomed to using. > Having a COPR would be nice for this. With tools like Packit and such able to continuously build in COPR for every PR and every commit, you can provide a fairly nice experience here. I do this with rpmlint, for example. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
Hi list, I own the rhcontainerbot account. Apologies it took so long to respond to this thread. A number of legitimate concerns have been raised about the bot, so let me address those below on behalf of the Containers team. 1. We have disabled all autobuilds for now. 2. The podman RC build landing in updates a month ago was a one-off and it has been discussed at: https://lists.podman.io/archives/list/pod...@lists.podman.io/thread/WYNTH224D5MVBC2RFOG6RPIC52JZFKAB/ 3. The fuse-overlayfs downgrade occurred unintendedly during the upstream branch rename from master to main. That has been fixed at: https://koji.fedoraproject.org/koji/buildinfo?buildID=1775442 Disabling autobuilds during the branch rename phase would’ve evidently avoided this issue. 1. Going forward: 1. We will only manually build upstream release tags for Fedora releases. We would prefer to send RC tags to Fedora rawhide as that will trigger gating tests and allow us to test podman with FCOS and toolbox CIs, so please let us know if that would be a deal-breaker. 2. We may look at re-enabling the bot only for koji builds of upstream releases, while bodhi updates will still be manual. We’ll make sure to check for breakages / version downgrades before re-enablement. The bot has so far compared upstream tags, rpm installability, version number sanity, but evidently it has missed a lot of cases including git branch changes. 3. If we re-enable the bot, we will mention the human’s name and email for every changelog entry in every relevant package as well as regularly monitor the bot’s email. Please let us know if there are any concerns with this approach. 4. We will use openSUSE’s OBS for builds of the latest upstream commits for our users who need the latest packages. We would need this to check if the latest commits in podman work well with new kernel features and selinux. 1. Team members will not add karma to containers’ packages, with the exception of our QE Engineer who owns our gating tests and is in charge of final testing of our builds. Currently Ed Santiago (FAS: @santiago) owns that responsibility. 1. We will also notify the containers’ communities that rawhide will no longer contain the latest builds as some of them are accustomed to using. Please let us know if there are any concerns that were left unaddressed or if you have any further recommendations or feedback. -- Lokesh On Fri, Jun 25, 2021 at 12:13 AM Dan Čermák wrote: > > > On June 22, 2021 1:26:30 PM UTC, "Miroslav Suchý" > wrote: > >Dne 20. 06. 21 v 10:42 Miro Hrončok napsal(a): > >> Rather than "no bots allowed" policy, we might need a "bots that > >violate our policies and guidelines or have a > >> tendency to break things will be disabled until fixed" policy. > > > >Every bot has been written by somebody. 1) it should be always clear > >who is responsible for the bot 2) The owner should > >take conseq^H^H^H responsibility for it. > > I'd prefer this policy over an outright ban. > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam on the list, report it: > https://pagure.io/fedora-infrastructure > ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On June 22, 2021 1:26:30 PM UTC, "Miroslav Suchý" wrote: >Dne 20. 06. 21 v 10:42 Miro Hrončok napsal(a): >> Rather than "no bots allowed" policy, we might need a "bots that >violate our policies and guidelines or have a >> tendency to break things will be disabled until fixed" policy. > >Every bot has been written by somebody. 1) it should be always clear >who is responsible for the bot 2) The owner should >take conseq^H^H^H responsibility for it. I'd prefer this policy over an outright ban. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On Tue, Jun 22, 2021 at 04:48:26PM -0700, Adam Williamson wrote: > Matthew was referring to a plan (AIUI) to have two locations where > "Rawhide" composes would be synced, one where all completed composes > would be synced (as today), one where only composes that passed gating > would be synced. I don't recall this plan, and don't know what happened > to it, but that's what the idea was, I think. Yes, that was it. -- Matthew Miller Fedora Project Leader ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On Tue, 2021-06-22 at 18:46 -0400, Neal Gompa wrote: > On Tue, Jun 22, 2021 at 6:42 PM Florian Weimer wrote: > > > > * Matthew Miller: > > > > > On Sun, Jun 20, 2021 at 09:48:15AM -0700, Adam Williamson wrote: > > > > We've talked about various concerns around this in the past (the > > > > technicalities of exactly how to implement it, and the concern that not > > > > enough composes actually meet the requirements so we'd wind up with few > > > > composes synced and a big disconnect between what's in the repos and > > > > what's in Koji), but the *idea* has been there all along and I agree > > > > with Neal that it was tied up with 'no more Alphas'. > > > > > > What happened with Dennis Gilmore's idea of making two levels of "rawhide" > > > rawhide, one like it is today (but not pushed to mirrors), in addition to > > > a > > > fully gated one? > > > > The ungated rawhide (aka the buildroot) is not pushed to mirrors today. > > The discrepancy between buildroots and mirrors is a significant source > > of developer confusion, I think. > > > > No. This never got implemented. There is Rawhide gating and such, but > we've never blocked syncs out to the master mirror. > > And for having two levels of rawhide, we'd need both levels synced to > the mirrors to be useful. Otherwise people can't really effectively > use it. I think you're talking about different things. Matthew was referring to a plan (AIUI) to have two locations where "Rawhide" composes would be synced, one where all completed composes would be synced (as today), one where only composes that passed gating would be synced. I don't recall this plan, and don't know what happened to it, but that's what the idea was, I think. Florian seems to be talking about how Rawhide builds go into the buildroot as soon as they pass Bodhi and get tagged, so other package builds start pulling them in right away, but those builds may not make it out to the public repos (which testers and packagers consume) for some time if composes are failing. That's a rather different thing and doesn't really have anything much to do with "gating" per se, but it is worth noting that this would likely only become a more significant issue if we started gating Rawhide compose sync on *tests passing* as well as the compose completing. The more stuff has to work before we sync a compose, the bigger the gap between what's in the buildroot and what's in the repos can grow, potentially. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On Tue, Jun 22, 2021 at 6:42 PM Florian Weimer wrote: > > * Matthew Miller: > > > On Sun, Jun 20, 2021 at 09:48:15AM -0700, Adam Williamson wrote: > >> We've talked about various concerns around this in the past (the > >> technicalities of exactly how to implement it, and the concern that not > >> enough composes actually meet the requirements so we'd wind up with few > >> composes synced and a big disconnect between what's in the repos and > >> what's in Koji), but the *idea* has been there all along and I agree > >> with Neal that it was tied up with 'no more Alphas'. > > > > What happened with Dennis Gilmore's idea of making two levels of "rawhide" > > rawhide, one like it is today (but not pushed to mirrors), in addition to a > > fully gated one? > > The ungated rawhide (aka the buildroot) is not pushed to mirrors today. > The discrepancy between buildroots and mirrors is a significant source > of developer confusion, I think. > No. This never got implemented. There is Rawhide gating and such, but we've never blocked syncs out to the master mirror. And for having two levels of rawhide, we'd need both levels synced to the mirrors to be useful. Otherwise people can't really effectively use it. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
* Matthew Miller: > On Sun, Jun 20, 2021 at 09:48:15AM -0700, Adam Williamson wrote: >> We've talked about various concerns around this in the past (the >> technicalities of exactly how to implement it, and the concern that not >> enough composes actually meet the requirements so we'd wind up with few >> composes synced and a big disconnect between what's in the repos and >> what's in Koji), but the *idea* has been there all along and I agree >> with Neal that it was tied up with 'no more Alphas'. > > What happened with Dennis Gilmore's idea of making two levels of "rawhide" > rawhide, one like it is today (but not pushed to mirrors), in addition to a > fully gated one? The ungated rawhide (aka the buildroot) is not pushed to mirrors today. The discrepancy between buildroots and mirrors is a significant source of developer confusion, I think. Thanks, Florian ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On Sun, Jun 20, 2021 at 09:48:15AM -0700, Adam Williamson wrote: > We've talked about various concerns around this in the past (the > technicalities of exactly how to implement it, and the concern that not > enough composes actually meet the requirements so we'd wind up with few > composes synced and a big disconnect between what's in the repos and > what's in Koji), but the *idea* has been there all along and I agree > with Neal that it was tied up with 'no more Alphas'. What happened with Dennis Gilmore's idea of making two levels of "rawhide" rawhide, one like it is today (but not pushed to mirrors), in addition to a fully gated one? -- Matthew Miller Fedora Project Leader ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
Dne 20. 06. 21 v 10:42 Miro Hrončok napsal(a): Rather than "no bots allowed" policy, we might need a "bots that violate our policies and guidelines or have a tendency to break things will be disabled until fixed" policy. Every bot has been written by somebody. 1) it should be always clear who is responsible for the bot 2) The owner should take conseq^H^H^H responsibility for it. Miroslav ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On Mon, 2021-06-21 at 10:48 +0100, Ankur Sinha wrote: > On Sun, Jun 20, 2021 11:41:41 -0700, Kevin Fenzi wrote: > > On Sun, Jun 20, 2021 at 03:26:52PM +, Zbigniew Jędrzejewski-Szmek wrote: > > > > > > Yeah. I'm looking a the original ticket in > > > https://pagure.io/fesco/issue/2228, and I think this was a mistake. > > > We shouldn't have approved a bot that packages snapshot commits for > > > rawhide. In the discussion, we talked about load on the infra, and ability > > > to contact the maintainers, and even cooperation withb packit, but somehow > > > the question whether we want this at all didn't come up. > > > > > > I guess the effort to make rawhide palatable hadn't really sunk in > > > deep enough back then ;) > > > > Well, it's still not clear to me if these builds are not suitable for > > rawhide, or if the bot that is pushing them just has bugs. > > A few podman updates have now caused major breakages for users even in > stable releases. Here's a recent example: > > https://bodhi.fedoraproject.org/updates/FEDORA-2021-fc15685f73 > https://bugzilla.redhat.com/show_bug.cgi?id=1958546 > https://github.com/containers/podman/issues/10274 Well, this at least we can do something about fairly easily. We have an openQA podman test already (a pair of tests actually), which check podman can pull from the registry, then build a basic image which runs a web server, run a container with that image, and test the web server is reachable. We'd only been running those on IoT and CoreOS images, but there's no reason we can't run them in other contexts too. So as of now they will also run on critpath updates and any non- critpath updates containing podman, containernetworking-plugins, crun, fuse-overlayfs or slirp4netns (other suggestions welcome): https://pagure.io/fedora-qa/os-autoinst-distri-fedora/c/9174472 https://pagure.io/fedora-qa/fedora_openqa/c/d861f94 https://src.fedoraproject.org/rpms/podman/pull-request/43 is a PR which would cause updates for podman for F33 and F34 to be gated on those tests. We could do the same for the other packages easily, I just filed one PR for now to see what the feedback will be. The tests at present don't exercise the toolbox container at all, but it'd be fairly easy to add that if it's a good idea. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On Sun, Jun 20, 2021 at 7:09 AM Peter Robinson wrote: > > > With things like [0] (TL;DR: bots submitting broken builds to rawhide) > > becoming a more regular occurrence, I propose that we extend the > > existing Updates Policy [1] to make it explicit that bots are not > > allowed to submit builds / updates - even to rawhide - unattended: > > "Rawhide is not your CI environment." > > I don't think that's good for either the packages that are using bots > or for Fedora as a whole. > > While the bots certainly don't always get it right nor do the humans > and I would garner that there's a lot more problems introduced in this > way by humans, over the years I've spent considerable time fixing > issues like this introduced by humans. Do we ban humans too? Do we ban > RHBZ bots as well because they don't always get it right either? > Ultimately automation is hard, but ultimately I feel it generally gets > it wrong no less than humans do, and generally in a consistent way at > least. I definitely agree here, we want automation to evolve rather than simply banning it. I understand the problem with automation, but anyone can make mistakes. We should definitely have a contact person responsible for the bot. And we can just ban the bot if it does something unusual. > > > Peter > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam on the list, report it: > https://pagure.io/fedora-infrastructure ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On Mon, Jun 21, 2021 at 07:34:03AM +, Mattia Verga via devel wrote:
> Bodhi has a config setting ('automatic_updates_blacklist') which can
> avoid specific users/bots to automatically create updates.
>
> If those bots are supposed to only test Koji builds, but not pushing
> updates we can list them out using that setting.
I don't think we would go in this direction.
First, our Update Guidelines say that builds which are not intended to
become submitted as updates should not be done in koji.
Second, there are better ways to do scratch builds, doing them
as real builds but then ignoring would be just confusing.
Third, to do a real build in koji, the changes need to be pushed to
dist-git, and the assumption is that an unrelated build might be
triggered at any time by other packagers (mass rebuilds, so-version
bumps, etc.), and this would then build that snapshot commit.
Zbyszek
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On Mon, Jun 21, 2021 at 11:50 AM Ankur Sinha wrote: > > On Sun, Jun 20, 2021 11:41:41 -0700, Kevin Fenzi wrote: > > On Sun, Jun 20, 2021 at 03:26:52PM +, Zbigniew Jędrzejewski-Szmek wrote: > > > > > > Yeah. I'm looking a the original ticket in > > > https://pagure.io/fesco/issue/2228, and I think this was a mistake. > > > We shouldn't have approved a bot that packages snapshot commits for > > > rawhide. In the discussion, we talked about load on the infra, and ability > > > to contact the maintainers, and even cooperation withb packit, but somehow > > > the question whether we want this at all didn't come up. > > > > > > I guess the effort to make rawhide palatable hadn't really sunk in > > > deep enough back then ;) > > > > Well, it's still not clear to me if these builds are not suitable for > > rawhide, or if the bot that is pushing them just has bugs. > > A few podman updates have now caused major breakages for users even in > stable releases. Here's a recent example: > > https://bodhi.fedoraproject.org/updates/FEDORA-2021-fc15685f73 > https://bugzilla.redhat.com/show_bug.cgi?id=1958546 > https://github.com/containers/podman/issues/10274 > > ^ In this an RC went out all the way to the updates repo, but it seems > like pushing RCs to testing is very common for podman: > https://bodhi.fedoraproject.org/updates/?search==podman=F34=F33=2 > > (This is not OK in my book either---volunteers using updates-testing to > help test updates should not have to deal with updates that are not > intended for distribution to users, but yet somehow do make it through > from time to time.) > > It has now happened a couple of times (in my experience) that an update > went out, *very* quickly reached stable (sometimes before it was even > pushed to testing), and so reached users---where it was found that the > update broke core functionality. Because it was already marked stable, > by the time folks tested it out and gave negative karma, it could no > longer be un-pushed. > > I don't think there's a policy against this, but apart from podman, I > cannot recall seeing maintainers/dev teams give karma to their own > packages' updates. I don't think that works---the idea of Bodhi is to > allow ample opportunity for *others* to test the update, no? If > maintainers smoke test, push updates, and then again give karma based on > the smoke test, they're hardly likely to catch issues? Here's one where > an essentially broken update got 3 positive karma from folks involved in > podman development: > https://bodhi.fedoraproject.org/updates/FEDORA-2020-cd7e382e0c > > Here's another where folks related to podman development said "works" > (but gave negative karma to prevent it from going to stable) and then, > it turned out that the update was actually broken: > https://bodhi.fedoraproject.org/updates/FEDORA-2020-ad5899f2c3 > > I think bots are great but only as long as they continue to implement > the packaging policies that we've all agreed on. The bot isn't doing > that very well here. I agree. The Updates Policy even states that packagers should only ever push builds / updates if they're expected to be pushed to stable, and letting things sit in "updates-testing" for extended periods of time is forbidden. Even the current, latest podman update in rawhide is stuck, and nobody seems to be aware of this situation: - FEDORA-2021-147a47d0f8 : stuck in "testing" due to failed gating tests, obsoleted by: - FEDORA-2021-d963d68cb7 : stuck in "testing" due to failed gating tests, obsoleted by: - FEDORA-2021-a710045a4c : stuck in "testing" due to failed gating tests, obsoleted by: - https://bodhi.fedoraproject.org/updates/FEDORA-2021-b255b5da0d : stuck in "testing" due to failed gating tests At this point, that's just a waste of resources, when builds are submitted but never even reach repositories. Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On 21. 06. 21 11:48, Ankur Sinha wrote: I don't think there's a policy against this, but apart from podman, I cannot recall seeing maintainers/dev teams give karma to their own packages' updates. I don't think that works---the idea of Bodhi is to allow ample opportunity for*others* to test the update, no? If maintainers smoke test, push updates, and then again give karma based on the smoke test, they're hardly likely to catch issues? For the record: I do that with the alternate Pythons. If a teammate pushes a bodhi update I've already reviewed during the pull request process, I karma it up to expedite it to stable. (OTOH For the "main" Python version, we intentionally set the karma limits high enough to prevent "LGTM" karma pushing the the update to stable even before it reaches testing.) -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On Sun, Jun 20, 2021 11:41:41 -0700, Kevin Fenzi wrote: > On Sun, Jun 20, 2021 at 03:26:52PM +, Zbigniew Jędrzejewski-Szmek wrote: > > > > Yeah. I'm looking a the original ticket in > > https://pagure.io/fesco/issue/2228, and I think this was a mistake. > > We shouldn't have approved a bot that packages snapshot commits for > > rawhide. In the discussion, we talked about load on the infra, and ability > > to contact the maintainers, and even cooperation withb packit, but somehow > > the question whether we want this at all didn't come up. > > > > I guess the effort to make rawhide palatable hadn't really sunk in > > deep enough back then ;) > > Well, it's still not clear to me if these builds are not suitable for > rawhide, or if the bot that is pushing them just has bugs. A few podman updates have now caused major breakages for users even in stable releases. Here's a recent example: https://bodhi.fedoraproject.org/updates/FEDORA-2021-fc15685f73 https://bugzilla.redhat.com/show_bug.cgi?id=1958546 https://github.com/containers/podman/issues/10274 ^ In this an RC went out all the way to the updates repo, but it seems like pushing RCs to testing is very common for podman: https://bodhi.fedoraproject.org/updates/?search==podman=F34=F33=2 (This is not OK in my book either---volunteers using updates-testing to help test updates should not have to deal with updates that are not intended for distribution to users, but yet somehow do make it through from time to time.) It has now happened a couple of times (in my experience) that an update went out, *very* quickly reached stable (sometimes before it was even pushed to testing), and so reached users---where it was found that the update broke core functionality. Because it was already marked stable, by the time folks tested it out and gave negative karma, it could no longer be un-pushed. I don't think there's a policy against this, but apart from podman, I cannot recall seeing maintainers/dev teams give karma to their own packages' updates. I don't think that works---the idea of Bodhi is to allow ample opportunity for *others* to test the update, no? If maintainers smoke test, push updates, and then again give karma based on the smoke test, they're hardly likely to catch issues? Here's one where an essentially broken update got 3 positive karma from folks involved in podman development: https://bodhi.fedoraproject.org/updates/FEDORA-2020-cd7e382e0c Here's another where folks related to podman development said "works" (but gave negative karma to prevent it from going to stable) and then, it turned out that the update was actually broken: https://bodhi.fedoraproject.org/updates/FEDORA-2020-ad5899f2c3 I think bots are great but only as long as they continue to implement the packaging policies that we've all agreed on. The bot isn't doing that very well here. -- Thanks, Regards, Ankur Sinha "FranciscoD" (He / Him / His) | https://fedoraproject.org/wiki/User:Ankursinha Time zone: Europe/London signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
Bodhi has a config setting ('automatic_updates_blacklist') which can
avoid specific users/bots to automatically create updates.
If those bots are supposed to only test Koji builds, but not pushing
updates we can list them out using that setting.
Mattia
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On Sun, Jun 20, 2021 at 7:19 PM Zbigniew Jędrzejewski-Szmek wrote: > > On Sun, Jun 20, 2021 at 08:37:03AM -0500, Michael Catanzaro wrote: > > On Sun, Jun 20 2021 at 07:29:16 AM -0400, Neal Gompa > > wrote: > > >Most of our rules are designed to make sure there's someone ultimately > > >responsible for everything going into Fedora. Unfortunately, bots are > > >the opposite of that, because there's no one to reach to stop bad > > >behavior when it happens. > > > Hm, this seems pretty simple to solve though, right? Allow bots to > > submit updates on behalf of packagers, but not with their own bot > > FAS accounts. > > Let's not throw out the baby with the bath water. > > A human *is* responsible and known. When a bot account is given > permission, we make sure that there's a known human behind the account. > Things are no other in this particular case, see the original ticket [1]. > > Actually, if the bot were using their human's account, things would be *less* > transparent. By using a separate account, we are making it clear that > this update stream is made by this particular bot (as opposed to e.g. > some human occasionally using a script to release some updates). > > [1] https://pagure.io/fesco/issue/2228 > I wish our new FAS implementation gave us the ability to generate delegate/service accounts associated with a primary account. That way, we have a clear record of a human owning it, and when that human's account is known to no longer be active, the bot breaks with it. > > This would be like how GNOME package updates currently > > work, where a bot does the hard work but a human is ultimately > > responsible (and subscribed to each bodhi update, so feedback will > > at least not be completely missed). > > The line can be a big hazy, but I'd say that if: > - a human is just using a script or even a some program to fire off > the update — this particular person's account must be used. > - some bot prepares the update, but a human still need to make the final > step and may or may not publish the update: probably better to do it > using this person's account. > - the bot is set up once and then keeps releasing updating until stopped, > and may be managed by multiple people — a separate bot account is > preferable. > The problem is that this whole thing works off the premise that Rawhide is a dumping ground. It is not. It also works off the premise that nobody cares about the stuff being pushed into Dist-Git, Koji, and to users. And frankly, that has not been true for a *very* long time, if it ever was. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On Sun, Jun 20, 2021 at 8:45 PM Kevin Fenzi wrote: > > My question would be: Are these automated builds otherwise known working > and ready for integration testing with the rest of the distribution in > rawhide? > > If they are, we (just) need to fix the bot to not do the wrong thing. > > If they are not, then indeed, only those that meet that critera should > be pushed into rawhide. > > ie, it doesn't matter to me if the package version is > 0.0alpha-do-not-use if the maintainer(s) say it works and is > ready to integrate with the rest of the distribution. One projects > "0.001-danger" is another projects "10.1" stable. We should trust > maintainer(s) to know this... I wouldn't have a problem with this, but looking at those builds apparently - nobody monitors bodhi updates created for those builds, and - nobody reads emails for the rhcontainerbot account. The bodhi updates look very much like "throw this over the wall and forget it". They are never interacted with. If gating fails (which happens regularly), they're just stuck in rawhide's weird "testing" state, until the next update for that package obsoletes them. In combination with the bot pushing weird / wrong versions, this doesn't look too trustworthy to me. Who is even responsible for the rhcontainerbot account? I looked at the linked FESCo ticket (this was approved before my time), but I couldn't figure out who's actually using this bot now, or who is responsible for monitoring its actions. Because it sure seems like nobody is looking. Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On Sun, Jun 20, 2021 at 02:36:57PM +0200, Aleksandra Fedorova wrote: > Hi, > > On Sun, Jun 20, 2021 at 1:30 PM Neal Gompa wrote: ...snip... > > > > Rawhide is still not your CI environment. > > By the way I think we need to reconsider this statement. I think we should turn it around and say: Rawhide is your first stop to integrate your otherwise ready package(s) with the rest of the distribution. Meh, thats not great either... perhaps a slogan is bad. ;) kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On Sun, Jun 20, 2021 at 03:26:52PM +, Zbigniew Jędrzejewski-Szmek wrote: > > Yeah. I'm looking a the original ticket in > https://pagure.io/fesco/issue/2228, and I think this was a mistake. > We shouldn't have approved a bot that packages snapshot commits for > rawhide. In the discussion, we talked about load on the infra, and ability > to contact the maintainers, and even cooperation withb packit, but somehow > the question whether we want this at all didn't come up. > > I guess the effort to make rawhide palatable hadn't really sunk in > deep enough back then ;) Well, it's still not clear to me if these builds are not suitable for rawhide, or if the bot that is pushing them just has bugs. In the case that caused this thread, the bot pushed older pacakges? So, it wasn't that they were completely broken, it's that they were just _wrong_. > @lsm5, would you be willing to adjust the bot to only package actual > releases? Also, what is going on with the version jumps? My question would be: Are these automated builds otherwise known working and ready for integration testing with the rest of the distribution in rawhide? If they are, we (just) need to fix the bot to not do the wrong thing. If they are not, then indeed, only those that meet that critera should be pushed into rawhide. ie, it doesn't matter to me if the package version is 0.0alpha-do-not-use if the maintainer(s) say it works and is ready to integrate with the rest of the distribution. One projects "0.001-danger" is another projects "10.1" stable. We should trust maintainer(s) to know this... kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On Sun, 2021-06-20 at 10:42 +0200, Miro Hrončok wrote: > On 20. 06. 21 9:39, Zbigniew Jędrzejewski-Szmek wrote: > > I think we should just disable this particular bot until it fixed. > > We should also clarify/update the Update Guidelines so that > > undesirable > > updates are disallowed, no matter if submitted by a bot or a human. > > I think this is a good idea. This particular bot has a history of > misbehavior > and rather than banning all the well behaving bots (that be > definition we don't > even know about, because they behave good), we should disable this > particular one. > > Rather than "no bots allowed" policy, we might need a "bots that > violate our > policies and guidelines or have a tendency to break things will be > disabled > until fixed" policy. 100% agree, the problem is the bot that is, how to say without being offensive, not very smart. > -- > Miro Hrončok > -- > Phone: +420777974800 > IRC: mhroncok > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam on the list, report it: > https://pagure.io/fedora-infrastructure -- Sérgio M. B. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
Miro Hrončok writes: >> - releng and SIGs submitting scripted mass rebuilds (no actual package >> changes, triggered by a person) >> - bots submitting rawhide builds for ELN (no package change, just >> built for different buildroot) > Other random ideas of what should be allowed: > - a human approves a pull request, bot merges it and builds > - a bot rebuilds packages automatically (bump-only or no changes), > when dependencies change A simpler way to phrase that would be: "No autonomous bots are to push builds into rawhide or release branches." - FChE ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On Sun, 2021-06-20 at 12:52 +0100, Peter Robinson wrote: > > Rawhide is still not your CI environment. Years ago, we got rid of > > alphas for the express purpose to stabilize Rawhide into alpha > > quality. Stuff like this degrades the quality of Rawhide because they > > make the assumption that nobody cares about the quality of Rawhide. > > Being one of the people driving that in rel-eng at the time, the other > was dgilmore, that statement is incorrect. The dropping of Alphas was > because we improved the compose to produce all release artifacts on > the nightly composes, both rawhide and branched, instead of just the > network installer and live images. Previously to that we only produced > all artifacts with the TC/RC composes, we also got rid of the TCs as > part of that process. My recollection is that both of these things are correct. Having full nightly composes was one part of dropping Alphas, but so was the idea that Rawhide would continually maintain Alpha quality. This is why the Basic release criteria exist - they are the requirements that Rawhide is supposed to meet *all the time*. They say this right at the top: "The objectives for all Branched and Rawhide nightly composes, as well as Beta and Final releases, are to..." The whole "Rawhide gating" idea was part of this: the idea was/is that Rawhide composes should not be synced unless they meet the Basic criteria. The rawhide_compose_sync_to_mirrors greenwave policy exists to check this, but for whatever reason, the work of actually completing this effort so compose sync doesn't happen unless that policy passes was never done. We've talked about various concerns around this in the past (the technicalities of exactly how to implement it, and the concern that not enough composes actually meet the requirements so we'd wind up with few composes synced and a big disconnect between what's in the repos and what's in Koji), but the *idea* has been there all along and I agree with Neal that it was tied up with 'no more Alphas'. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
Dnia Sun, Jun 20, 2021 at 10:39:34AM +0200, Miro Hrončok napisał(a): > On 19. 06. 21 22:18, Fabio Valentini wrote: > > The following things should still be allowed: > > - releng and SIGs submitting scripted mass rebuilds (no actual package > > changes, triggered by a person) > > - bots submitting rawhide builds for ELN (no package change, just > > built for different buildroot) > > Other random ideas of what should be allowed: > > - a bot rebuilds packages automatically (bump-only or no changes), when > dependencies change We supposedly have it in Fedora already: https://fedoraproject.org/wiki/Koschei but it never worked for me. -- Tomasz Torcz “If you try to upissue this patchset I shall be seeking to...@pipebreaker.pl an IP-routable hand grenade.” — Andrew Morton (LKML) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On Sun, 2021-06-20 at 07:29 -0400, Neal Gompa wrote: > > If you want the features of Rawhide for CI, then we should talk about > enabling this in more places. For example, there's no technical reason > we couldn't do OpenQA runs for packages in COPR. There are serious > consequences to shipping packages into Rawhide, not the least of which > is that it locks an NVR forever into Koji and ships it to people using > Rawhide as a daily driver. Also having the ability to have COPR > auto-rebuild as stuff changes in the build root makes the CI case work > better, and that's something we literally can't do in Koji (by policy > and by design). I mean, there's no need for COPR to be involved. Rawhide runs through Bodhi these days. We could configure the package(s) concerned to gate the updates on testing. One major stumbling block there is that the openQA tests Peter values only run on nightly composes, and those only pull in stable packages. But it's not like we couldn't engineer our way around that if we wanted to. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On Sun, Jun 20, 2021 at 10:55:00AM +0200, Fabio Valentini wrote: > On Sun, Jun 20, 2021 at 10:45 AM Miro Hrončok wrote: > > > > I think this is a good idea. This particular bot has a history of > > misbehavior > > and rather than banning all the well behaving bots (that be definition we > > don't > > even know about, because they behave good), we should disable this > > particular one. > > > > Rather than "no bots allowed" policy, we might need a "bots that violate our > > policies and guidelines or have a tendency to break things will be disabled > > until fixed" policy. > > Yeah, that works for me too. Though I wouldn't want to make this a > special case and create an actual policy for this instead, that we > could point to when something like this happens. > > I also think that I probably was not clear in my original message. > > Any builds that are triggered by an actual human action, like > - scripted (mass) rebuilds with no *Version* changes, > - automatic builds after human-approved PRs, > - etc. > are of course exempt, because the action of an actual human being > triggered them. > > The only thing I *don't* want is: Bots submitting builds for new > *Versions*, without human interaction. So... what about new versions of clamav-data? I'm pretty sure that's OK to bot-ize. Then what about new minor version of thousands of rust packages? If we have %check sections and CI and gating, the act of releasing the update as a packager is not much different than a what a bot would do. Right now we mostly don't automatize this, but I think we could in the future. > Regarding Zbyszek's point: > > > Second, I think the guideline is simply wrong. As counterexamples, we > > currently have python3.10beta2 in rawhide, systemd-249-rc1, and > > kernel-5.13.0-0.rc6. Pushing pre-release vesions of low-level packages > > is a crucial part of development of the distro and collaboration with > > upstream projects and language ecosystems. > > There's actually already an exemption in the Updates Policy for those cases. > And I don't have any problem with those, because those builds are > prepared, built, tested, and shepherded by actual humans, instead of > created by a bot that just throws them at the wall to see what sticks > and what doesn't. Oh, OK. I didn't look at the policy page, just went by the quote provided. > If you look at bodhi updates for rhcontainerbot it's pretty obvious > that nobody even looks at the updates that are created for those > builds: > https://bodhi.fedoraproject.org/updates/?search==testing=rhcontainerbot > It looks like any build that receives -1 karma or fails gating tests > will just be stuck in "testing" until obsoleted by the next automated > build for that package. Yeah. I'm looking a the original ticket in https://pagure.io/fesco/issue/2228, and I think this was a mistake. We shouldn't have approved a bot that packages snapshot commits for rawhide. In the discussion, we talked about load on the infra, and ability to contact the maintainers, and even cooperation withb packit, but somehow the question whether we want this at all didn't come up. I guess the effort to make rawhide palatable hadn't really sunk in deep enough back then ;) @lsm5, would you be willing to adjust the bot to only package actual releases? Also, what is going on with the version jumps? Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On Sun, Jun 20, 2021 at 08:37:03AM -0500, Michael Catanzaro wrote: > On Sun, Jun 20 2021 at 07:29:16 AM -0400, Neal Gompa > wrote: > >Most of our rules are designed to make sure there's someone ultimately > >responsible for everything going into Fedora. Unfortunately, bots are > >the opposite of that, because there's no one to reach to stop bad > >behavior when it happens. > Hm, this seems pretty simple to solve though, right? Allow bots to > submit updates on behalf of packagers, but not with their own bot > FAS accounts. Let's not throw out the baby with the bath water. A human *is* responsible and known. When a bot account is given permission, we make sure that there's a known human behind the account. Things are no other in this particular case, see the original ticket [1]. Actually, if the bot were using their human's account, things would be *less* transparent. By using a separate account, we are making it clear that this update stream is made by this particular bot (as opposed to e.g. some human occasionally using a script to release some updates). [1] https://pagure.io/fesco/issue/2228 > This would be like how GNOME package updates currently > work, where a bot does the hard work but a human is ultimately > responsible (and subscribed to each bodhi update, so feedback will > at least not be completely missed). The line can be a big hazy, but I'd say that if: - a human is just using a script or even a some program to fire off the update — this particular person's account must be used. - some bot prepares the update, but a human still need to make the final step and may or may not publish the update: probably better to do it using this person's account. - the bot is set up once and then keeps releasing updating until stopped, and may be managed by multiple people — a separate bot account is preferable. Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On Sun, Jun 20 2021 at 07:29:16 AM -0400, Neal Gompa wrote: Most of our rules are designed to make sure there's someone ultimately responsible for everything going into Fedora. Unfortunately, bots are the opposite of that, because there's no one to reach to stop bad behavior when it happens. Hm, this seems pretty simple to solve though, right? Allow bots to submit updates on behalf of packagers, but not with their own bot FAS accounts. This would be like how GNOME package updates currently work, where a bot does the hard work but a human is ultimately responsible (and subscribed to each bodhi update, so feedback will at least not be completely missed). ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
Hi, On Sun, Jun 20, 2021 at 1:30 PM Neal Gompa wrote: > > On Sun, Jun 20, 2021 at 7:10 AM Peter Robinson wrote: > > > > > With things like [0] (TL;DR: bots submitting broken builds to rawhide) > > > becoming a more regular occurrence, I propose that we extend the > > > existing Updates Policy [1] to make it explicit that bots are not > > > allowed to submit builds / updates - even to rawhide - unattended: > > > "Rawhide is not your CI environment." > > > > I don't think that's good for either the packages that are using bots > > or for Fedora as a whole. > > > > While the bots certainly don't always get it right nor do the humans > > and I would garner that there's a lot more problems introduced in this > > way by humans, over the years I've spent considerable time fixing > > issues like this introduced by humans. Do we ban humans too? Do we ban > > RHBZ bots as well because they don't always get it right either? > > Ultimately automation is hard, but ultimately I feel it generally gets > > it wrong no less than humans do, and generally in a consistent way at > > least. > > > > > Currently, the Updates Policy states: > > > > > > - packagers must verify that no known broken builds are pushed, > > > - packagers must announce ABI and API changes once week in advance, > > > - packagers must not push pre-release versions of low-level packages. > > > > > > While it is debatable whether podman + friends + > > > container-stuff-dependencies count as "low-level" packages, they *are* > > > installed by default in Workstation. I think it is clear that by using > > > a bot to automatically push pre-release snapshots as rawhide updates, > > > the first two requirements CANNOT be met. > > > > > > I would like to make this conflict explicit and add a statement like > > > this to the Updates Policy: "Automated systems / bots are not allowed > > > to submit new builds for inclusion into Fedora without the involvement > > > of a packager." > > > > Please no. > > > > > The following things should still be allowed: > > > - releng and SIGs submitting scripted mass rebuilds (no actual package > > > changes, triggered by a person) > > > - bots submitting rawhide builds for ELN (no package change, just > > > built for different buildroot) > > > > > > The following should be explicitly banned: > > > - bots submitting new, non-scratch snapshot builds of software to > > > rawhide unattended (often leading to broken versions, versioning > > > snafus, or blatant errors leading to package downgrades, as it > > > happened today [0]) > > > > The problem with this is we will likely drive away contributors, > > packaging takes a lot of time, especially if you've got a stack of > > packages that are part of a group. > > > > Over the years there's been a bunch of effort all over the place to > > automate some of this, from tito, the upstream monitoring project > > thing (the new hottness or what ever it's called), packit and the rh > > container bot. None of them are perfect and all have their quirks. > > > > I would much sooner an officially support/ed/sanctioned auto packaging > > bot that upstreams can integrate into their projects to do automated > > packaging so they can get on and do more useful work either upstream > > or within the project. With something that is a "one true automated > > way" it would at least be consistent in the results. > > > > > There is already a requirement that no packager should submit builds > > > that are never intended to go "stable", and I see this as a similar > > > requirement - since those snapshot builds are presumably only done for > > > automated CI purposes without the intention of them ever reaching > > > stable Fedora releases, where they are superseded by packager-created > > > manual builds of those packages - but leaving Rawhide with unstable, > > > bot-created snapshot builds. > > > > I find the automated builds, in conjunction with OpenQA testing, very > > useful for IoT as they've caught a number of issues with upstream > > which likely wouldn't have been caught before the release goes stable > > and hence gets into stable IoT releases, it doesn't catch everything > > but the more it catches the better off the project is as a whole. > > > > I would much sooner the effort here goes into a sanctioned package > > build bot than trying to ban things TBH. > > > > Most of our rules are designed to make sure there's someone ultimately > responsible for everything going into Fedora. Unfortunately, bots are > the opposite of that, because there's no one to reach to stop bad > behavior when it happens. > > Rawhide is still not your CI environment. By the way I think we need to reconsider this statement. I think I understand the intent, but the statement itself is misleading, if not wrong, when taken out of context. For a person outside of the bubble it is hard to understand and it seems to be contradicting with the idea of Rawhide being the integrated development branch of the Fedora Linux
Re: RFC: Banning bots from submitting automated koji builds
On Sun, Jun 20, 2021 at 12:31 PM Neal Gompa wrote: > > On Sun, Jun 20, 2021 at 7:10 AM Peter Robinson wrote: > > > > > With things like [0] (TL;DR: bots submitting broken builds to rawhide) > > > becoming a more regular occurrence, I propose that we extend the > > > existing Updates Policy [1] to make it explicit that bots are not > > > allowed to submit builds / updates - even to rawhide - unattended: > > > "Rawhide is not your CI environment." > > > > I don't think that's good for either the packages that are using bots > > or for Fedora as a whole. > > > > While the bots certainly don't always get it right nor do the humans > > and I would garner that there's a lot more problems introduced in this > > way by humans, over the years I've spent considerable time fixing > > issues like this introduced by humans. Do we ban humans too? Do we ban > > RHBZ bots as well because they don't always get it right either? > > Ultimately automation is hard, but ultimately I feel it generally gets > > it wrong no less than humans do, and generally in a consistent way at > > least. > > > > > Currently, the Updates Policy states: > > > > > > - packagers must verify that no known broken builds are pushed, > > > - packagers must announce ABI and API changes once week in advance, > > > - packagers must not push pre-release versions of low-level packages. > > > > > > While it is debatable whether podman + friends + > > > container-stuff-dependencies count as "low-level" packages, they *are* > > > installed by default in Workstation. I think it is clear that by using > > > a bot to automatically push pre-release snapshots as rawhide updates, > > > the first two requirements CANNOT be met. > > > > > > I would like to make this conflict explicit and add a statement like > > > this to the Updates Policy: "Automated systems / bots are not allowed > > > to submit new builds for inclusion into Fedora without the involvement > > > of a packager." > > > > Please no. > > > > > The following things should still be allowed: > > > - releng and SIGs submitting scripted mass rebuilds (no actual package > > > changes, triggered by a person) > > > - bots submitting rawhide builds for ELN (no package change, just > > > built for different buildroot) > > > > > > The following should be explicitly banned: > > > - bots submitting new, non-scratch snapshot builds of software to > > > rawhide unattended (often leading to broken versions, versioning > > > snafus, or blatant errors leading to package downgrades, as it > > > happened today [0]) > > > > The problem with this is we will likely drive away contributors, > > packaging takes a lot of time, especially if you've got a stack of > > packages that are part of a group. > > > > Over the years there's been a bunch of effort all over the place to > > automate some of this, from tito, the upstream monitoring project > > thing (the new hottness or what ever it's called), packit and the rh > > container bot. None of them are perfect and all have their quirks. > > > > I would much sooner an officially support/ed/sanctioned auto packaging > > bot that upstreams can integrate into their projects to do automated > > packaging so they can get on and do more useful work either upstream > > or within the project. With something that is a "one true automated > > way" it would at least be consistent in the results. > > > > > There is already a requirement that no packager should submit builds > > > that are never intended to go "stable", and I see this as a similar > > > requirement - since those snapshot builds are presumably only done for > > > automated CI purposes without the intention of them ever reaching > > > stable Fedora releases, where they are superseded by packager-created > > > manual builds of those packages - but leaving Rawhide with unstable, > > > bot-created snapshot builds. > > > > I find the automated builds, in conjunction with OpenQA testing, very > > useful for IoT as they've caught a number of issues with upstream > > which likely wouldn't have been caught before the release goes stable > > and hence gets into stable IoT releases, it doesn't catch everything > > but the more it catches the better off the project is as a whole. > > > > I would much sooner the effort here goes into a sanctioned package > > build bot than trying to ban things TBH. > > > > Most of our rules are designed to make sure there's someone ultimately > responsible for everything going into Fedora. Unfortunately, bots are > the opposite of that, because there's no one to reach to stop bad > behavior when it happens. So that should be fixed so that each bot has a contact so that problems have means of escalation. > Rawhide is still not your CI environment. Years ago, we got rid of > alphas for the express purpose to stabilize Rawhide into alpha > quality. Stuff like this degrades the quality of Rawhide because they > make the assumption that nobody cares about the quality of Rawhide. Being one of the people
Re: RFC: Banning bots from submitting automated koji builds
On Sun, Jun 20, 2021 at 7:10 AM Peter Robinson wrote: > > > With things like [0] (TL;DR: bots submitting broken builds to rawhide) > > becoming a more regular occurrence, I propose that we extend the > > existing Updates Policy [1] to make it explicit that bots are not > > allowed to submit builds / updates - even to rawhide - unattended: > > "Rawhide is not your CI environment." > > I don't think that's good for either the packages that are using bots > or for Fedora as a whole. > > While the bots certainly don't always get it right nor do the humans > and I would garner that there's a lot more problems introduced in this > way by humans, over the years I've spent considerable time fixing > issues like this introduced by humans. Do we ban humans too? Do we ban > RHBZ bots as well because they don't always get it right either? > Ultimately automation is hard, but ultimately I feel it generally gets > it wrong no less than humans do, and generally in a consistent way at > least. > > > Currently, the Updates Policy states: > > > > - packagers must verify that no known broken builds are pushed, > > - packagers must announce ABI and API changes once week in advance, > > - packagers must not push pre-release versions of low-level packages. > > > > While it is debatable whether podman + friends + > > container-stuff-dependencies count as "low-level" packages, they *are* > > installed by default in Workstation. I think it is clear that by using > > a bot to automatically push pre-release snapshots as rawhide updates, > > the first two requirements CANNOT be met. > > > > I would like to make this conflict explicit and add a statement like > > this to the Updates Policy: "Automated systems / bots are not allowed > > to submit new builds for inclusion into Fedora without the involvement > > of a packager." > > Please no. > > > The following things should still be allowed: > > - releng and SIGs submitting scripted mass rebuilds (no actual package > > changes, triggered by a person) > > - bots submitting rawhide builds for ELN (no package change, just > > built for different buildroot) > > > > The following should be explicitly banned: > > - bots submitting new, non-scratch snapshot builds of software to > > rawhide unattended (often leading to broken versions, versioning > > snafus, or blatant errors leading to package downgrades, as it > > happened today [0]) > > The problem with this is we will likely drive away contributors, > packaging takes a lot of time, especially if you've got a stack of > packages that are part of a group. > > Over the years there's been a bunch of effort all over the place to > automate some of this, from tito, the upstream monitoring project > thing (the new hottness or what ever it's called), packit and the rh > container bot. None of them are perfect and all have their quirks. > > I would much sooner an officially support/ed/sanctioned auto packaging > bot that upstreams can integrate into their projects to do automated > packaging so they can get on and do more useful work either upstream > or within the project. With something that is a "one true automated > way" it would at least be consistent in the results. > > > There is already a requirement that no packager should submit builds > > that are never intended to go "stable", and I see this as a similar > > requirement - since those snapshot builds are presumably only done for > > automated CI purposes without the intention of them ever reaching > > stable Fedora releases, where they are superseded by packager-created > > manual builds of those packages - but leaving Rawhide with unstable, > > bot-created snapshot builds. > > I find the automated builds, in conjunction with OpenQA testing, very > useful for IoT as they've caught a number of issues with upstream > which likely wouldn't have been caught before the release goes stable > and hence gets into stable IoT releases, it doesn't catch everything > but the more it catches the better off the project is as a whole. > > I would much sooner the effort here goes into a sanctioned package > build bot than trying to ban things TBH. > Most of our rules are designed to make sure there's someone ultimately responsible for everything going into Fedora. Unfortunately, bots are the opposite of that, because there's no one to reach to stop bad behavior when it happens. Rawhide is still not your CI environment. Years ago, we got rid of alphas for the express purpose to stabilize Rawhide into alpha quality. Stuff like this degrades the quality of Rawhide because they make the assumption that nobody cares about the quality of Rawhide. If you want the features of Rawhide for CI, then we should talk about enabling this in more places. For example, there's no technical reason we couldn't do OpenQA runs for packages in COPR. There are serious consequences to shipping packages into Rawhide, not the least of which is that it locks an NVR forever into Koji and ships it to people using Rawhide as a
Re: RFC: Banning bots from submitting automated koji builds
> With things like [0] (TL;DR: bots submitting broken builds to rawhide) > becoming a more regular occurrence, I propose that we extend the > existing Updates Policy [1] to make it explicit that bots are not > allowed to submit builds / updates - even to rawhide - unattended: > "Rawhide is not your CI environment." I don't think that's good for either the packages that are using bots or for Fedora as a whole. While the bots certainly don't always get it right nor do the humans and I would garner that there's a lot more problems introduced in this way by humans, over the years I've spent considerable time fixing issues like this introduced by humans. Do we ban humans too? Do we ban RHBZ bots as well because they don't always get it right either? Ultimately automation is hard, but ultimately I feel it generally gets it wrong no less than humans do, and generally in a consistent way at least. > Currently, the Updates Policy states: > > - packagers must verify that no known broken builds are pushed, > - packagers must announce ABI and API changes once week in advance, > - packagers must not push pre-release versions of low-level packages. > > While it is debatable whether podman + friends + > container-stuff-dependencies count as "low-level" packages, they *are* > installed by default in Workstation. I think it is clear that by using > a bot to automatically push pre-release snapshots as rawhide updates, > the first two requirements CANNOT be met. > > I would like to make this conflict explicit and add a statement like > this to the Updates Policy: "Automated systems / bots are not allowed > to submit new builds for inclusion into Fedora without the involvement > of a packager." Please no. > The following things should still be allowed: > - releng and SIGs submitting scripted mass rebuilds (no actual package > changes, triggered by a person) > - bots submitting rawhide builds for ELN (no package change, just > built for different buildroot) > > The following should be explicitly banned: > - bots submitting new, non-scratch snapshot builds of software to > rawhide unattended (often leading to broken versions, versioning > snafus, or blatant errors leading to package downgrades, as it > happened today [0]) The problem with this is we will likely drive away contributors, packaging takes a lot of time, especially if you've got a stack of packages that are part of a group. Over the years there's been a bunch of effort all over the place to automate some of this, from tito, the upstream monitoring project thing (the new hottness or what ever it's called), packit and the rh container bot. None of them are perfect and all have their quirks. I would much sooner an officially support/ed/sanctioned auto packaging bot that upstreams can integrate into their projects to do automated packaging so they can get on and do more useful work either upstream or within the project. With something that is a "one true automated way" it would at least be consistent in the results. > There is already a requirement that no packager should submit builds > that are never intended to go "stable", and I see this as a similar > requirement - since those snapshot builds are presumably only done for > automated CI purposes without the intention of them ever reaching > stable Fedora releases, where they are superseded by packager-created > manual builds of those packages - but leaving Rawhide with unstable, > bot-created snapshot builds. I find the automated builds, in conjunction with OpenQA testing, very useful for IoT as they've caught a number of issues with upstream which likely wouldn't have been caught before the release goes stable and hence gets into stable IoT releases, it doesn't catch everything but the more it catches the better off the project is as a whole. I would much sooner the effort here goes into a sanctioned package build bot than trying to ban things TBH. Peter ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On Sun, Jun 20, 2021 at 10:45 AM Miro Hrončok wrote: > > I think this is a good idea. This particular bot has a history of misbehavior > and rather than banning all the well behaving bots (that be definition we > don't > even know about, because they behave good), we should disable this particular > one. > > Rather than "no bots allowed" policy, we might need a "bots that violate our > policies and guidelines or have a tendency to break things will be disabled > until fixed" policy. Yeah, that works for me too. Though I wouldn't want to make this a special case and create an actual policy for this instead, that we could point to when something like this happens. I also think that I probably was not clear in my original message. Any builds that are triggered by an actual human action, like - scripted (mass) rebuilds with no *Version* changes, - automatic builds after human-approved PRs, - etc. are of course exempt, because the action of an actual human being triggered them. The only thing I *don't* want is: Bots submitting builds for new *Versions*, without human interaction. Regarding Zbyszek's point: > Second, I think the guideline is simply wrong. As counterexamples, we > currently have python3.10beta2 in rawhide, systemd-249-rc1, and > kernel-5.13.0-0.rc6. Pushing pre-release vesions of low-level packages > is a crucial part of development of the distro and collaboration with > upstream projects and language ecosystems. There's actually already an exemption in the Updates Policy for those cases. And I don't have any problem with those, because those builds are prepared, built, tested, and shepherded by actual humans, instead of created by a bot that just throws them at the wall to see what sticks and what doesn't. If you look at bodhi updates for rhcontainerbot it's pretty obvious that nobody even looks at the updates that are created for those builds: https://bodhi.fedoraproject.org/updates/?search==testing=rhcontainerbot It looks like any build that receives -1 karma or fails gating tests will just be stuck in "testing" until obsoleted by the next automated build for that package. Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On 20. 06. 21 9:39, Zbigniew Jędrzejewski-Szmek wrote: I think we should just disable this particular bot until it fixed. We should also clarify/update the Update Guidelines so that undesirable updates are disallowed, no matter if submitted by a bot or a human. I think this is a good idea. This particular bot has a history of misbehavior and rather than banning all the well behaving bots (that be definition we don't even know about, because they behave good), we should disable this particular one. Rather than "no bots allowed" policy, we might need a "bots that violate our policies and guidelines or have a tendency to break things will be disabled until fixed" policy. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On 19. 06. 21 22:18, Fabio Valentini wrote: The following things should still be allowed: - releng and SIGs submitting scripted mass rebuilds (no actual package changes, triggered by a person) - bots submitting rawhide builds for ELN (no package change, just built for different buildroot) Other random ideas of what should be allowed: - a human approves a pull request, bot merges it and builds - a bot rebuilds packages automatically (bump-only or no changes), when dependencies change -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: RFC: Banning bots from submitting automated koji builds
On Sat, Jun 19, 2021 at 10:18:45PM +0200, Fabio Valentini wrote: > Hi everybody, > > With things like [0] (TL;DR: bots submitting broken builds to rawhide) > becoming a more regular occurrence, I propose that we extend the > existing Updates Policy [1] to make it explicit that bots are not > allowed to submit builds / updates - even to rawhide - unattended: > "Rawhide is not your CI environment." I agree with the goal of avoiding random snapshot packages in rawhide, but I disagree that banning bots is the way to achieve that. In this particular case, the problem is that the bot is either buggy or misconfigured or both. (I hope that the discussion in the other thread can clarify what went wrong.) A blanket ban for automated updates doesn't directly address the problem (because even a human could push the same broken snapshots to rawhide), and will constrain future development. [For example, let's say we make a bot that looks at the huge Python3.10 FTBFS/FTI dependency tree, watches bugzilla events, and automatically fires off a no-change rebuild of packages for which all depended-on have been closed. It would be forbidden by this policy, even though it wouldn't cause those kinds of problems we have now.] I think we should just disable this particular bot until it fixed. We should also clarify/update the Update Guidelines so that undesirable updates are disallowed, no matter if submitted by a bot or a human. > Currently, the Updates Policy states: > > - packagers must verify that no known broken builds are pushed, > - packagers must announce ABI and API changes once week in advance, > - packagers must not push pre-release versions of low-level packages. > > While it is debatable whether podman + friends + > container-stuff-dependencies count as "low-level" packages, they *are* > installed by default in Workstation. I think it is clear that by using > a bot to automatically push pre-release snapshots as rawhide updates, > the first two requirements CANNOT be met. First, I think podman fits the intent of this guideline, even if not the letter. Second, I think the guideline is simply wrong. As counterexamples, we currently have python3.10beta2 in rawhide, systemd-249-rc1, and kernel-5.13.0-0.rc6. Pushing pre-release vesions of low-level packages is a crucial part of development of the distro and collaboration with upstream projects and language ecosystems. Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
RFC: Banning bots from submitting automated koji builds
Hi everybody, With things like [0] (TL;DR: bots submitting broken builds to rawhide) becoming a more regular occurrence, I propose that we extend the existing Updates Policy [1] to make it explicit that bots are not allowed to submit builds / updates - even to rawhide - unattended: "Rawhide is not your CI environment." Currently, the Updates Policy states: - packagers must verify that no known broken builds are pushed, - packagers must announce ABI and API changes once week in advance, - packagers must not push pre-release versions of low-level packages. While it is debatable whether podman + friends + container-stuff-dependencies count as "low-level" packages, they *are* installed by default in Workstation. I think it is clear that by using a bot to automatically push pre-release snapshots as rawhide updates, the first two requirements CANNOT be met. I would like to make this conflict explicit and add a statement like this to the Updates Policy: "Automated systems / bots are not allowed to submit new builds for inclusion into Fedora without the involvement of a packager." The following things should still be allowed: - releng and SIGs submitting scripted mass rebuilds (no actual package changes, triggered by a person) - bots submitting rawhide builds for ELN (no package change, just built for different buildroot) The following should be explicitly banned: - bots submitting new, non-scratch snapshot builds of software to rawhide unattended (often leading to broken versions, versioning snafus, or blatant errors leading to package downgrades, as it happened today [0]) There is already a requirement that no packager should submit builds that are never intended to go "stable", and I see this as a similar requirement - since those snapshot builds are presumably only done for automated CI purposes without the intention of them ever reaching stable Fedora releases, where they are superseded by packager-created manual builds of those packages - but leaving Rawhide with unstable, bot-created snapshot builds. Fabio [0]: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/LYCPRPFTAED4OA7FVCHVHXP6GWVGGEFI/ [1]: https://docs.fedoraproject.org/en-US/fesco/Updates_Policy/#_rawhide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
