Re: What is this update waiting on?
On Wed, 2023-07-12 at 15:18 -0700, Kevin Fenzi wrote: > On Wed, Jul 12, 2023 at 01:57:16PM -0700, Adam Williamson wrote: > > On Wed, 2023-07-12 at 21:45 +0100, Richard W.M. Jones wrote: > > > https://bodhi.fedoraproject.org/updates/FEDORA-2023-129c8286a0 > > > > > > This one seems a bit different. There seem to be no tests running. > > > The test gating status is "ignored" (I didn't do that). And otherwise > > > it seems to be doing nothing. > > > > Ignored means there's no automated testing or gating involved; none of > > the gating policies apply to the update. > > > > koji list-history shows: > > > > Wed Jul 12 19:07:38 2023 why3-1.6.0-1.fc39 tagged into > > f39-updates-candidate by rjones [still active] > > Wed Jul 12 19:07:39 2023 why3-1.6.0-1.fc39 tagged into f39-signing-pending > > by bodhi [still active] > > > > which implies it's waiting for robosignatory to notice it's pending > > signing, and sign it (at which point bodhi takes over again). I don't > > know what the hold-up with robosignatory is. Kevin may. > > I'm not sure. It seems like it somehow didn't see the message about the > signing-pending tag. > > Jul 12 19:07:38 autosign01... [robosignatory.tagconsumer INFO] Build > why3-1.6.0-1.fc39 (2231707) tagged into f39-updates-candidate on primary > > I've retagged it and it's moving now. > > I'll try and see if I can figure out why that message didn't get seen. I think we were looking at another update which had a delay like this last week, weren't we? I wonder if something systematic is going on... -- Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @ad...@fosstodon.org https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: What is this update waiting on?
On Wed, Jul 12, 2023 at 01:57:16PM -0700, Adam Williamson wrote: > On Wed, 2023-07-12 at 21:45 +0100, Richard W.M. Jones wrote: > > https://bodhi.fedoraproject.org/updates/FEDORA-2023-129c8286a0 > > > > This one seems a bit different. There seem to be no tests running. > > The test gating status is "ignored" (I didn't do that). And otherwise > > it seems to be doing nothing. > > Ignored means there's no automated testing or gating involved; none of > the gating policies apply to the update. > > koji list-history shows: > > Wed Jul 12 19:07:38 2023 why3-1.6.0-1.fc39 tagged into f39-updates-candidate > by rjones [still active] > Wed Jul 12 19:07:39 2023 why3-1.6.0-1.fc39 tagged into f39-signing-pending by > bodhi [still active] > > which implies it's waiting for robosignatory to notice it's pending > signing, and sign it (at which point bodhi takes over again). I don't > know what the hold-up with robosignatory is. Kevin may. I'm not sure. It seems like it somehow didn't see the message about the signing-pending tag. Jul 12 19:07:38 autosign01... [robosignatory.tagconsumer INFO] Build why3-1.6.0-1.fc39 (2231707) tagged into f39-updates-candidate on primary I've retagged it and it's moving now. I'll try and see if I can figure out why that message didn't get seen. kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: What is this update waiting on?
On Wed, 2023-07-12 at 21:45 +0100, Richard W.M. Jones wrote: > https://bodhi.fedoraproject.org/updates/FEDORA-2023-129c8286a0 > > This one seems a bit different. There seem to be no tests running. > The test gating status is "ignored" (I didn't do that). And otherwise > it seems to be doing nothing. Ignored means there's no automated testing or gating involved; none of the gating policies apply to the update. koji list-history shows: Wed Jul 12 19:07:38 2023 why3-1.6.0-1.fc39 tagged into f39-updates-candidate by rjones [still active] Wed Jul 12 19:07:39 2023 why3-1.6.0-1.fc39 tagged into f39-signing-pending by bodhi [still active] which implies it's waiting for robosignatory to notice it's pending signing, and sign it (at which point bodhi takes over again). I don't know what the hold-up with robosignatory is. Kevin may. -- Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @ad...@fosstodon.org https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: What is this update waiting on?
On Wed, Jul 12, 2023 at 10:46 PM Richard W.M. Jones wrote: > > > https://bodhi.fedoraproject.org/updates/FEDORA-2023-129c8286a0 > > This one seems a bit different. There seem to be no tests running. > The test gating status is "ignored" (I didn't do that). And otherwise > it seems to be doing nothing. The builds in this update are not signed yet (no "Signed: ✓" under "Metadata" in the sidebar, compared to other updates). Not sure why this is the case, as other updates that were submitted later than this one are already signed and "stable", but it's correctly tagged as f39-signing-pending. Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: What is this update waiting on?
https://bodhi.fedoraproject.org/updates/FEDORA-2023-129c8286a0 This one seems a bit different. There seem to be no tests running. The test gating status is "ignored" (I didn't do that). And otherwise it seems to be doing nothing. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: What is this update waiting on?
On Wed, 2023-07-12 at 18:37 +0200, Fabio Valentini wrote: > On Wed, Jul 12, 2023 at 6:26 PM Richard W.M. Jones wrote: > > > > > > https://bodhi.fedoraproject.org/updates/FEDORA-2023-8d5b08b005 > > > > I don't understand what this update is waiting on / why it cannot go > > to Fedora immediately. > > > > I waived the tests. > > > > There are still apparently 5 tests "running", but clicking through to > > the Automated Tests tab shows only 4. There's no way to show the > > status of these 4 tests, like are they running now, are they waiting > > for something, why do they need to run at all if I waived them? > > This might be related to the recent enablement of OpenQA tests for > critpath packages in rawhide? I'm not sure whether you can waive > OpenQA tests that are still running, or whether you can only waive > actual failures once they happen. > > I'd just wait a bit. With updates this large, even signing all the > packages takes a while, and then OpenQA tests run ... > That said, it's also not *that* unusual to see rawhide updates be in > "testing" state for more than an hour now (at least this also happened > to one of my recent updates). Probably depends on how large the queue > of OpenQA tests is. A test time of about 90 minutes is the minimum (for updates that require the tests dicussed below anyway), because two of the gating tests are this process: * Build a live image (using the update) * Run an install from that live image * Test the installed system boots for KDE and GNOME. Another is: * Deploy an N-1 FreeIPA server and client * Upgrade both server and client to N, with the update included * Check everything works All of these tests inherently take quite a long time, because building live images and running an install from them takes time, and because doing FreeIPA deployments and system upgrades takes time. The required time usually works out to be about 90 minutes. I can't really optimize these tests much beyond their current state. I believe they are important and necessary tests, so they are in the gating list and updates have to wait for them. This has been the case for stable releases for quite a long time at this point; it's only relatively "new" for Rawhide. There is another set of tests which takes even longer - build a Silverblue ostree, then build an ostree installer image from that ostree, then install it and test it works - but that set of tests is not currently in the gating set, so it does not need to complete before the update goes stable. I believe Fabio is correct that - as the system is currently designed - you can't effectively waive a running test. You can file the waiver, but I believe the gating status calculation only considers waivers for *failed* tests, not queued or running ones. Honestly, I think this is the right design; waiving is intended to mean "I have examined this failure and I'm very sure it was a false one", not "I cannot possibly wait for this test to complete!" However, the UI design around this could clearly be improved: if we aren't going to apply waivers to running/queued tests we probably shouldn't allow Bodhi to *file* them at that point, and if one somehow exists, Bodhi probably shouldn't indicate that it's a "live" waiver on the Automated Results tab. Here's a Bodhi issue on that: https://github.com/fedora-infra/bodhi/issues/5414 I appreciate that waiting can be a bit frustrating, but I believe it's a necessary compromise for the substantial increase in quality that gating on these tests can afford. I'd really hope folks can live with waiting (usually) 90 minutes before the update is tagged. The waiver messages Richard created gave the justification "I'll install and test the packages myself"; unless you're doing all the tests openQA does, including tests of image builds and installs, you're not necessarily going to find all the problems it would. In this case, it took a bit longer than 90 minutes - more like two hours. Looking into why, the FreeIPA upgrade test wasn't applied to this update (as it's not in the "relevant" critical path group), but the live install tests were (because the update is in the 'critical-path- compose' group, likely because of libguestfs I guess). KDE live build test took 1hr 14mins, the 'install and boot' test took 20mins 13secs, so that's only just over 90 mins. But there was a bit of a delay starting the tests, because the live build test started at 15:56 while the update was created at 15:29. It seems, lately, like there's sometimes a delay between an update being created and the message that triggers openQA to test it being published, and that's what happened in this case: the message that triggered the tests, https://apps.fedoraproject.org/datagrepper/v2/id?id=15d5bad4-32a5-47e0-abe3-17142c7ab86a_raw=true=extra-large , was published at 15:54, 25 minutes after the update was created. openQA can't start testing the update till that message is published. I'm not sure why they seem to be sometimes
Re: What is this update waiting on?
On Wed, Jul 12, 2023 at 6:37 PM Fabio Valentini wrote: > > On Wed, Jul 12, 2023 at 6:26 PM Richard W.M. Jones wrote: > > > > > > https://bodhi.fedoraproject.org/updates/FEDORA-2023-8d5b08b005 > > > > I don't understand what this update is waiting on / why it cannot go > > to Fedora immediately. > > > > I waived the tests. > > > > There are still apparently 5 tests "running", but clicking through to > > the Automated Tests tab shows only 4. There's no way to show the > > status of these 4 tests, like are they running now, are they waiting > > for something, why do they need to run at all if I waived them? > > This might be related to the recent enablement of OpenQA tests for > critpath packages in rawhide? I'm not sure whether you can waive > OpenQA tests that are still running, or whether you can only waive > actual failures once they happen. > > I'd just wait a bit. With updates this large, even signing all the > packages takes a while, and then OpenQA tests run ... > That said, it's also not *that* unusual to see rawhide updates be in > "testing" state for more than an hour now (at least this also happened > to one of my recent updates). Probably depends on how large the queue > of OpenQA tests is. Looks like the update was indeed waiting for the mandatory OpenQA tests. It was just pushed to stable, as soon as the last one of them passed. Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: What is this update waiting on?
On Wed, Jul 12, 2023 at 6:26 PM Richard W.M. Jones wrote: > > > https://bodhi.fedoraproject.org/updates/FEDORA-2023-8d5b08b005 > > I don't understand what this update is waiting on / why it cannot go > to Fedora immediately. > > I waived the tests. > > There are still apparently 5 tests "running", but clicking through to > the Automated Tests tab shows only 4. There's no way to show the > status of these 4 tests, like are they running now, are they waiting > for something, why do they need to run at all if I waived them? This might be related to the recent enablement of OpenQA tests for critpath packages in rawhide? I'm not sure whether you can waive OpenQA tests that are still running, or whether you can only waive actual failures once they happen. I'd just wait a bit. With updates this large, even signing all the packages takes a while, and then OpenQA tests run ... That said, it's also not *that* unusual to see rawhide updates be in "testing" state for more than an hour now (at least this also happened to one of my recent updates). Probably depends on how large the queue of OpenQA tests is. Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
