On Mon, Jul 15, 2024 at 10:39:37AM +0200, Cristian Le via devel wrote: > Hi Zbyszek, > > On 2024/07/14 20:04, Zbigniew Jędrzejewski-Szmek wrote: > > I'm looking for a solution which doesn't just skip the installability > > tests altogether. > > On PRs with zuul or FedoraCI automation, the same instability tests that are > done for Bodhi are performed. But what would help is to make these tests as > required to pass unless they are manually waved. Manually that can be done > by setting `gating.yaml`. There was some discussion on making some of these > tests as gating by default. > > Another issue specific to installability is that the issue is often further > down the stream, particularly with the SELinux test. Definitely these need > to be tracked down and fixed.
I fully agree. But a test that just does 'dnf install rpms-from-update/*.rpm' will predicatably fail. > > A second problem is that when the tests fail, it's just soooo hard to > > find out why they failed.From the bodhi status page, one has to > > go to the Jenkins status page, guess that it's useful to look at > > Console Output, scroll over a few pages of incomrehensible JSON > > gibberish, guess that it's worth clicking on Testing Farm Artifacts URL, > > click that, scroll pages of output to see > > "guest setup failed: Test environment installation failed: Install > > packages". > > Weird, when the test is finished, you should have only the final > testing-farm results page. Here's an example [1]. Maybe in your case it > encountered an internal failure? > > [1]: https://bodhi.fedoraproject.org/updates/FEDORA-2024-57f489c90d Maybe I'm doing things wrong. I'd be happy to learn. I do the following: 1. Look at the bodhi update page (https://bodhi.fedoraproject.org/updates/FEDORA-2024-3aafcac6a8) 2. Click on 'Automated Tests' (There seems to be no URL for the view. This means that it's always and extra click after every page load or reload.) 3. I click on one of the pinkish lines, e.g. the first one. (Another usability problem here is that the click open a new page in new tab/window. Why, oh why? I want to use left-click to open a link in the existing tab, and middle-click to open a new tab. The current UI breaks navigation.) 4. I switch to the new tab and see https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipeline/job/master/398487/ (BTW, I mentioned unrelated scary-looking warnings in my OP. Here: The following steps that have been detected may have insecure interpolation of sensitive variables (click here for an explanation): httpRequest: [TESTING_FARM_API_KEY] ) 5. Click on 'Console Output' 6. Click on 'Testing Farm Artifacts URL: https://artifacts.dev.testing-farm.io/25316385-50d8-42b4-b4c1-3eff059034eb' 7. Click on 'build installation' (https://artifacts.dev.testing-farm.io/25316385-50d8-42b4-b4c1-3eff059034eb/guest-setup-09b7edc6-0b7e-431b-ae68-afac2527fbb1/artifact-installation-09b7edc6-0b7e-431b-ae68-afac2527fbb1) 8. Click on 60-Install-packages.txt So 8 steps to get to the actual result… Zbyszek -- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
