On Wed, Apr 28, 2021 at 6:56 PM Adam Williamson <adamw...@fedoraproject.org> wrote:
> We do sort of have this already, because openQA tests all updates, and > one of the tests it runs on them is an upgrade test. > > Now, the iptables thing is interesting, because the logs of that test > on an F34 update show the issue: > > 2021-04-28T04:30:39-0400 WARNING > Problem: cannot install both iptables-libs-1.8.7-3.fc34.x86_64 and > iptables-libs-1.8.7-6.fc34.x86_64 > - package iptables-1.8.7-3.fc34.x86_64 requires iptables-libs(x86-64) = > 1.8.7-3.fc34, but none of the providers can be installed > - cannot install the best update candidate for package > iptables-libs-1.8.5-6.fc33.x86_64 > - cannot install the best update candidate for package > iptables-1.8.5-6.fc33.x86_64 > > but it's only logged as a WARNING and does not prevent the upgrade > running. So the test passes. It doesn't even need to use `-- > allowerasing` so it doesn't register as a soft fail. > I assume that's because iptables from the main (fedora) repo are used instead. But the upgrade would fail if we added "--best" (I tested it). Here's a thought. What if we modified the openQA test to first try with "--best". If there's any error, log it as a softfail, and continue without it (and if if fails again, log it as a hard error). This way, we can catch *some* broken updates (it's still possible that one broken update obscures a second one and we'll not know about the latter - but the benefit is definitely there). If we did it this way, what exactly happens with the softfailed result? Does it disable Bodhi's autopush? Or does it entirely depend on QA noticing it soon enough? As for manual testing, what do you think about my previous proposal? > we could extend our upgrade testing matrices with two variants - one would test with stable updates, the other would test with updates-testing enabled
_______________________________________________ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
