Re: autopkgtest missed regression for kinetic
On Sun, May 15, 2022 at 05:46:49PM -0400, Jeremy Bicha wrote: > On Sun, May 15, 2022 at 4:26 PM Steve Langasek > wrote: > > > The test has never passed; it has had neutral results and failing results, > > > but by policy, proposed-migration does not treat neutral->fail as a > > > migration-blocking regression, only pass->fail. Neutral test results are > > > informational only. > > Having said this, I am just this moment looking at cases where britney is > > treating neutral->fail as a regression. I still believe the intended policy > > is as I described, but one way or another there appear to be some bugs here > > in the implementation ;) > devhelp's autopkgtest is marked as "superficial". In Debian, a passing > superficial autopkgtest won't speed up migration to Testing. But if > the test fails, it indicates that there is a serious bug and migration > to Testing is blocked. Please see the description of the intent of > "superficial": > https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst You're right, sorry for misremembering. So the other consideration is that proposed-migration, by necessity, has to cache a lot of information about the autopkgtest results. Once it has decided that a test is not a regression based on available results, it has no reason to revisit this. So in https://people.canonical.com/~ubuntu-archive/proposed-migration/log/kinetic/2022-05-10/22:14:21.log.gz we see: I: [2022-05-10T22:43:48+] - Fetched test result for devhelp/41.2-2/amd64 20220510_212456_139c1@ (triggers: ['webkit2gtk/2.36.1-1']): fail I: [2022-05-10T22:43:48+] - -> matches pending request devhelp/amd64 for trigger webkit2gtk/2.36.1-1 I: [2022-05-10T22:43:48+] - Checking for new results for devhelp/amd64 for trigger migration-reference/0 I: [2022-05-10T22:43:48+] - Requesting devhelp autopkgtest on amd64 to verify migration-reference/0 I: [2022-05-10T22:43:48+] - Updating pending requested tests in data/kinetic/state/autopkgtest-pending.json In the corresponding https://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses/kinetic/2022-05-10/22:14:21.yaml.xz this shows: devhelp/41.2-2: amd64: - RUNNING-REFERENCE - https://autopkgtest.ubuntu.com/results/autopkgtest-kinetic/kinetic/amd64/d/devhelp/20220510_212456_139c1@/log.gz - https://autopkgtest.ubuntu.com/packages/d/devhelp/kinetic/amd64 - null - https://autopkgtest.ubuntu.com/request.cgi?release=kinetic&arch=amd64&package=devhelp&trigger=webkit2gtk%2F2.36.1-1 i.e. the test failed but we're checking if the baseline failed. In the next run, https://people.canonical.com/~ubuntu-archive/proposed-migration/log/kinetic/2022-05-10/23:59:06.log.gz we have: I: [2022-05-11T00:10:00+] - Checking for new results for devhelp/amd64 for trigger migration-reference/0 I: [2022-05-11T00:10:00+] - Fetched test result for devhelp/41.2-2/amd64 20220510_230253_250a6@ (triggers: ['migration-reference/0']): neutral I: [2022-05-11T00:10:00+] - -> matches pending request devhelp/amd64 for trigger migration-reference/0 and: devhelp/41.2-2: amd64: - ALWAYSFAIL - https://autopkgtest.ubuntu.com/results/autopkgtest-kinetic/kinetic/amd64/d/devhelp/20220510_212456_139c1@/log.gz - https://autopkgtest.ubuntu.com/packages/d/devhelp/kinetic/amd64 - null - null I'm not sure why this is. It is similar to other issues we've had in the past where the order in which results are seen cause britney to see failures as regressions when they're actually progressions, but the underlying cause here may be different. That seems pretty obviously a bug; we shouldn't request a baseline retest, query for the result of that test, and then ignore the result in calculating whether we've regressed. I've opened https://bugs.launchpad.net/britney/+bug/1973864 for this issue. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developer https://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: PGP signature -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: autopkgtest missed regression for kinetic
On Sun, May 15, 2022 at 4:26 PM Steve Langasek wrote: > > The test has never passed; it has had neutral results and failing results, > > but by policy, proposed-migration does not treat neutral->fail as a > > migration-blocking regression, only pass->fail. Neutral test results are > > informational only. > > Having said this, I am just this moment looking at cases where britney is > treating neutral->fail as a regression. I still believe the intended policy > is as I described, but one way or another there appear to be some bugs here > in the implementation ;) devhelp's autopkgtest is marked as "superficial". In Debian, a passing superficial autopkgtest won't speed up migration to Testing. But if the test fails, it indicates that there is a serious bug and migration to Testing is blocked. Please see the description of the intent of "superficial": https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst Thank you, Jeremy Bicha -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: autopkgtest missed regression for kinetic
On Sun, May 15, 2022 at 12:57:34PM -0700, Steve Langasek wrote: > This is because the test failure is not a regression. > * autopkgtest for devhelp/41.2-2: amd64: Not a regression, arm64: Not a > regression, armhf: Test in progress, ppc64el: Not a regression, s390x: Not a > regression > The test has never passed; it has had neutral results and failing results, > but by policy, proposed-migration does not treat neutral->fail as a > migration-blocking regression, only pass->fail. Neutral test results are > informational only. Having said this, I am just this moment looking at cases where britney is treating neutral->fail as a regression. I still believe the intended policy is as I described, but one way or another there appear to be some bugs here in the implementation ;) -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developer https://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: PGP signature -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: autopkgtest missed regression for kinetic
On Sun, May 15, 2022 at 08:06:44AM -0400, Jeremy Bicha wrote: > On Sat, May 14, 2022 at 10:18 AM Brian Murray wrote: > > On Fri, May 13, 2022 at 08:05:20PM +0200, Julian Andres Klode wrote: > > > On Fri, May 13, 2022 at 12:06:46PM -0400, Jeremy Bicha wrote: > > > > I am concerned that the excuses page is ignoring that webkit2gtk > > > > 2.36.1-1 caused an autopkgtest regression for devhelp. > > > > https://autopkgtest.ubuntu.com/packages/d/devhelp/kinetic/amd64 > > > > https://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html > > > > The regression was correctly caught in Debian: > > > > https://tracker.debian.org/pkg/webkit2gtk > > > > I am concerned because I would expect other packages could have > > > > introduced regressions but been allowed in to Kinetic. > > > It seems the state of the 2 autopkgtest-web workers is inconsistent, > > > not all results were copied up on all instances. > > > I noticed that it retried a migration-reference/0 test after the failure > > > which indicated it did not allow the failure, but a later run might have > > > hit the different backend and hence might have a different view. > > > So it's unclear if that's the reason, but bdmurray is cleaning that up > > > right now. > > The cleanup finished overnight and now both instances of the > > autopkgtest-web servers have the same information. > My test case is still broken. devhelp is not blocking webkit2gtk on > the excuses page. This is because the test failure is not a regression. * autopkgtest for devhelp/41.2-2: amd64: Not a regression, arm64: Not a regression, armhf: Test in progress, ppc64el: Not a regression, s390x: Not a regression The test has never passed; it has had neutral results and failing results, but by policy, proposed-migration does not treat neutral->fail as a migration-blocking regression, only pass->fail. Neutral test results are informational only. If you want failures of this package's tests to be treated as regressions, then the devhelp source package should be fixed so that its autopkgtests return the correct error code. This version of webkit2gtk has yet to migrate, so you could do this now, get new devhelp to migrate, and then have webkit2gtk be blocked; or you could just open a block-proposed bug if this is something that should be considered a one-off. In any case, hopefully this clarification about the policy is useful. It is a policy that should be kept on the proposed-migration end because it provides greater flexibility than a strictly pass/fail system, and also we shouldn't deviate from Debian's behavior here and categorically have a stricter gate than Debian which we don't have the capacity to maintain. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developer https://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: PGP signature -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: autopkgtest missed regression for kinetic
On Sat, May 14, 2022 at 10:18 AM Brian Murray wrote: > On Fri, May 13, 2022 at 08:05:20PM +0200, Julian Andres Klode wrote: > > On Fri, May 13, 2022 at 12:06:46PM -0400, Jeremy Bicha wrote: > > > I am concerned that the excuses page is ignoring that webkit2gtk > > > 2.36.1-1 caused an autopkgtest regression for devhelp. > > > > > > https://autopkgtest.ubuntu.com/packages/d/devhelp/kinetic/amd64 > > > https://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html > > > > > > The regression was correctly caught in Debian: > > > https://tracker.debian.org/pkg/webkit2gtk > > > > > > I am concerned because I would expect other packages could have > > > introduced regressions but been allowed in to Kinetic. > > > > It seems the state of the 2 autopkgtest-web workers is inconsistent, > > not all results were copied up on all instances. > > > > I noticed that it retried a migration-reference/0 test after the failure > > which indicated it did not allow the failure, but a later run might have > > hit the different backend and hence might have a different view. > > > > So it's unclear if that's the reason, but bdmurray is cleaning that up > > right now. > > The cleanup finished overnight and now both instances of the > autopkgtest-web servers have the same information. My test case is still broken. devhelp is not blocking webkit2gtk on the excuses page. Thank you, Jeremy Bicha -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: autopkgtest missed regression for kinetic
On Fri, May 13, 2022 at 08:05:20PM +0200, Julian Andres Klode wrote: > On Fri, May 13, 2022 at 12:06:46PM -0400, Jeremy Bicha wrote: > > Hi, > > > > I am concerned that the excuses page is ignoring that webkit2gtk > > 2.36.1-1 caused an autopkgtest regression for devhelp. > > > > https://autopkgtest.ubuntu.com/packages/d/devhelp/kinetic/amd64 > > https://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html > > > > The regression was correctly caught in Debian: > > https://tracker.debian.org/pkg/webkit2gtk > > > > I am concerned because I would expect other packages could have > > introduced regressions but been allowed in to Kinetic. > > It seems the state of the 2 autopkgtest-web workers is inconsistent, > not all results were copied up on all instances. > > I noticed that it retried a migration-reference/0 test after the failure > which indicated it did not allow the failure, but a later run might have > hit the different backend and hence might have a different view. > > So it's unclear if that's the reason, but bdmurray is cleaning that up > right now. The cleanup finished overnight and now both instances of the autopkgtest-web servers have the same information. -- Brian Murray -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: autopkgtest missed regression for kinetic
On Fri, May 13, 2022 at 12:06:46PM -0400, Jeremy Bicha wrote: > Hi, > > I am concerned that the excuses page is ignoring that webkit2gtk > 2.36.1-1 caused an autopkgtest regression for devhelp. > > https://autopkgtest.ubuntu.com/packages/d/devhelp/kinetic/amd64 > https://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html > > The regression was correctly caught in Debian: > https://tracker.debian.org/pkg/webkit2gtk > > I am concerned because I would expect other packages could have > introduced regressions but been allowed in to Kinetic. It seems the state of the 2 autopkgtest-web workers is inconsistent, not all results were copied up on all instances. I noticed that it retried a migration-reference/0 test after the failure which indicated it did not allow the failure, but a later run might have hit the different backend and hence might have a different view. So it's unclear if that's the reason, but bdmurray is cleaning that up right now. -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
autopkgtest missed regression for kinetic
Hi, I am concerned that the excuses page is ignoring that webkit2gtk 2.36.1-1 caused an autopkgtest regression for devhelp. https://autopkgtest.ubuntu.com/packages/d/devhelp/kinetic/amd64 https://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html The regression was correctly caught in Debian: https://tracker.debian.org/pkg/webkit2gtk I am concerned because I would expect other packages could have introduced regressions but been allowed in to Kinetic. Thank you, Jeremy Bicha -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
