Sorry to hear that you had a frustrating time with +1 maintenance last week. Large transitions can definitely be frustrating because many times, it's so unclear how to make progress on them.
Some notes: - The request.cgi API endpoint does deduplication of test requests on the backend. So provided you are generating your list of tests with matching logic for triggers and options (--all-proposed vs not), it's safe to blindly rerun retry-autopkgtest-regressions as many times as you need to without worrying about duplicates driving up the build queue times. - Although you didn't feel that you made much progress, it is still very valuable in situations like this to have somebody monitoring the things that have continued to fail and drill into them as necessary. So I would not consider this time wasted! If my suggestion had been more than an IRC driveby on the weekend due to my limited time, I would have mentioned the specific pipeline I was using to retrigger tests and why: $ retry-autopkgtest-regressions --all-proposed | grep '=perl%2F' \ | xargs -rn1 -P10 wget --load-cookies ~/.cache/autopkgtest.cookie -O- First, this skips the normal '--blocks perl' option to retry-autopkgtest-regressions in favor of a postprocessing grep. The reason for this is that using --blocks doesn't just limit the tests reported, it also limits the triggers emitted. So if there is a package, say, libcatmandu-mab2-perl, which is failing its autopkgtests against perl because it's using the release version of libcatmandu-mab2-perl, *and* failing its autopkgtests against itself because it's using the release version of perl, this picks up both triggers; which not only ensures the package gets tested with both of these packages from -proposed, it also credits any test pass against both packages in update_excuses. Second, even though the above should already pick up more packages from -proposed that we need, I've also added --all-proposed, saying to grab all packages from -proposed. Two reasons for this: - As previously discussed, there were select packages in the autopkgtest base images that were newer than the versions in lunar release, due to building against kinetic-security. For a mass-retry, --all-proposed was more likely to succeed than to cause false negatives. - In some cases, a test might require two packages (both perl and libfoo-perl) from -proposed in order to pass, but for whatever reason the test against libfoo-perl had already passed (because of luck on the autopkgtest -proposed fallback handling, or a manual retry done in isolation without considering perl itself, or $other), while the same test against perl had failed. In that case, retry-autopkgtest-regressions would output a line with only perl as the trigger, so without --all-proposed, we would fail to pick up libfoo-perl from -proposed. Using --all-proposed for a mass-retry here again increases the chances of the test being run correctly and getting a passing result. I'm sorry this is all as esoteric as it is. I've braindumped to the release team a bit about improvements that could be made to britney to automate retries and improve our success rate in these situations without requiring quite so much manual intervention. On Mon, Dec 05, 2022 at 05:04:47PM +0000, Robie Basak wrote: > The mechansism used that translates triggers into magic "just these from > proposed" is done via the --apt-pocket=proposed=... autopkgtest > commandline, so the actual apt pinning is done by autopkgtest itself. > The use of --env=ADT_TEST_TRIGGERS= etc was a red herring for me. I > couldn't find anything in autopkgtest itself that uses this variable. I > only saw it being used during test result collection. So what does > ADT_TEST_TRIGGERS= do, anyway? I didn't know the answer to this, so I googled and found https://wiki.debian.org/debci/britneyIntegration telling me that: "This line is the essential bit for recording the test trigger in result.tar so that britney can match a result to a request" And sure enough it appears in the britney source. > However, sometimes you might find --apt-pocket=proposed=... *not* used, > even when triggers appear to exist. I didn't get to the bottom of this. > Maybe it's something to do with magic fallback behaviour I've heard > mentioned (an automatic retry with different conditions?) but couldn't > find anything documented and didn't find any implementation for this. > For example, looking at [1], vorlon's test dated 2022-11-28 19:26:59 UTC > and my subsequent retest dated 2022-11-30 10:20:55 UTC were subsequently > identical, but looking at the logs, his one was called with > --apt-pocket=proposed with no pinning defined, whereas mine was. All I > did was hit the retry button against vorlon's test request and failure, > and mine got run differently! I'd appreciate someone explaining this to > me. This seems to simply be a bug in what autopkgtest.ubuntu.com considers a 'retry', in that it is not propagating the all-proposed=1 option. > In investigating the above, I chose to use lxd since a package > dependency resolution issue should reproduce just fine in a container, > and working with containers is usually much quicker. But when I couldn't > reproduce against ubuntu-daily:lunar, I tried images:ubuntu/lunar/amd64 > for a minimalist base image, but that doesn't exist. I found this > surprising given that images:debian/sid/amd64 does exist. I tried > upgrading up from images:ubuntu/kinetic/amd64, and that worked (and > didn't take long, with it being minimal). It also helped me notice the > libssl3 thing, because openssl happens to be higher in kinetic-updates > at the moment than the lunar release pocket, because it hasn't migrated > from lunar proposed yet. images: is an endpoint maintained by the LXD community and is not the base for the images we use in autopkgtest. That's ubuntu-daily:. Of course, at archive series opening there are bootstrapping questions about providing those images before we've gotten the base packages updated in the devel release pocket that would allow us to generate them. ubuntu-daily:lunar exists now, but for reasons discussed in other threads wrt kinetic-security vs kinetic release pocket used when bootstrapping the autopkgtest images for the new dev series, would not necessarily be a useful base for trying to reproduce some of these more inscrutable autopkgtest failures. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developer https://www.debian.org/ slanga...@ubuntu.com vor...@debian.org
signature.asc
Description: PGP signature
-- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel