I forgot to mention but I'd like to thank Brian Murray for helping me
with the autopkgtest SQLite database and also suggestions for
improvements. And sharing the results for other teams was his idea as
well :)
Em 16/04/2024 18:54, Lucas Kanashiro escreveu:
Hi everyone,
As Noble release is approaching and in the final part of the cycle we
had some big changes in the archive (time_t transition, xz's CVE fix),
the Server team decided to compare the autopkgtest results from before
the end of February, more precisely 2024-02-28 (a guess of a date
before the time_t work started), and now (2024-04-16). This comparison
could show us any potential regression due to big changes in the
archive, and allow us to try to address those issues before the release.
What we did is basically get the latest test result of the packages in
all architectures before the reference date (2024-02-28) and compare
with the latest test run (2024-04-16) of the packages on the same
architecture. We are using the autopkgtest SQLite database available
here [1]. I am calling it "bad news" when the tests of a package in a
given architecture was passing before the reference date and now they
are not.
The script I used to do this is available here [2]. And I used the
mapping of packages and teams [3] to get the list of packages. The
output of the script is a JSON file that looks like the following for
one package:
"adsys": {
"arm64": {
"before": {
"result": "all tests passed",
"test_run_id": "20240227_182345_d0549@",
"triggers": "samba/2:4.19.5+dfsg-1ubuntu1",
"test_log":
"https://autopkgtest.ubuntu.com/results/autopkgtest-noble/noble/arm64/a/adsys/20240227_182345_d0549@/log.gz"
},
"after": {
"result": "at least one test failed",
"test_run_id": "20240416_122755_08ec0@",
"triggers": "sssd/2.9.4-1.1ubuntu6 c-ares/1.27.0-1.0ubuntu1
samba/2:4.19.5+dfsg-4ubuntu9",
"test_log":
"https://autopkgtest.ubuntu.com/results/autopkgtest-noble/noble/arm64/a/adsys/20240416_122755_08ec0@/log.gz"
}
}
},
Attached are the output of the scripts for the following teams:
- foudantions-bugs
- desktop-packages
- kernel-package
- ubuntu-server
The Server team is already going through the list to check if there is
any real regression requiring some work. Keep in mind that not all
packages listed there are necessarily real problems, maybe the test
failed because of a bad trigger, or autopkgtest infra issue, so manual
check is required to make sure this is a real regression.It is also
important to note that the script always gets the latest test result
before the reference date, so the failure being analysed could be a
flaky test for instance, too.
If you have any question or suggestion on this let me know.
I hope that's useful for other teams.
[1] https://autopkgtest.ubuntu.com/static/autopkgtest.db
[2] https://github.com/lucaskanashiro/autopkgtest-diff
[3] http://reqorts.qa.ubuntu.com/reports/m-r-package-team-mapping.json
--
Lucas Kanashiro
--
Lucas Kanashiro
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel