Re: Searching for autopkgtest regressions in Noble
I forgot to mention but I'd like to thank Brian Murray for helping me
with the autopkgtest SQLite database and also suggestions for
improvements. And sharing the results for other teams was his idea as
well :)
Em 16/04/2024 18:54, Lucas Kanashiro escreveu:
Hi everyone,
As Noble release is approaching and in the final part of the cycle we
had some big changes in the archive (time_t transition, xz's CVE fix),
the Server team decided to compare the autopkgtest results from before
the end of February, more precisely 2024-02-28 (a guess of a date
before the time_t work started), and now (2024-04-16). This comparison
could show us any potential regression due to big changes in the
archive, and allow us to try to address those issues before the release.
What we did is basically get the latest test result of the packages in
all architectures before the reference date (2024-02-28) and compare
with the latest test run (2024-04-16) of the packages on the same
architecture. We are using the autopkgtest SQLite database available
here [1]. I am calling it "bad news" when the tests of a package in a
given architecture was passing before the reference date and now they
are not.
The script I used to do this is available here [2]. And I used the
mapping of packages and teams [3] to get the list of packages. The
output of the script is a JSON file that looks like the following for
one package:
"adsys": {
"arm64": {
"before": {
"result": "all tests passed",
"test_run_id": "20240227_182345_d0549@",
"triggers": "samba/2:4.19.5+dfsg-1ubuntu1",
"test_log":
"https://autopkgtest.ubuntu.com/results/autopkgtest-noble/noble/arm64/a/adsys/20240227_182345_d0549@/log.gz";
},
"after": {
"result": "at least one test failed",
"test_run_id": "20240416_122755_08ec0@",
"triggers": "sssd/2.9.4-1.1ubuntu6 c-ares/1.27.0-1.0ubuntu1
samba/2:4.19.5+dfsg-4ubuntu9",
"test_log":
"https://autopkgtest.ubuntu.com/results/autopkgtest-noble/noble/arm64/a/adsys/20240416_122755_08ec0@/log.gz";
}
}
},
Attached are the output of the scripts for the following teams:
- foudantions-bugs
- desktop-packages
- kernel-package
- ubuntu-server
The Server team is already going through the list to check if there is
any real regression requiring some work. Keep in mind that not all
packages listed there are necessarily real problems, maybe the test
failed because of a bad trigger, or autopkgtest infra issue, so manual
check is required to make sure this is a real regression.It is also
important to note that the script always gets the latest test result
before the reference date, so the failure being analysed could be a
flaky test for instance, too.
If you have any question or suggestion on this let me know.
I hope that's useful for other teams.
[1] https://autopkgtest.ubuntu.com/static/autopkgtest.db
[2] https://github.com/lucaskanashiro/autopkgtest-diff
[3] http://reqorts.qa.ubuntu.com/reports/m-r-package-team-mapping.json
--
Lucas Kanashiro
--
Lucas Kanashiro
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Searching for autopkgtest regressions in Noble
Hi everyone,
As Noble release is approaching and in the final part of the cycle we
had some big changes in the archive (time_t transition, xz's CVE fix),
the Server team decided to compare the autopkgtest results from before
the end of February, more precisely 2024-02-28 (a guess of a date before
the time_t work started), and now (2024-04-16). This comparison could
show us any potential regression due to big changes in the archive, and
allow us to try to address those issues before the release.
What we did is basically get the latest test result of the packages in
all architectures before the reference date (2024-02-28) and compare
with the latest test run (2024-04-16) of the packages on the same
architecture. We are using the autopkgtest SQLite database available
here [1]. I am calling it "bad news" when the tests of a package in a
given architecture was passing before the reference date and now they
are not.
The script I used to do this is available here [2]. And I used the
mapping of packages and teams [3] to get the list of packages. The
output of the script is a JSON file that looks like the following for
one package:
"adsys": {
"arm64": {
"before": {
"result": "all tests passed",
"test_run_id": "20240227_182345_d0549@",
"triggers": "samba/2:4.19.5+dfsg-1ubuntu1",
"test_log":
"https://autopkgtest.ubuntu.com/results/autopkgtest-noble/noble/arm64/a/adsys/20240227_182345_d0549@/log.gz";
},
"after": {
"result": "at least one test failed",
"test_run_id": "20240416_122755_08ec0@",
"triggers": "sssd/2.9.4-1.1ubuntu6 c-ares/1.27.0-1.0ubuntu1
samba/2:4.19.5+dfsg-4ubuntu9",
"test_log":
"https://autopkgtest.ubuntu.com/results/autopkgtest-noble/noble/arm64/a/adsys/20240416_122755_08ec0@/log.gz";
}
}
},
Attached are the output of the scripts for the following teams:
- foudantions-bugs
- desktop-packages
- kernel-package
- ubuntu-server
The Server team is already going through the list to check if there is
any real regression requiring some work. Keep in mind that not all
packages listed there are necessarily real problems, maybe the test
failed because of a bad trigger, or autopkgtest infra issue, so manual
check is required to make sure this is a real regression.It is also
important to note that the script always gets the latest test result
before the reference date, so the failure being analysed could be a
flaky test for instance, too.
If you have any question or suggestion on this let me know.
I hope that's useful for other teams.
[1] https://autopkgtest.ubuntu.com/static/autopkgtest.db
[2] https://github.com/lucaskanashiro/autopkgtest-diff
[3] http://reqorts.qa.ubuntu.com/reports/m-r-package-team-mapping.json
--
Lucas Kanashiro
bad_news_2024-02-28_ubuntu-server.json
Description: application/json
bad_news_2024-02-28_desktop-packages.json
Description: application/json
bad_news_2024-02-28_foundations-bugs.json
Description: application/json
bad_news_2024-02-28_kernel-packages.json
Description: application/json
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: pastebinit default target on Ubuntu
I think we should be pointing it back to paste.ubuntu.com, because our existing users expect it will go to a distro owned pastebin, and we should remain consistent. I am also all for keeping user data on IS controlled assets, we don't exactly know who controls dpaste, and if they parse dmesg or ceph logs for call traces etc that might contain juicy data. For the login issue, perhaps we could do a quick entropy test on the uploaded data. Log data is very repetitive, even on long logs, so it will have low entropy. Base64 encoded data will have high entropy. We just reject any high entropy submissions, and remove the login to view requirement. I use pastebinit all the time, and it would be nice to default back to paste.ubuntu.com. Thanks, Matthew On Tue, 16 Apr 2024 at 08:44, Stéphane Graber wrote: > > On Mon, Apr 15, 2024 at 4:14 PM Steve Langasek > wrote: > > > > On Mon, Apr 15, 2024 at 04:48:17PM +0100, Robie Basak wrote: > > > Prior to Noble, the pastebinit command defaulted to paste.ubuntu.com. In > > > Noble, this has changed to dpaste.com due to an upstream change[1]. > > > > > What do Ubuntu developers think the default should be? If it should > > > remain paste.ubuntu.com, we can ask upstream to change it back, or add a > > > delta for now. > > > > > Reason to keep it dpaste.com: > > > > > People have complained that the login requirement makes it unusable for > > > helping Ubuntu users at large who don't necessarily have an Ubuntu SSO > > > account. > > > > > Reason to keep it paste.ubuntu.com: > > > > > I'm not keen on relying on third party services when not necessary, > > > especially ad-supported ones. I have no reason to distrust the current > > > operator, but in general, these things tend to go wrong sooner or later. > > > > > There was more discussion on IRC[2]. > > > > > [1] > > > https://github.com/pastebinit/pastebinit/commit/5c668fb3ed9b4a103eb22b16e603050a539951e0 > > > [2] https://irclogs.ubuntu.com/2024/04/15/%23ubuntu-devel.html#t14:17 > > > > I was not pleased to see the switch to dpaste.com. I've found that it's > > pretty unusable on mobile, and I don't like this pointing to a service we > > don't control. > > > > And if there are issues with the usability of paste.ubuntu.com, uh, we own > > that service? So let's work with our IS team to make it fit for purpose. > > (I don't know why it currently requires a login to *view* paste contents; > > that seems straightforwardly a bug that we should just get sorted.) > > That's because pastebin servers are frequently abused as a way to get > free mass storage. > > It's not very practical to require login to post to a pastebin as the > whole point is for a tool like "pastebinit" to work without needing > user configuration as it's commonly used as a debug tool on cloud > instances and other random servers random than a user's personal > system. > > With that in mind, a bunch of folks noticed that you could abuse a > service like paste.ubuntu.com by pushing large files (base64 encoded > or the like) and then retrieve them with a very trivial amount of html > parsing (if no raw option is offered directly). > > > There are obviously alternatives to this, but they tend to require a > bunch more server side logic, basically trying to find the right set > of restrictions to both poster and reader so that legitimate users can > use the service normally while abusers get sufficiently annoyed to > stay away from it. > > > -- > > Steve Langasek Give me a lever long enough and a Free OS > > Debian Developer to set it on, and I can move the world. > > Ubuntu Developer https://www.debian.org/ > > slanga...@ubuntu.com vor...@debian.org > > -- > > ubuntu-devel mailing list > > ubuntu-devel@lists.ubuntu.com > > Modify settings or unsubscribe at: > > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel > > > > -- > Stéphane > > -- > ubuntu-devel mailing list > ubuntu-devel@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
