Dear QA Team In Debian, before the release of an ISO image, we give it one final round of smoke testing. This includes a bunch of tests and spot checks to ensure that the releases are usable as intended.
Unfortunately, our platform for managing this is rather tedious, and may discourage many people who might otherwise be involved in testing from contributing (and lack of testing has caused big trouble at least for the live images, see: https://lists.debian.org/debian-live/2017/06/msg00064.html). Currently, it's tracked in a wiki page: https://wiki.debian.org/Teams/DebianCD/ReleaseTesting/Stretch While it does work, working with large tables in wikis isn't everyone's idea of fun, and it takes some extra co-ordination to deal with wiki locks and edit conflicts. On top of that, defining additional tests, linking them to images, and tracking all the information that's submitted better will just make that wiki page even more monstrous. When I used to do testing on Ubuntu ISO images, we used the Ubuntu ISO QA tracker (http://iso.qa.ubuntu.com). It worked quite well, you can define milestones (like releases, alphas and daily builds), products (ISOs, VMs, etc) and various test definitions that you could link to to those products. Multiple people can test an iso and easily report the results, and once the person responsible for signing off an ISO is satisfied with it, they can mark it as ready. The code for the Ubuntu ISO QA tracker is free software, and I briefly considered suggesting implementing it for Debian, but it's not that actively maintained, and I don't want to be responsible for deploying more PHP code and wouldn't really want to maintain it either. Recently I've been toying with the idea of re-writing a minimal viable product version of the QA tracker, that at least matches our functionality of using the wiki and solving its biggest issues, although, I'm allergic to committing to write anything because I know I have a tendency to lose focus and move on to the next exciting thing. Today I joined a commitment on twitter to code for an hour a day (https://twitter.com/highvoltage/status/965100119505997825), and looked over my wishlist for something that would fit nicely into ~100 hours and this looks like a good fit. I intend to call it Deqalo (it comes from Debian QA Logger, but it doesn't have to be specific to Debian or just for Debian), and the plan is to write it as a generic python module that can be used in anything, with a web front-end written in Flask. I also intend to add an API so that users who test can log their results from a local command line without having to visit the web page (might also be useful for bots that do testing). I'm also planning to use the SB-Admin theme for the web front-end: https://blackrockdigital.github.io/startbootstrap-sb-admin/ At this stage it's mostly vapourware, today I mostly looked at the best way to package SB-Admin (and check what we can do to update our version of Bootstrap that's a bit outdated in the archives), so any feedback/changes are welcome. If it's straight-out a bad idea or if there's already something else we should be using, I'd rather hear about it now than invest too much time into it. So, any thoughts or suggestions are welcome. -Jonathan