Scott Kitterman wrote: > I think the only path to better tested releases is higher quality test > releases. <and later> > Personally, I think it means we need to be doing a much better job of testing > and bug fixing as developers. <...> > The moral of the story is that my project had increased in complexity beyond > what my QA approach could sustain and I had to re-engineer QA, even at the > cost of some features. Once I had a QA approach that matched the complexity > of the project, then release quality went up again. > > I suspect that we may be in a similar position with Ubuntu. We need to > radically rethink testing and how test results get back into fixes. I > believe that Ubuntu has gotten more complex and we need to match our test/fix > methodology to match. I would like to hear ideas on the subject.
There is currently a battery of automated tests run, testing the installability of packages, compilation of packages, policy compliance of packages, file conflicts of packages, dependency maps of packages, etc. As developers we would do well to track these more carefully, and try to ensure that fewer packages appear in any of these lists. Most of these lists are linked from http://qa.ubuntuwire.com. Additionally, there is coordinated testing done of each testing release, with a clear set of test procedures. Adding more test cases (especially flavour-specific test cases), and getting more people to run through these procedures (currently only about 5 or 6 people run most of them) would be a great help towards ensuring that the releases meet quality goals. Further information and image results are available from http://iso.qa.ubuntu.com/qatracker/build/all/all Setting up an automatic install / upgrade / remove / purge tester would be good, perhaps using piuparts or similar infrastructure, although this requires considerable resources in terms of local storage and processing power. There are likely also several other automatable targets in terms of package state or archive consistency that would benefit from additional tools to track them. > I'm not certain, however, that we need more bugs. We need better bugs and we > need more fixes. Suggestions? In recent times there seems to be some disconnect between the presence of bugs and the presence of developers working on these bugs. While we all tend to be busy much of the time, perhaps there are ways that we can improve the view of bugs in need of attention, or otherwise help understand which bugs are likely to be perceived as painful to users at release time. I think the recent decision to align the use of bug importance with the SRU criteria will help us all to better communicate what bugs need to be fixed now, and better understand whether a given release is likely to have issues. Further, I'd like to see more bugs. I hear lots of stories of people who don't want to bother filing a bug, and would rather complain in their blogs, on the forums, in IRC, or even in the mailing lists: helping encourage lots of bug filing helps reduce these cases, and presents a better picture of what is working, and what is broken. While this necessarily means more bugs that are really support requests, or duplicate, or require upstream decisions, or need completely new code, it also means that we can develop metrics to understand what is sought, and try to meet that. We should also be better about chasing bugs that are fixed. There are a number of bugs marked fixed upstream, or fixed in Debian, or with patches. There are plenty of others for which there are good fixes in Fedora, SuSE, Gentoo, etc. At least in those cases where we can track that a given bug exists in Ubuntu and a fix is available, we ought assiduously chase these: the more quickly we can go from a fix being available somewhere to a fix being available in Ubuntu, the more likely we are to be informed of a fix being available somewhere, and the better chance we have of a relatively small number of bugs. -- Emmet HIKORY -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss