Hi Ian, First a brief question: > The source package provides a test metadata file > debian/tests/control. This is a file containing zero or more > RFC822-style stanzas, along these lines: Do you still have somewhere that awk package demo package which had debian/tests/ ? Currently our archive does not carry any package having debian/tests/ (unfortunately).
Just a brief summary for such a ignorant DDs as myself who did not know about those additional projects in Ubuntuland before they got mentioned in this thread: Ubuntu "Testing Automation" project has been going on for a while: https://wiki.ubuntu.com/Testing/Automation , with active (i.e. actually being developed/used) components such as - checkbox (package in Ubuntu): user-land tool (packaged and available in Ubuntu) to provide primarily hardware testing, with some basic software tests available; reporting back to launchpad.net with gathered information on hardware. - mago (package in Ubuntu): built on top of LDTP (package in Debian), provides testing of GNOME GUI. - qa-regression-testing: collection of unit and regression tests for various components (from kernel to xine and apache) of the systems. Is not designed for distribution to the users (yet?) All of the above approaches seems to separate testing "materials" from the actual packages. Both mago and checkbox come with user interfaces, thus enabling users to test/validate their own systems without requiring setting up any virtual environment. And they ship their tests along (there seems to be some discovery mechanisms but I haven't checked in detail yet). On the other hand, Ian's autopkgtest aims to provide a unified testing framework built around packages, so that it becomes possible for us, maintainers, to equip packages with necessary tests batteries; which could be reused by others (e.g. QA teams). So it seems to go closer toward the goals of qa-regression-testing project above, but tries to scale via making tests materials provided by the corresponding maintainers in the source packages. (As is now at least) it requires relatively complex setup of the system to start crunching the tests batteries, thus not user-oriented. For our purposes of testing scientific applications, as it was previously mentioned and exposed in the "case studies" on http://neuro.debian.net/proj_debtest.html page, we want the cocktail of all the above solutions ;-) with less accent on GUI testing but with convenient user-interface not requiring root access (besides possibly installation of the tests batteries). And it seems that the autopkgtest basis, maybe with some functionality from checkbox and mago (e.g. for collecting relevant software/hardware statistics and running basic system tests) could unroll into the ultimate solution for our goals. Unfortunately the core aspect of the current autopkgtest - relying on tests in source packages -- imho to be not the ideal solution to target both sides of the userbase (i.e. maintainers/QA vs mortals). https://wiki.ubuntu.com/AutomatedTesting associated with autopkgtest addresses the FAQ on why source packages: ,--- | Q. Why put the tests in the source package ? | | A. Other possibilities include a special .deb generated by the source | (which is a bit strange and what happens to this .deb and will make it even | harder to reuse upstream test suites), or putting them in the .deb to be tested | (definitely wrong - most people won't want the tests and they might be very | large) or having them floating about separately somewhere (which prevents us | from sharing and exchanging tests with other parts of the Free Software | community). The source package is always available when development is taking | place. `--- Ian -- could you please unroll your arguments a bit? I still do not see why source packages are beneficial besides build-time testing (which we often do already without any additional framework) In our view ideal solution from the user's (or maintainer/QA) point of view should involve exactly that -- interaction with binary packages since that is what everyone knows how to deal with, e.g.: sudo apt-get install apache2-tests adt-run apache2 or even, having installed X tests batteries adt-run --all # to run all "installed" tests batteries We could also complement it with sudo adt-install --all-relevant which would install test batteries complementing installed software packages, thus providing tailored coverage for a particular user needs. To accomplish above with tests only in source packages, would require apt-src like infrastructure (and do version/dependencies tracking on them), and moreover why should I care to keep sources on my (user's) system at all! What is relevant for testing seems to be: testing materials and already installed applications. And that is what other (checkbox/mago/qa-regression-testing) frameworks rely upon. And sure thing, for maintainers/QA it could get more evolved indeed requiring adt-run --all-available --- virt-server [virt-server-arg...] so that testing could be performed on a distant or virtualized environment. So, where do we start/continue sharing the thoughts on a tentative DEP? ;) -- =------------------------------------------------------------------= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110201231745.gx8...@onerussian.com