Re: Dpkg testing framework, take 5 (was: Re: How are things going?)
Esteban Manchado Velázquez writes (Re: Dpkg testing framework, take 5 (was: Re: How are things going?)): On Sun, Oct 08, 2006 at 03:36:16PM +0200, Nicolas François wrote: Is the testsuite intended to be run at build time, or manually? My initial idea was manually, mostly because it fits my testing style better, but I don't know. If the tests do not take an inordinate time to run (as a rule of thumb, if they're not more time-consuming than the build itself) then you should run them automatically as part of the build. Otherwise they should be left to be run manually. I would also like to make a request: please try to make it relatively straightforward to run the tests on the installed version of the package, on the running system, as well as in the obvious `playground' style. This will make whole-system testing much easier. In particular, it will make it easier to grind my automated in-system testing axe (http://wiki.ubuntu.com/AutomatedTesting). Ian.
Re: Dpkg testing framework, take 5 (was: Re: How are things going?)
Hi Nicolas! On Sun, Oct 08, 2006 at 03:36:16PM +0200, Nicolas François wrote: Hi Esteban, I had a look at the testsuite. It basically fulfill my requirements: * I'm able to read the results of the testsuite * I'm able to write tests Great :-) Just a few things (not really related to the test environment, but more on the current tests) [...] OK, thanks for your comments. There's LOTS of things I want to change, that was just an experiment, a proof of concept. I would like to document it, integrate it well with the dpkg codebase, and fix the obvious bits _before_ it getting into some SCM. If nobody has anything against it, I will begin that clean up and documentation work, so I get it in shape to let it enter the repository... Is the testsuite intended to be run at build time, or manually? My initial idea was manually, mostly because it fits my testing style better, but I don't know. Perhaps we could leave that as manually, _but_ show some note when the build is complete, telling the user how to run the test suite :-? I would like to see it in the dpkg repository, I don't think it's necessary to distribute it with the source package. If it does not interfer with the current build (does not change the source binary packages), it would be nice to have it in trunk, otherwise in a separate branch. AFAIK, it should never need any changes to the source, except: 1) Some change to the main Makefile, if you want the testsuite to be executed from the tree root (not the case) 2) Any refactoring you may need to make the code unit-testable (not the case, either). Regards, -- Esteban Manchado Velázquez [EMAIL PROTECTED] EuropeSwPatentFree - http://EuropeSwPatentFree.hispalinux.es Help spread it through the Net in signatures, webpages, whatever! signature.asc Description: Digital signature
Re: Dpkg testing framework, take 5 (was: Re: How are things going?)
Hi Esteban, I had a look at the testsuite. It basically fulfill my requirements: * I'm able to read the results of the testsuite * I'm able to write tests Just a few things (not really related to the test environment, but more on the current tests) * can you mention the dependencies of the testsuite (the check and libtest-unit-perl packages), in a README file * I would like to see the testsuite able to test builds done from a subversion working directory (objects and binaries are compiled in the build-tree directory) * libdpkg.a is not in ../lib * dpkg-scanpackages should be ../script/dpkg-scanpackages.pl (it may require some changes to be runnable from the source directory) Currently, on my system, the dpkg-scanpackages test check the dpkg-multicd implementation. Is the testsuite intended to be run at build time, or manually? I would like to see it in the dpkg repository, I don't think it's necessary to distribute it with the source package. If it does not interfer with the current build (does not change the source binary packages), it would be nice to have it in trunk, otherwise in a separate branch. It could be nice to play with the testsuite during the freeze. Kind Regards, -- Nekral -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Dpkg testing framework, take 5 (was: Re: How are things going?)
Hi, On Thu, Oct 05, 2006 at 01:52:29PM +0200, Raphael Hertzog wrote: [...] I bet for lack of time... all the dpkg developers that I know are reasonable and wouldn't want to throw away the work of someone else if he has taken care to ask for regular review of his work. OK, thanks for your reply. Ian Jackson said he's ready to give advice on dpkg developement in general. Furthermore it did some groundwork on a testing framework for Ubuntu (it was a package testing framework), so I'm sure that he could review your work in that area. OK, I would love to have some comments/criticism from Ian (and anybody else, for that matter). Just go ahead! Right now there are two proposals. If people want to review them and make some comments, but lack the time to review both, _please_ ignore the first proposal (well, even if you have lots of free time: begin with the _second_ one and only review the first if you don't like the second). Just to try to speed things up, I have prepared a kind of executive summary. But before that, PLEASE TAKE INTO ACCOUNT: *testing is way easier than it seems*, it you don't understand terms or want to help but think it's too hard, please don't think so, and ask whatever you want. We could even arrange an IRC meeting to make a demo, explain the framework or the underlying ideas, or just to discuss the design of the whole thing. - 8 - Dpkg Testing Framework Proposals Executive Summary == Both proposals -- The main idea is integrate both unit and functional testing for both C-based and Perl-based programs. If new utilities appear in other languages, of course the tests for them should be integrated in the same system. Right now both proposals are proofs of concept with examples of C unittesting[1] and Perl utilities functional testing [3]. One of the most important features of the system would be _not_ to force the developers to write all the testing code themselves: when possible, micro-formats, simple text files, macros and other tools will be used to ease common cases. Second proposal --- Uses Test::Unit, a xUnit-style module for Perl. The main advantage over the first proposal are: (1) more dpkg hackers know Perl than Tcl, and (2) we already have Perl code in the dpkg codebase (related utilities as dpkg-parsechangelogs), so it's easier to integrate unit tests for those cases. Another advantage is that xUnit is very well thought, and that probably most testing-aware guys know xUnit in _some_ language, so they will be more comfortable with a xUnit-style solution. It's available in a Darcs repo, at http://www.demiurgo.org/darcs/testsuite/ More info in http://lists.debian.org/debian-dpkg/2006/10/msg0.html First proposal -- Uses DejaGNU[4], written in Tcl. I don't like the language, either ;-) I tried with DejaGNU because it was supposed to have the different kinds of tests integration thing already solved, but I'm not sure that's that hard to do ourselves. It's available in an Arch/Bazaar repo, at http://people.debian.org/~zoso/arch/dpkg--test--1.0/ - 8 - HTH, [1] I.e., making calls to specific functions or modules; I'm using the check[2] library for this [2] http://check.sourceforge.net/ [3] http://www.gnu.org/software/dejagnu/ [4] I.e., executing programs with some parameters and see if they behave correctly -- Esteban Manchado Velázquez [EMAIL PROTECTED] EuropeSwPatentFree - http://EuropeSwPatentFree.hispalinux.es Help spread it through the Net in signatures, webpages, whatever! signature.asc Description: Digital signature
Re: Dpkg testing framework, take 5 (was: Re: How are things going?)
On Fri, 29 Sep 2006, Esteban Manchado Velázquez wrote: Hi people! [...] As I was the one who talked to Steve, I think I should say something :-) The thing that worried me the most was the silence after my proposals or questions on the testing framework. I assume that we mostly agree about the _having_ some test suite for the most important utility in Debian, so the problem seems to be either lack of time or agreement. I bet for lack of time... all the dpkg developers that I know are reasonable and wouldn't want to throw away the work of someone else if he has taken care to ask for regular review of his work. Ian Jackson said he's ready to give advice on dpkg developement in general. Furthermore it did some groundwork on a testing framework for Ubuntu (it was a package testing framework), so I'm sure that he could review your work in that area. Just go ahead! Cheers, -- Raphaël Hertzog Premier livre français sur Debian GNU/Linux : http://www.ouaza.com/livre/admin-debian/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
