RFC: Deb 2.0 testing process
Hello everyone, The testers are starting to think about how to organized the 2.0 testing effort. One idea that the testers seemed to like is to create a checklist for checking each package. Before we just checked to see if the packages installed and if the entire system seemed to work correctly. This time we plan on being a bit more thurough, but it will require a bit of help from the developers. What I would like to do is keep a list of checklist for all the packages in a format similar to the packages.gz file. I'd like to create the initial checklist from what the maintainers advice. I don't expect it to be complicated, e.g. Package: xbiff (assuming there is a package for this single binary) Checklist: - by default it checks the users mailbox - switches display, geometry, file, update, volume, bg, fg, rv, and shape work This list can be added to by anyone. What I'd like to ask for now is any comments on this. Please do NOT send any checklist at this time. After a few weeks, I'll post a request for the list, and inform everyone of the final decision. Thanks, Brandon - Brandon Mitchell <[EMAIL PROTECTED]> "We all know linux is great... it PGP: finger -l [EMAIL PROTECTED] does infinite loops in 5 seconds" Phone: (757) 221-4847 --Linus Trovalds -- TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word "unsubscribe" to [EMAIL PROTECTED] . Trouble? e-mail to [EMAIL PROTECTED] .
Re: RFC: Deb 2.0 testing process
Brandon Mitchell <[EMAIL PROTECTED]> writes: >The testers are starting to think about how to organized the 2.0 > testing effort. One idea that the testers seemed to like is to create a > checklist for checking each package. [details snipped] Excellent! If this comes off, I think it will probably work very well; certainly, it seems to be the way to go. Apart from anything else, having such a checklist will hopefully encourage the developers to check their packages against it before they release them! OTOH, if you make this too simplistic, then I fear you're going to miss most of the problems: I'm sure the majority of developers do test their packages at least a little bit before releasing them. I certainly do. But one of the things Debian has been bad at in the past is getting a distribution which is consistent within itself, and that is something that these checklists won't address. That little niggle aside -- and I'm sure you will be doing other things than just this -- the testing made a big difference to Bo, and it sounds like it's going to be even better for Hamm. Keep it up! Andy -- Andy Mortimer, [EMAIL PROTECTED] http://www.poboxes.com/andy.mortimer PGP public key available on key servers -- I found myself alone, alone above a raging sea That stole the only girl I loved and drowned her deep inside of me. -- TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word "unsubscribe" to [EMAIL PROTECTED] . Trouble? e-mail to [EMAIL PROTECTED] .
Re: RFC: Deb 2.0 testing process
In article <[EMAIL PROTECTED]> you wrote: : This list can be added to by anyone. What I'd like to ask for now is any : comments on this. A checklist like this is a good idea, particularly if it eventually provides the list of things that initially need to be part of a regression suite for the package. The downer is that if you don't get working on a regression suite for each package, testing a couple thousand packages against non-null checklists seems like it could take a wickedly long time... particularly if, as I suspect, many packages need to get revisioned during the testing interval to fix bugs. Bdale -- TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word "unsubscribe" to [EMAIL PROTECTED] . Trouble? e-mail to [EMAIL PROTECTED] .
Re: RFC: Deb 2.0 testing process
Andy Mortimer <[EMAIL PROTECTED]> wrote: > OTOH, if you make this too simplistic, then I fear you're going to miss most > of the problems: I'm sure the majority of developers do test their packages at > least a little bit before releasing them. I certainly do. But one of the > things Debian has been bad at in the past is getting a distribution which is > consistent within itself, and that is something that these checklists won't > address. This can be dealt with by creating a check list based on the policy manual (the first think to check in *any* package should be its compliance with the current policy). Such a check list would be very useful even for developers, since people like me, that are not completely aware of the last policy discussions may very well miss some policy aspects when releasing a package. Regards, M. S. Martin A. Soto J. Profesor Departamento de Ingenieria de Sistemas y Computacion Universidad de los Andes [EMAIL PROTECTED] -- TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word "unsubscribe" to [EMAIL PROTECTED] . Trouble? e-mail to [EMAIL PROTECTED] .
Re: RFC: Deb 2.0 testing process
> "Brandon" == Brandon Mitchell <[EMAIL PROTECTED]> writes: Brandon> One idea that the testers seemed to like is to create a Brandon> checklist for checking each package. It seems to me that a fairly comprehensive set of general guidelines would be more useful. The checklist format might work well. Perhaps it should become part of the developer and packager's guide(s) or the policy manual. -- mailto:[EMAIL PROTECTED] (Karl M. Hegbloom) http://www.inetarena.com/~karlheg Portland, OR USA Debian GNU 1.3.1+hamm Linux 2.0.32 AMD K5 PR-133 -- TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word "unsubscribe" to [EMAIL PROTECTED] . Trouble? e-mail to [EMAIL PROTECTED] .
Re: RFC: Deb 2.0 testing process
On 5 Dec 1997, Karl M. Hegbloom wrote: > Well, I guess just a checklist outlining things from the policy guide > is what I had in mind. Like whether it follows the FSSTND or FHS, > there's a copyright statement, compressed man page, menu file with > the dwww link and a menu item for X (with semi mandatory icon) if > appropriate. Ok, I see what you mean now. I was thinking of putting things like this into a generic checklist for all packages. > Hmm. You seem to be suggesting that the package maintianer submit a > checklist for other testers to use? To ensure that it works > elswhere, not just on the build machine. For that, the standard > checklist, then package specific ones I suppose. Yes, the package maintainer submits the initial checklist since they should know the most about each package. Then the maintainer, testers, or anyone for that matter, has the option of adding to the checklist. The reasoning behind this is to make sure the package works on fresh installs, upgraded installs, and doesn't have any problems when other packages are installed. A few examples: - nedit maintainer mentions nc but the tester gets netcat. - config on a fresh install is different that an upgrade, like the X11 library problem we had a while back. - rplayer may work fine on maintainers system if they have the latest bash, but the tester may have an older version and have problems with netscape. These are all old problems that theoretically could be caught with testing. The idea being, what works on a maintainers system may not work everywhere. For the initial list, I'll probably have them all sent to me. But once we get into the swing of things, I'll ask that they submit them to the testing group so if a tester has an addition to the list, it's easy to start a discussion. Also I may not maintain this forever, and I don't feel like asking for an account on master for the checklist maintainer. > I'll watch and see what some more experienced people have to say. I'm hoping to get more opinions before I start doing this. Fixing any problems now is much easier than if we wait until half of the maintainers have already submitted list. Thanks for your comments, Brandon - Brandon Mitchell <[EMAIL PROTECTED]> "We all know linux is great... it PGP: finger -l [EMAIL PROTECTED] does infinite loops in 5 seconds" Phone: (757) 221-4847 --Linus Torvalds -- TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word "unsubscribe" to [EMAIL PROTECTED] . Trouble? e-mail to [EMAIL PROTECTED] .
Checklist request (was: RFC: Deb 2.0 testing process)
[ the orig. message is avail at: http://www.debian.org/Lists-Archives/debian-devel-9711/msg01597.html ] Hi everyone, I haven't heard many responses about the checklist, so I think it's time to get started. I'd like to do this in several phases: I. Get a few required packages for comments so everyone knows what's going on. II. Get all required packages. III. Get all important packages. IV. Get all standard packages. V. Get all optional and extra packages. Now, to start with I., I'd like to pick a few packages and ask the maintainers to post what they think would be a good checklist for their package. There are no wrong answers, just what you think you and a tester should do to verify your package is working correctly. The idea being "what works for a maintainer may not work on a testers system for some reason or another". What I'd like to try to get is a good balance between simplicity and thoroughness. For example, with the diff package: Package: diff - cmp works on identical and different binary or text files - diff works on files, directories, normal or 2 column - sdiff correctly merges two files - diff3 correctly compares 3 files Diff is probably one of the easier packages. There weren't many programs, and there isn't too much to test for each program (although I'd be interested to see what changes the maintainer of this package will make). At this time, I'd like to ask for the maintainers of the following packages to post a list (I'm just trying to get several important packages here): adduser, diff, dpkg, grep, gzip, hostname, login, mount, passwd, tar. Also, if anyone wants to go through the packaging manual, I'd like to create some checklist for groups of packages. E.g. a web server checklist. Also a checklist that applies to all packages should be made (e.g. almost all files should be readable by everyone and in locations required by the standard). The first entry of a web server will then say it should pass the general web server checklist. Thanks, Brandon - Brandon Mitchell <[EMAIL PROTECTED]> "We all know linux is great... it PGP: finger -l [EMAIL PROTECTED] does infinite loops in 5 seconds" Phone: (757) 221-4847 --Linus Torvalds -- TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word "unsubscribe" to [EMAIL PROTECTED] . Trouble? e-mail to [EMAIL PROTECTED] .
Checklist request (was: RFC: Deb 2.0 testing process)
Philip Hands: > It seems a shame to have to ask people to do this sort of thing. > > It strikes me that one should be able to come up with a script that > does a test of this sort in not much more that the time required to > write the list (in this simple case at least ;-) > > I really think we should encourage people to do this where possible. I agree. These tests should be shipped with the package source (not in the .deb file, since most users won't want them). Here's a draft scheme of things: Each source package has an executable file debian/tests/test_ for each binary package. This program, when run with debian/tests as its current directory (which may contain other files used for the tests), either: produces an exit status of 0 and some possibly-irrelevant data on stderr which can be ignored by the caller produces an exit status !=0 some stderr output which tells humans what went wrong It may also print to stdout lines of the form desire DEBIAN_TEST_* require DEBIAN_TEST_* indicating that a more meaningful test can be done if the relevant variable(s) below are set, or that no meaningful test can be done if they are not. The test script must be invoked with the Source-Depends of the source package satisfied. In all cases the test script may create files starting with `tmp_', which must be cleaned up by its caller. The test script may not create other files. It may never modify the source package of which it is a part, or other associated files and directories, even if DEBIAN_TEST_SCRATCH is set. Its behaviour is modified by at least these environment variables: DEBIAN_TEST_SCRATCH if set to a nonempty string then the test is allowed to randomly mess with the system, including installing or removing packages, reconfiguring things randomly, eating mail belonging to the user that invokes it, etc. Otherwise it is not. Test scripts should try to avoid the necessity for this. DEBIAN_TEST_GAINROOT if set to a nonempty string then the test is allowed to become root by invoking DEBIAN_TEST_GAINROOT in front of its arguments. DEBIAN_TEST_GAINROOT may not contain spaces or other shell metacharacters. (A la dpkg-buildpackage.) DEBIAN_TEST_RECOMMENDED indicates that the packages which are Recommended by the binary package in question are installed DEBIAN_TEST_SUGGESTED indicates that the packages which are Suggested by the binary package in question are installed. It is not permitted for a package to require this option for meaningful testing using require DEBIAN_TEST_SUGGESTED because Suggested packages may not be in main. These scripts could be written by package maintainers, or by the testing group and submitted as bugs. Ian. -- TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word "unsubscribe" to [EMAIL PROTECTED] . Trouble? e-mail to [EMAIL PROTECTED] .
Re: Checklist request (was: RFC: Deb 2.0 testing process)
> For example, with the diff package: > > Package: diff > - cmp works on identical and different binary or text files > - diff works on files, directories, normal or 2 column > - sdiff correctly merges two files > - diff3 correctly compares 3 files It seems a shame to have to ask people to do this sort of thing. It strikes me that one should be able to come up with a script that does a test of this sort in not much more that the time required to write the list (in this simple case at least ;-) I really think we should encourage people to do this where possible. I also think that a reasonable way to proceed in the cases where automated testing is not possible, would be to write scripts that ask say: Do this test. Did it work [y/N] Another thing is that the tests or checklists that are written, should be testing for problems that have actually occured in the past. For example, if the diff package has never failed to provide a cmp program that works as expected, there is little point spending time writing a checklist or script to test for this event, and then wasting valuable testers time running those tests. We should look at this as a hunt for bugs that have occurred before, rather than an attempt to prove that everything is working. Otherwise, it is easy to fall into the trap of writing test that you know will work, but don't actually prove very much. To take the diff example again, lets say that a bug that was resolved recently involved diff not noticing the difference between files that end with a linefeed and the same file missing the linefeed. Since this is a bug that would have actually occurred, it is worth testing for, so we create: /usr/doc/diff/TESTS: # diff test 1 echo -n "Test File" > /tmp/difftest1 echo "Test File" > /tmp/difftest2 diff /tmp/difftest1 /tmp/difftest2 > /dev/null 2>&1 && echo "diff: test 1 failed" rm /tmp/difftest? and so on, for each of the things that the maintainer knows to have gone wrong at some time in the past. Not only does this test for the bug, but it also gives diff a workout that is likely to spot other bugs. When new bugs are reported and fixed, the maintainer should be encouraged to add a test that fails on the pre-fix version and succeeds on the new one. Please don't interpret this as criticism of the checklists idea, I'm just trying to make sure that effort expended in this direction is used as effectively as possible. Also, I realise that we can publish checklists on the web much more quickly than we can persuade each maintainer to incorporate test scripts into their packages, so we should definitely have the checklists. I'd just like it to be an interim measure, until the test scripts become a reality. Does this make any sense to anyone else ? If so we should probably start an effort in parallel to the checklists effort, to define a few standards for where to put test scripts, what to call them etc. Some support programs for running the tests and submitting test results would be good too. Cheers, Phil. -- TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word "unsubscribe" to [EMAIL PROTECTED] . Trouble? e-mail to [EMAIL PROTECTED] .
Re: Checklist request (was: RFC: Deb 2.0 testing process)
On Wed, 10 Dec 1997, Philip Hands wrote: > > For example, with the diff package: > > > > Package: diff > > - cmp works on identical and different binary or text files > > - diff works on files, directories, normal or 2 column > > - sdiff correctly merges two files > > - diff3 correctly compares 3 files > > It seems a shame to have to ask people to do this sort of thing. > > It strikes me that one should be able to come up with a script that does a > test of this sort in not much more that the time required to write the list > (in > this simple case at least ;-) This is the second time I've heard this, and it is a valid point. The reason I don't fully back it is a tester using their own test may catch some case the package designer never thought of. How about this, a maintainer can make a script, called /usr/doc//testme.sh. It can run any test the maintainer wants to do. In the checklist, the maintainer writes that the script is available, and test for the below things... This way, the testers can add their own things to the checklist even if the maintainer disagrees. This aproach favors more testing than less, since if a maintainer disagrees with a test, it's still in the checklist, and if they want a test not in the list (should be rare if ever), they can put it in their script. Testers will be encouraged to make any modifications to the given script (but not required), and then use their own script. > Another thing is that the tests or checklists that are written, should be > testing for problems that have actually occured in the past. Just because something works in the past doesn't mean it won't fail in the future. It would be nice if we can catch some bugs that haven't happened yet. The bash bugs come to mind (with netscape not running any helper apps). Comments? Brandon -- TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word "unsubscribe" to [EMAIL PROTECTED] . Trouble? e-mail to [EMAIL PROTECTED] .
Re: Checklist request (was: RFC: Deb 2.0 testing process)
> On Wed, 10 Dec 1997, Philip Hands wrote: > > > > For example, with the diff package: > > > > > > Package: diff > > > - cmp works on identical and different binary or text files > > > - diff works on files, directories, normal or 2 column > > > - sdiff correctly merges two files > > > - diff3 correctly compares 3 files > > > > It seems a shame to have to ask people to do this sort of thing. > > > > It strikes me that one should be able to come up with a script that does a > > test of this sort in not much more that the time required to write the > > list (in this simple case at least ;-) > > This is the second time I've heard this, and it is a valid point. The > reason I don't fully back it is a tester using their own test may catch > some case the package designer never thought of. True. People should certainly not be discouraged from free-form testing. This could also be a criticism of checklists to some extent, since they will have a tendency to guide people into thinking about testing things in a common way. > How about this, a > maintainer can make a script, called /usr/doc//testme.sh. It can run > any test the maintainer wants to do. In the checklist, the maintainer > writes that the script is available, and test for the below things... > This way, the testers can add their own things to the checklist even if > the maintainer disagrees. Sounds good. > Just because something works in the past doesn't mean it won't fail in the > future. It would be nice if we can catch some bugs that haven't happened > yet. True, but impossible of course ;-) How can anyone be expected to come up with tests for bugs that have not happened ? IMHO saying things like ``gut feeling'' or ``experience'' is just another way of saying that it's happened before, maybe not to Debian, or maybe not as an official bug, but it will have happened before (probably rather painfully to the person with the experience ;-) I just wanted to make sure that the checklists did not end up having items on them that have never been known to fail, because testing such things is a waste of testers time. The testers would soon get the idea that those tests are pointless. Admittedly, I cannot think of anything that has never gone wrong, off hand ;-) but this approach struck me as a fairly easy basis to generate checklists. Is there somewhere that one can see closed bugs that are older than 28 days ? > The bash bugs come to mind (with netscape not running any helper apps). But a test of some other aspect of Netscape that _has_ failed in the past would most likely have spotted that, as a side effect. I seem to remember netscape being totally unusable on my system, so it wouldn't have needed to be an exhaustive test ;-) Cheers, Phil. -- TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word "unsubscribe" to [EMAIL PROTECTED] . Trouble? e-mail to [EMAIL PROTECTED] .
Re: Checklist request (was: RFC: Deb 2.0 testing process)
>> For example, with the diff package: >> >> Package: diff - cmp works on identical and different binary or text >> files - diff works on files, directories, normal or 2 column - >> sdiff correctly merges two files - diff3 correctly compares 3 files "Philip" == Philip Hands <[EMAIL PROTECTED]> writes: > It seems a shame to have to ask people to do this sort of thing. Yes! Maybe even against policy? [Followups on this to debian-policy, please.] I applaud the ambitiousness of making test suites for debian core packages, but I wonder whether Debian developers should focus the packaging and installation system rather than trying to fix all the bugs in GNU, etc. In other words, I think the test suite should focus, at least at the outset, on implementing the policy and making sure that installation and upgrades go smoothly. Here's a draft of a checklist geared to that: * init scripts, if any, comply with debian policy (probably only 20% do now;) * package does not modify any files from other packages * any installation shell scripts work with /bin/sh -> /bin/ash, or they specifically have #!/bin/bash * any installation perl scripts work w/ perl 5.003 (?) * [de]installation script output complies with policy Another big thing is that the transition from 1.3 to 2.0 is _very_ smooth, which is not the case now. Have we defined the supported upgrade paths? I know this is all a moving target w/ pkg ordering stuff apparently coming in and (?) dselect being dropped as the default installation mechanism. [BTW, I'm not trying to criticize the current state of hamm, I know the freeze is a ways off and there's a lot of instability going on.] .A. P. [EMAIL PROTECTED]http://www.onShore.com/> -- TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word "unsubscribe" to [EMAIL PROTECTED] . Trouble? e-mail to [EMAIL PROTECTED] .
Re: Checklist request (was: RFC: Deb 2.0 testing process)
On Thu, 11 Dec 1997, Adam P. Harris wrote: > "Philip" == Philip Hands <[EMAIL PROTECTED]> writes: > > It seems a shame to have to ask people to do this sort of thing. > > Yes! Maybe even against policy? [Followups on this to debian-policy, > please.] We are asking, not requiring. If you don't want to make a checklist, the testers _will_, but understand we don't know how half of these packages work, and this also means less time for testing. Why are you against testing packages? If we don't catch it now, a poor user down the road will. They claim Debian is a piece of junk, and move on to a better distribution like Red Hat. > I applaud the ambitiousness of making test suites for debian core > packages, but I wonder whether Debian developers should focus the > packaging and installation system rather than trying to fix all the > bugs in GNU, etc. In other words, I think the test suite should > focus, at least at the outset, on implementing the policy and making > sure that installation and upgrades go smoothly. We already test installs and upgrades. If you'd like to know what we do in more detail, read an earlier post "Debian needs guinea pigs" that I sent here and to deb-user. It is focused on what the user sees with a fairly default setup, so if we have time, I'll suggest some /bin/sh -> /bin/ash testing. > [BTW, I'm not trying to criticize the current state of hamm, I know > the freeze is a ways off and there's a lot of instability going on.] But given my schedule over the next month, any freeze will be too soon. Brandon - Brandon Mitchell <[EMAIL PROTECTED]> "We all know linux is great... it PGP: finger -l [EMAIL PROTECTED] does infinite loops in 5 seconds" Phone: (757) 221-4847 --Linus Torvalds -- TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word "unsubscribe" to [EMAIL PROTECTED] . Trouble? e-mail to [EMAIL PROTECTED] .
Re: Checklist request (was: RFC: Deb 2.0 testing process)
On Thu, 11 Dec 1997, Ian Jackson wrote: > Philip Hands: > > It seems a shame to have to ask people to do this sort of thing. > > > > It strikes me that one should be able to come up with a script that > > does a test of this sort in not much more that the time required to > > write the list (in this simple case at least ;-) > > > > I really think we should encourage people to do this where possible. > > I agree. These tests should be shipped with the package source (not > in the .deb file, since most users won't want them). Requiring the testers to download the package and the source is a bad things IMO. Also, when I'm trying a new package, I think it might be nice to see what it can do by trying the test script. These scripts should be small, using files already on the system if possible (or from /usr/doc//examples). Finally, the maintainer doesn't have to include the script, it's just something extra they want to do. > These scripts could be written by package maintainers, or by the > testing group and submitted as bugs. If it gets to the point that the testers are making testing scripts, we'll probably send it to the maintainer if they are interested, but keep it on a web site for faster distribution among ourselves. Waiting for a package to be uploaded for a test script during a frozen period may make this process painfully long. Brandon - Brandon Mitchell <[EMAIL PROTECTED]> "We all know linux is great... it PGP: finger -l [EMAIL PROTECTED] does infinite loops in 5 seconds" Phone: (757) 221-4847 --Linus Torvalds -- TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word "unsubscribe" to [EMAIL PROTECTED] . Trouble? e-mail to [EMAIL PROTECTED] .
Re: Checklist request (was: RFC: Deb 2.0 testing process)
Ian Jackson writes: > I agree. These tests should be shipped with the package source (not > in the .deb file, since most users won't want them). I am an active tester. Speaking from this viewpoint, your proposal is excellent, except for the location of the tests. From my viewpoint it would be far better if they were included with the .debs, or if they available for downloading apart from the sources. During the bo round of testing, I found that downloading packages was a significant part of the time spent. This proposal would make the tester unload sources as well as the .debs for all packages being tested. This would be an additional demand on the tester's time and on his ppp resources. Bob -- TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word "unsubscribe" to [EMAIL PROTECTED] . Trouble? e-mail to [EMAIL PROTECTED] .
