On Wed, Aug 26, 2009 at 7:39 PM, Bruce Momjian<br...@momjian.us> wrote: > Peter Eisentraut wrote: >> Much of the delay and uncertainty during beta in my mind comes from the >> situation that we wait for negative results and don't trust the release >> until we have seen and fixed enough of them. Instead of waiting for >> concrete, positive results and producing the release with confidence. > > Yep, that is our dilemma.
To get positive results in which you can have confidence, you have to know that the testing which was done actually did a reasonably good job exercising the code in a way that would have flushed out bugs, had any been present. That sounds a lot like the definition of a regression test suite. Of course, we have that already, but it's nowhere near comprehensive. Maybe we should be looking at an expanded test suite that runs on a time scale of hours rather than seconds. Actually, didn't Peter talk about something like this at PGCon? I don't think that any test suite is going to eliminate the need for beta-testing. But if we could say that we had a regression test suite which covered X% of our code, and it passed on all Y platforms tested, that would certainly be a confidence booster, especially for large values of X. What we're basically doing now is hoping that beta-testers come up with some novel tests, and that's a bit hit-or-miss. Part of the question, of course, is how to build up such a regression test suite. One way to do it is try to add a test every time you fix a bug, such that the new test fails on the unpatched code and passes with the fix... or we could have a expanded-regression-test only commitfest during beta, or something. I think people would be willing to put in some work on this. I certainly would. I run the regression tests frequently during development but unless I'm making system catalog changes they rarely catch anything. I'm not prepared to write the whole thing myself, but I would definitely be willing to develop and contribute some tests. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers