On Fri, 2004-09-17 at 01:43, Tom Lane wrote: > PS: this points up once again that the regression tests do not do very > well at testing concurrent behavior. We need to think about some sort > of parallel-test harness that would let us write tests that have better > odds of catching bugs like this.
I agree -- something like this would be very useful. I'm a little stumped about how to actually write such a thing, though... One incremental improvement to the existing RT system would be to make each test self-contained (i.e. as far as possible it would assume nothing about DBMS state before it ran, and not leave the DBMS state modified when it finished running). If writing completely self-contained tests isn't possible, we could specify a dependency graph for the tests. Then we could have the regression test system generate a randomized grouping of tests to be executed. In order to have reproducible results, we could associate each randomized grouping with a numeric key (there should be some way to produce a 1-to-1 mapping). So someone reporting a regression test problem would say: "I see the following regression test failures, the test key is xyz", and a developer could run the regression tests specifying that key and have the RT system run the same combination of tests in the same order that produced the problem the user saw. This would be a complement to the existing method of running the RTs. I still think we need a separate way to test parallel behavior though, in addition to the above idea. Can anyone think of a good way to do this? -Neil ---------------------------(end of broadcast)--------------------------- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html