On Tue, Sep 28, 2010 at 4:47 AM, Greg Stein <gst...@gmail.com> wrote: > On Mon, Sep 27, 2010 at 13:25, Julian Foad <julian.f...@wandisco.com> wrote: >> On Fri, 2010-09-24, Greg Stein wrote: >>> On Fri, Sep 24, 2010 at 11:16, Julian Foad <julian.f...@wandisco.com> wrote: >>> >... >>> > I think we should produce a test framework that can give us a WC >>> > containing all the different possible WC states. Then we can write >>> > tests against this framework, some tests that test specific state, and >>> > other tests that apply the same operations to all the states and check >>> > that it works in all the states that it should. >>> >>> This requires a manual process of thinking of all states and all >>> permutations. I don't trust it. >> >> This kind of testing is more about checking that the design is >> implemented and that the code paths are exercised ... >> >>> If we could somehow capture working copies from during normal test >>> runs, *then* I think we'd have "everything". We can easily get the >>> terminal state for each test, which is a great first step. It would be >>> great if we could also get the intermediate steps. >> >> ... while this kind is more about checking for regression or unwanted >> changes of behaviour. The two approaches are complementary. > > Fair enough. > > I took the 1.6.x branch and ran the test suite. It produced about 950 > working copies, and the (compressed) tarball sits at about 850k. I'm > thinking about recording 'svn status' for each working copy, and > checking the tarball in as test data. We can then run an 'svn upgrade' > on each working copy, then 'svn status', and verify that the two > status results match. (caveat minor improvements in state tracking and > status reporting) > > But... running upgrade on about 950 working copies and checking their > status isn't cheap. The tarball sizes don't bother me too much... I'm > more concerned about test suite runtime. > > Anybody else? Should this be "normal test run"? Or should we set up an > "extended" set of tests and drop this into that batch?
I've been advocating for a long time the need for an "extended" test suite. (I mean, really, merge_tests.py exercises all of the "basic" functions in so many more ways than basic_tests.py, that the latter is pretty much redundant when doing pre-commit checks.) This might be an opportunity to start that ball rolling, but it's probably also a lot of work. :) -Hyrum