On Fri, Mar 20, 2015 at 9:42 AM, Johan Corveleyn <jcor...@gmail.com> wrote: ... > Unfortunately, I can't verify the rev > file, since I don't have it anymore, it has been overwritten by trying > to reproduce it (grrrr, should remember to backup leftover > repositories and working copies after a failed test run, before trying > to reproduce it). Whatever I try now, I can't reproduce it anymore > :-(.
I'm wondering if something can be improved in our test suite to help diagnosis of hard-to-reproduce test failures. When this happens, you typically wish you could analyse as much data as possible (i.e. the potentially corrupt repository, working copy, dump file, ... that was used in the test). Currently, I can think of three causes for losing this information: 1) You run a series of test runs in sequence from a script (ra_local, ra_svn, ra_serf), all using the same target directory for running the tests (R:\test in my case, where R: is a ram drive). If something fails in ra_svn, but succeeds in ra_serf, your broken test data is overwritten. 2) You don't know in advance that the failure will turn out to be non-reproducible. You can't believe your eyes, try to run it again to be sure, and lo and behold, the test succeeds (and the broken test data is overwritten), and succeeds ever since. 3) Your test data is on a RAM drive, and you reboot or something. Or you copy the data to a fixed disk afterwards, but lose a bit of information because last-modified timestamps of the copied files are reset by copying them between disks. For 1, maybe the outer script could detect that ra_svn had a failure, and stop there (does win-tests.py emit an exit code != 0 if there is a test failure? That would make it easy. Otherwise the outer script would have to parse the test summary output)? Another option is to let every separate test run (ra_local, ra_svn, ra_serf) use a distinct target test directory. But if you're running them on a RAM disk, theoretically you might need three times the storage (hm, maybe not, because --cleanup ensures that successful test data is cleaned up, so as long as you don't run the three ways in parallel, it should be fine). I guess I will do that already, and adjust my script accordingly. Addressing 2 seems harder. Can the second test execution, on encountering stale test data, put that data aside instead of overwriting it? Or maybe every test execution can use a unique naming pattern (with a timestamp or a pid) so it doesn't overwrite previous data? Both approaches would leak data from failed test runs of course, but that's more or less the point. OTOH, you don't know that stale test data is from a previous failed run, or from a successful run that did not use --cleanup. And 3, well, I already reboot as little as possible, so this is more just a point of attention. Maybe addressing all three at the same time could also be: after a failed test run, automatically zip the test data and copy it to a safe location (e.g. the user's home dir). Thoughts? -- Johan