On Wed, 04 Aug 2010 19:53:55 +0200, =?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?= <mar...@v.loewis.de> wrote: > >>> It happens when running test_smtplib before test_smtpb: > >> > >> Nice! How did you work that out? I'd like to learn how to diagnose > >> this sort of thing, because it seems to come up a lot, and I'm not > >> much use at the moment :-) > > > > I simply tried to run test_smtplib before test_smtpd. > > A more deterministic (and more tedious) way is this: if you > suspect that some failure might be due to the order of test cases, > take a build log from the build bot where it fails, and run the tests > in the exact same order. See whether the problem reproduces. > > If it does, drop tests from the test sequence until you end up with > the minimum number of tests that you need to run in sequence (and yes, > I had interworkings of three test modules at some point). > > Of course, educated guessing can accelerate the process.
Two bits of info to help you implement this process: The buildbots have regrtest print out the random seed used for the test run. The --randseed option of regtest can be used to repeat this run. That's the first step, which proves that the specific order produced the test failure. Then you cut and paste the test list into a file, and use the regrtest -f option to run the tests from that file, and do your binary search by breaking up the list into pieces. Hmm. If we added a 'binsearch' option to regrtest, you could just pass it the random seed and that option and off it would go... -- R. David Murray www.bitdance.com _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
