In article <mailman.4488.1325870461.27778.python-l...@python.org>, Ian Kelly <ian.g.ke...@gmail.com> wrote:
> Randomizing the order is not a bad idea, but you also need to be able > to run the tests in a consistent order, from a specific random seed. > In the real world, test conflicts and dependencies do happen, and if > we observe a failure, make a change, rerun the tests and observe > success, we need to be able to be sure that we actually fixed the bug, > and that it didn't pass only because it was run in a different order. I've seen this argument play out multiple times on this group. Executive summary: OP: "I want to do X" Peanut Gallery: "You're not supposed to do that" Here's my commentary on that. The classic unittest philosophy says that tests should be independent of each other, which means they should be able to be run in arbitrary order. Some people advocate that the test framework should intentionally randomize the order, to flush out inter-test dependencies that the author didn't realize existed (or intend). Test independence is a good thing. Many people don't understand this when writing tests, and inadvertently write tests that depend on each other. I've worked with those systems. They're a bear to debug. You've got some test suite that runs for 15 minutes and dies at case 37 of 43. If you try to run case 37 by itself, it won't run, and you can't figure out what state cases 1-36 were supposed to leave the system in to make 37 work. You could sink days or weeks into debugging this kind of crap. BTDT. That being said, the unittest module, while designed to support the "all tests must be independent" philosophy, is a useful piece of software for lots of things. It provides an easy to use framework for writing tests, lots of convenient assertions, reporting, test discovery, etc, etc. If somebody (i.e. the classic "consenting adult" of the Python world) wants to take advantage of that to write a test suite where the tests *do* depend on each other, and have to be run in a certain order, there's nothing wrong with that. As long as they understand the consequences of their actions, don't try to preach unittest religion to them. They're in the best position to know if what they're trying to do is the best thing for their particular situation. -- http://mail.python.org/mailman/listinfo/python-list
