Here's an example of why *running* tests in order can make sense.
You could have a bunch of tests of increasing complexity. The first bunch
of tests all run in a few seconds and test some basic functionality. From
experience, you also know that these are the tests that are most likely to
fail as you port to a new environment.
There's also some tests which take a long time to run. If the basic stuff
that's being tested by the earlier tests doesn't work, there's no way these
tests could pass, but they still take a long time to fail.
It's really handy to have the simple tests RUN first. If you see they
fail, you can cancel the rest of the test run and get on with fixing your
code faster.
I don't see this as something that can be solved by ordering tests -
*especially* not on a per-method-level as the OP suggested, because I
tend to have test suites that span several files.
Instead when I've been in the situation that you describe before, I
resorted to an approach where I annotated tests as long or short-running
(or any other criteria), and then ran the tests that were appropriate.
For example, post-commit-tests needed to be short, as otherwise the
feedback came to slow.
Annotation could be done explicit, or implicit by grouping tests
together that had the desired property. However, as I prefer tests that
share e.g. the same module to test the same functionality, I rather have
an annotation mechanism.
So *if* anything should be changed IMHO it would be the introduction of
a tagging system or something equivalent, together with a selection
mechanism based on that.
It's a good thing to make it easy to do things the right way, and difficult
to do things the wrong way. The danger is when you let your pre-conceived
notions of right and wrong trick you into making it difficult to do things
any way but YOUR way.
So far, the strongest argument I've seen against the OP's idea is that it's
not portable to Iron Python. That's a legitimate argument. All the rest
of the "You're not supposed to do it that way" arguments are just religion.
So far the reasons for introducing them haven't been compelling either.
Neither does it work over several testsuites, nor is it the only thing
that can order *results*, which you (rightly so) claimed as being useful.
Diez
--
http://mail.python.org/mailman/listinfo/python-list
