Michael G Schwern wrote:
Ovid wrote:
However, if you use the '-s' switch to shuffle your tests and bailout
is not first, then some tests will run until the BAIL_OUT is hit. This
seems to violate the principle that tests should be able to run in any
order without dependencies.
It doesn't violate the principle since the tests are not dependent on BAIL_OUT
happening, its just a convenience. The tests should still run fine in any
order, it'll just be a lot noisier.
This nails it right on the head.
Correctness should rely on order.
However, optimisation of the testing process may be built into the order.
For example, the order of the test scripts is generally from simplest to
most complex functionality, so that the best failure to try to fix is
generally the first one. This helps optimises the debugging process.
All BAIL_OUT does is save you some time by getting to the end of the
test scripts faster.
Personally, I've always wanted a per-file bail_out as well, that can
just abort the current test script, rather than the entire testing process.
Schwern? :)
Adam K
