> OK, but you still have to clean out your database before you start each
> independent chunk of your test suite, otherwise you start from an
> unknown state.
In a lot of cases, this isn't true. This pattern is quite common:
1. Insert entity.
2. Test with entity just inserted.
Since all that my test cares about is the unique entity or entities, the
state of the rest of database doesn't matter. The state the matters is
in a "known state".
> What about when you're not running under Jenkins. Like when you're
> writing and testing your code. You still need to start testing from a
> known state each time, which means you must clean out the database at
> startup.
We have a cron job that runs overnight to clean up anything that was
missed in Jenkin's runs.
We expect our tests to generally work in the face of a "dirty" database.
If they don't, that's considered a flaw in the test. This is important
to run several tests against the same database at the same time. Even if
we did wipe the database for we tested, all the other tests running in
parallel would be considered to making the database "dirty". Thus, if a
pristine database is a requirement, only one test could run against the
database at the time.
We run our tests 4x parallel against the same database, matching the
cores available in the machine.
We also share the same database between developers and the test suite.
This "dirty" environment can work like a feature, as it can sometimes
produce unexpected and "interesting" states that were missed by a
clean-room testing approach that so carefully controlled the environment
that some real-world possibilities.
For example, perhaps a column allows null values, but the test suite
never tests that case because it "shouldn't happen". A developer might
manually create a value, which could expose a problem spot-- perhaps the
field should be "not null", or the app should handle that case gracefully.
A perfect clean-room approach would cover all these cases, but I don't
assume our tests are perfect.
Mark
