I've completed the first phase of refactoring, separating out the
environment setup and test run into separate interfaces as per
appended. This should allow the setup and teardown and execution
of the tests to remain separate concerns. At present all tests
implement both interfaces, so nothing has really changed, yet.
When the qa test run is complete and has passed all tests, I'll
upload the changes for review.
The next stage will be the separation of setup and test execution
for the tests that have experienced failures recently in trunk.
This will simplify test execution and should make it easier to
manage shared state.
From there, perhaps we should merge the changes made in trunk back
in incrementally.
Since a considerable amount of work is required to refactor all
tests, I suggest we migrate the tests to the new format slowly over
time on an as needed basis.
The names of interfaces and methods can be changed if necessary:
public interface TestFactory {
/**
* Performs setup for this test. If an exception is thrown, the
* <code>tearDown</code> method will still be called.
*
* @param config the test properties
* @throws Exception if setup fails for any reason
*/
Test setup(QAConfig config) throws Exception;
/**
* Tears down any setup that was needed to run the test. This
method is
* called even if setup throws an exception, and so must be
designed to
* be tolerant of test state that is not completely initialized.
*/
void tearDown();
}
public interface Test {
/**
* Failure type. Indicates that a test failed because of a
recognized
* environmental problem. These will typically be due to
* intermittent or recurring configuration dependent failures
* which have been determined to result from problems external
* to the product.
*/
int ENV = 1;
/**
* Failure type. Indicates that a test failed but the reason for
* failure may be external to the product being tested.
*/
int INDEF = 2;
/**
* Failure type indicating no failure occured. This is a special
* value which should be returned by a FailureAnalyzer to indicate
* that the exception represents a passing condition for the test
*/
int PASSED = 6;
/**
* Failure type indicating that the test should be rerun.
*/
int RERUN = 4;
/**
* Failure type indicating that the test should not be run.
*/
int SKIP = 3;
/**
* Failure type. Indicates that a test failed because the product
failed,
* or an unrecognized environmental failure occured.
*/
int TEST = 0;
/**
* Failure type (for analyzers) indicating failure not recognized
*/
int UNKNOWN = 5;
/**
* Execute the body of the test.
*
* @throws Exception if the test fails for any reason
*/
void run() throws Exception;
}
Regards,
Peter.
Peter Firmstone wrote:
Presently most tests have a lot of shared mutable state.
This isn't helped by the setup(QAConfig) mutator method in the Test
interface appended.
An alternative might be:
public Test setup(QAConfig config) throws Exception;
This would allow the test to return another Test object, fully
constructed with immutable state.
It would also allow the test to create an object that can run
several other Test objects, rather than relying on inheritance.
Thoughts?
/**
* This interface must implemented by all tests supported by the test
* harness. The following sequence of events is performed for each
* test:
* <p><ul>
* <li> the test class is instantiated and it's no-arg constructor
called.
* The constructor will typically perform minimal
initialization,
* since the test does not have access to the test environment
* <li> the setup method is called, passing the config object.
* This provides an opportunity for performing any test setup
* that relies on accessing configuration parameters.
* <li> the run method is called to run the test
* <li> the teardown method is called to clean up state or services
* created by the test. This method is called
* even if <code>setup</code> or <code>run</code> throws an
exception.
* </ul>
*/
public interface Test {
/**
* Failure type. Indicates that a test failed because the
product failed,
* or an unrecognized environmental failure occured.
*/
public final static int TEST = 0;
/**
* Failure type. Indicates that a test failed because of a
recognized
* environmental problem. These will typically be due to
* intermittent or recurring configuration dependent failures
* which have been determined to result from problems external
* to the product.
*/
public final static int ENV = 1;
/**
* Failure type. Indicates that a test failed but the reason for
* failure may be external to the product being tested.
*/
public final static int INDEF = 2;
/** Failure type indicating that the test should not be run. */
public final static int SKIP = 3;
/** Failure type indicating that the test should be rerun. */
public final static int RERUN = 4;
/** Failure type (for analyzers) indicating failure not
recognized */
public final static int UNKNOWN = 5;
/**
* Failure type indicating no failure occured. This is a special
* value which should be returned by a FailureAnalyzer to indicate
* that the exception represents a passing condition for the test
*/
public final static int PASSED = 6;
/**
* Performs setup for this test. If an exception is thrown, the
* <code>tearDown</code> method will still be called.
*
* @param config the test properties
* @throws Exception if setup fails for any reason
*/
public void setup(QAConfig config) throws Exception;
/**
* Execute the body of the test.
*
* @throws Exception if the test fails for any reason
*/
public void run() throws Exception;
/**
* Tears down any setup that was needed to run the test. This
method is
* called even if setup throws an exception, and so must be
designed to
* be tolerant of test state that is not completely initialized.
*/
public void tearDown();
}