On Tuesday, August 12, 2003, at 07:37 am, Bruce Snyder wrote:
This one time, at band camp, Daniel F. Savarese said:
DFS>In message <[EMAIL PROTECTED]
DFS>om>, Bruce Snyder writes:
DFS>>I'm wondering if there's a need for two kinds of tests - unit and
DFS>>spec. Of course, these two can be achieved in the same *Test.java files
DFS>>and separated simply by a comment line.
DFS>
DFS>The first thing that occurred to me when I started reading the testing
DFS>thread is that we're talking about at least two, if not more, different
DFS>kinds of testing here. The immediate need is straight up vanilla unit
DFS>testing and it should be kept separate from specification compliance
DFS>testing. It's easy enough to keep the two separate by maintaining
DFS>the tests in distinct directory trees. Ultimately, some sepcification
DFS>tests will alse be more basic unit tests, but they should be kept
DFS>with the other spec-related tests. There will be plenty of classes
DFS>and methods that have nothing to do with the specification and all
DFS>of those will require testing.
I absolutely believe that we need both vanilla unit tests as well as spec
compliance tests. But these two test types should reside in completely
different files. So I think that there are at least two problems to solve:
1) We need to decide on how to separate the two test types using the package structure.
The vanilla unit tests will reside in:
modules/<component-name>/src/test
I think that a completely separate directory structure is needed for the spec compliance tests. I propose that we create another directory as follows:
modules/<component-name>/src/spec-test
The spec compliance tests are wholly separate from the vanilla tests and the
two should not be mixed.
2) Another issue is how to label the spec compliance tests in order to parse them later for reporting, etc.
For this issue, not only should each spec compliance test be fully
documented via Javadoc, but we need to devise some meta tags for labeling
each test case for doclet parsing. I suggest that we use the following:
@spec name="J2EE" version="1.4" chapter-name="Transaction Management" chapter-number="4" chapter-section-name="Transactional JDBC Technology Support" chapter-section-number="4.2.7"
The above example may not be directly applicable to a given test. It's simply a chapter I chose at random.
Feedback? Better ideas?
I think this is a great idea. We just need to be a little careful with spec tests that we don't break any NDAs etc on the TCK.
But absolutely we should have unit tests for detailed parts of the code and functional tests to check that Geronimo does what we think it does. I like the doclet idea of relating functional tests to the section of the spec to which they apply.
Further down the line, when we've got clustering working, I'd also like us to try some system tests using multiple machines in a cluster - testing that a cluster of machines works as you'd expect. I've been working on & with Sysunit recently which is quite nice for this kinda thing: http://sysunit.codehaus.org)
James ------- http://radio.weblogs.com/0112098/
