George Harley wrote:
Hi,
Just seen Tim's note on test support classes and it really caught my
attention as I have been mulling over this issue for a little while
now. I think that it is a good time for us to return to the topic of
class library test layouts.
The current proposal [1] sets out to segment our different types of
test by placing them in different file locations. After looking at the
recent changes to the LUNI module tests (where the layout guidelines
were applied) I have a real concern that there are serious problems
with this approach. We have started down a track of just continually
growing the number of test source folders as new categories of test
are identified and IMHO that is going to bring complexity and
maintenance issues with these tests.
Yes, you'll see our ant scripts get more and more complex. :-)
Consider the dimensions of tests that we have ...
API
Harmony-specific
Platform-specific
Run on classpath
Run on bootclasspath
Behaves different between Harmony and RI
Stress
...and so on...
If you weigh up all of the different possible permutations and then
consider that the above list is highly likely to be extended as things
progress it is obvious that we are eventually heading for large
amounts of related test code scattered or possibly duplicated across
numerous "hard wired" source directories. How maintainable is that
going to be ?
If we want to run different tests in different configurations then
IMHO we need to be thinking a whole lot smarter. We need to be
thinking about keeping tests for specific areas of functionality
together (thus easing maintenance); we need something quick and simple
to re-configure if necessary (pushing whole directories of files
around the place does not seem a particularly lightweight approach);
and something that is not going to potentially mess up contributed
patches when the file they patch is found to have been recently pushed
from source folder A to B.
To connect into another recent thread, there have been some posts
lately about handling some test methods that fail on Harmony and have
meant that entire test case classes have been excluded from our test
runs. I have also been noticing some API test methods that pass fine
on Harmony but fail when run against the RI. Are the different
behaviours down to errors in the Harmony implementation ? An error in
the RI implementation ? A bug in the RI Javadoc ? Only after some
investigation has been carried out do we know for sure. That takes
time. What do we do with the test methods in the meantime ? Do we push
them round the file system into yet another new source folder ? IMHO
we need a testing strategy that enables such "problem" methods to be
tracked easily without disruption to the rest of the other tests.
It's really worth thinking about our testing strategy...
A couple of weeks ago I mentioned that the TestNG framework [2] seemed
like a reasonably good way of allowing us to both group together
different kinds of tests and permit the exclusion of individual
tests/groups of tests [3]. I would like to strongly propose that we
consider using TestNG as a means of providing the different test
configurations required by Harmony. Using a combination of annotations
and XML to capture the kinds of sophisticated test configurations that
people need, and that allows us to specify down to the individual
method, has got to be more scalable and flexible than where we are
headed now.
Will try to study TestNG before I can give comment ;-)
Thanks for reading this far.
Best regards,
George
[1]
http://incubator.apache.org/harmony/subcomponents/classlibrary/testing.html
[2] http://testng.org
[3]
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200606.mbox/[EMAIL PROTECTED]
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
Richard Liang
China Software Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
