Pulling out of the various threads where we have been discussing, can we
agree on the problem :
We have unique problems compared to other Java projects because we need
to find a way to reliably test the things that are commonly expected to
be a solid point of reference - namely the core class library.
Further, we've been implicitly doing "integration testing" because - so
far - the only way we've been testing our code has been 'in situ' in the
VM - not in an isolated test harness. To me, this turns it into an
integration test.
Sure, we're using JUnit, but because of the fact we are implmenting core
java.* APIs, we aren't testing with a framework that has been
independently tested for correctness, like we would when testing any
other code.
I hope I got that idea across - I believe that we have to go beyond
normal testing approaches because we don't have a normal situation.
So I think there are three things we want to do (adopting the
terminology that came from the discussion with Tim and Leo ) :
1) implementation tests
2) spec/API tests (I'll bundle together)
3) integration/functional tests
I believe that for #1, the issues related to being on the bootclasspath
don't matter, because we aren't testing that aspect of the classes
(which is how they behave integrated w/ the VM and security system) but
rather the basic internal functioning.
I'm not sure how to approach this, but I'll try. I'd love to hear how
Sun, IBM or BEA deals with this, or be told why it isn't an issue :)
Implementation tests : I'd like to see us be able to do #1 via the
standard same-package technique (i.e. testing a.b.C w/ a.b.CTest) but
we'll run into a tangle of classloader problems, I suspect, becuase we
want to be testing java.* code in a system that already has java.* code.
Can anyone see a way we can do this - test the classlibrary from the
integration point of view - using some test harness + any known-good
JRE, like Sun's or IBM's?
Spec/API tests : these are, IMO, a kind of integration test, because
proper spec/API behavior *is* dependent on factors beyond the actual
code itself (like classloader configuration, and security context).
Because of this, the *.test.* pattern makes perfect sense. Assuming we
could produce something useful for #1 (i.e. a test harness/framework),
could we then augment it to simulate the classloader config + security
config that we'd get in a real VM? That will give us the ability to
test in isolation of the VM, and also let us 'break' the environment to
ensure that the code fails in a predictable way.
Intgration/functional : this is a whole range of things, from doing the
Spec/API tests in an actual VM, to the tests that exercise the code
through interaction with external systems (like network, RMI, GUI, etc)
***
Now, it might be suggested that we just ignore the implementation
testing (#1) and just do #2 and #3 as we are now, and hope we have a
good enough test suite. It could be argued that when Sun started, they
didn't have a known-good platform to do implementation testing on like
we do now. I don't know if that's true.
The difference is that we need to produce something of the same quality
as Sun's Java 5, not Sun's Java 1.0. We've had 11 years since 1.0 to
learn about testing, but they've had 11 years to get things solid.
What to do....
geir
