Tim Ellison wrote:
Stepan Mishura wrote:
A test framework is quite essential part and we should define it on early
stage.
I think first we should agree on which test we are going to provide and form
requirements to them. There are no doubts that we will provide unit tests
for classlib. So unit tests are provided by code authors in JUnit format and
intended to verify all code functionality. They are placed in the same
package as tested code and run on the bootclasspath.
Anything to add? Are there any objections?
You have seen that there *are* objections to putting the test code into
the implementation packages and running it on the bootclasspath. I can
restate the reasons if it helps.
Yes please. Because my suggestion of a test framework was directly
targeted at that well-founded objection of yours.
And I do think we need to step back and think about unit tests vs API
tests vs functional tests vs whatever.
Conformance tests we will receive form TCK and what about other test suites:
functional, stress tests, performance? Will we create a separate framework
for each?
Running the TCK is a certification step, not a suite of tests that we
develop against to get a progressively compliant implementation.
That is correct, but we will be using the TCK as part of our CI. I
learned that lesson well in Geronimo.
I believe we need:
- implementation tests : using the 'trick' that Geir refers to below to
expose some internal APIs to the application code. We would only expect
these to pass on Harmony code.
yes. This is normally referred to as a 'unit test'. No trick is needed
if we have a test harness to run in, rather than putting on the boot
classpath of the VM du jour.
- API unit tests : driving the published APIs, these should pass on all
compliant implementations (modulo known bugs in those impls.).
yes - these are the "API tests", which IMO do *not* belong in the java.*
packages.
- then, as you say, additional 'system', 'stress', 'performance', tests
which are likely to fall outside the unit test framework.
Yes.
I'd prefer if we distinguished between 'unit tests', which do test the
internals of the implementations, and 'API tests', which live on the API
surface area. No internals, no package, no private, etc.
geir
Regards,
Tim
Thanks,
Stepan Mishura,
Intel Middleware Products Division
On 1/16/06, Geir Magnusson Jr <[EMAIL PROTECTED]> wrote:
One thing that's popped up on the "Test suite layout" thread is the
thought that we need to b0rk the canonical package and naming
conventions for unit tests in order to be able to run things on the boot
classpath of the VM. I think this issue is important enough and
fundamental enough to warrant it's own thread.
First, do we really need to do this? I thought that we (Tim and I) had
informally discussed this at ApacheCon and came to some good conclusion
where we were able to figure out a trick.
Second, shouldn't we think about providing a test environment in which
we can completely control the environment - we can test the class
library in a container that can be run in any VM so we have full control
over security and other issues?
Of course, I'd like both. If we do have the "trick" that we talked
about, then we can use canonical JUnit (or TestNG) naming and package
conventions, which I think is important.
geir
