Geir Magnusson Jr wrote: > Tim Ellison wrote: >> Stepan Mishura wrote: >>> A test framework is quite essential part and we should define it on >>> early >>> stage. >>> >>> I think first we should agree on which test we are going to provide >>> and form >>> requirements to them. There are no doubts that we will provide unit >>> tests >>> for classlib. So unit tests are provided by code authors in JUnit >>> format and >>> intended to verify all code functionality. They are placed in the same >>> package as tested code and run on the bootclasspath. >>> >>> Anything to add? Are there any objections? >> >> You have seen that there *are* objections to putting the test code into >> the implementation packages and running it on the bootclasspath. I can >> restate the reasons if it helps. >> > > Yes please. Because my suggestion of a test framework was directly > targeted at that well-founded objection of yours.
Assuming that you are familiar with the classloading model in Java [1], then you'll know that classes loaded from the bootstrap classloader get special privileges throughout classlibrary and VM code (grep for places where we check that the classloader == null). Running 'all tests in the same package and on the bootclasspath' will avoid loading any other code (framework, helpers, etc.) with the application classloader or extensions classloader, which makes it singularly unlike any application code that we are trying to support. It also prevents the API tests being run in any other context like a webapp or whatever that sets a specific context classloader. Take JNDI as an example. The framework code will be implemented in a JAR that lives on the bootclasspath in a Harmony runtime, but the provider can be specified by applications on the application classpath. The JNDI framework has to load classes from the provider by setting the thread context classloader to do an 'inverse' delegation (if you see what I mean). By flattening all the classloaders to a single instance you can have code that passes all your tests, but just won't work in real life. [1] http://www-128.ibm.com/developerworks/library/j-dyn0429/index.html > And I do think we need to step back and think about unit tests vs API > tests vs functional tests vs whatever. > > >>> Conformance tests we will receive form TCK and what about other test >>> suites: >>> functional, stress tests, performance? Will we create a separate >>> framework >>> for each? >> >> Running the TCK is a certification step, not a suite of tests that we >> develop against to get a progressively compliant implementation. > > That is correct, but we will be using the TCK as part of our CI. I > learned that lesson well in Geronimo. > >> >> I believe we need: >> >> - implementation tests : using the 'trick' that Geir refers to below to >> expose some internal APIs to the application code. We would only expect >> these to pass on Harmony code. > > yes. This is normally referred to as a 'unit test'. No trick is needed > if we have a test harness to run in, rather than putting on the boot > classpath of the VM du jour. > >> >> - API unit tests : driving the published APIs, these should pass on all >> compliant implementations (modulo known bugs in those impls.). > > yes - these are the "API tests", which IMO do *not* belong in the java.* > packages. > >> >> - then, as you say, additional 'system', 'stress', 'performance', tests >> which are likely to fall outside the unit test framework. > > Yes. > > I'd prefer if we distinguished between 'unit tests', which do test the > internals of the implementations, and 'API tests', which live on the API > surface area. No internals, no package, no private, etc. I think the mists are clearing (for me), perhaps it is a terminology thing. I agree that there will be writing "implementation tests", and that they will require some code to be placed on the bootclasspath (or the use of reflection) to call package private methods. In my world, 'implementation tests' and 'API tests' can both be Unit tests (i.e. that's the style of test, not what they are testing). I disagree with Stepan that all tests should be defined in the same package as the implementation and be run on the bootclasspath. boot classloader <- helpers to expose non-public | methods/fields (optional) |_ ext classloader <- my extensions (XML, security,...) | |_ app classloader <- test logic here | <- impl. tests use helpers/reflect' |_ <any other random classloader> <- .. or tests run here <- .. or here Regards, Tim > geir > >> >> Regards, >> Tim >> >>> Thanks, >>> Stepan Mishura, >>> Intel Middleware Products Division >>> >>> >>> On 1/16/06, Geir Magnusson Jr <[EMAIL PROTECTED]> wrote: >>>> One thing that's popped up on the "Test suite layout" thread is the >>>> thought that we need to b0rk the canonical package and naming >>>> conventions for unit tests in order to be able to run things on the >>>> boot >>>> classpath of the VM. I think this issue is important enough and >>>> fundamental enough to warrant it's own thread. >>>> >>>> First, do we really need to do this? I thought that we (Tim and I) had >>>> informally discussed this at ApacheCon and came to some good conclusion >>>> where we were able to figure out a trick. >>>> >>>> Second, shouldn't we think about providing a test environment in which >>>> we can completely control the environment - we can test the class >>>> library in a container that can be run in any VM so we have full >>>> control >>>> over security and other issues? >>>> >>>> Of course, I'd like both. If we do have the "trick" that we talked >>>> about, then we can use canonical JUnit (or TestNG) naming and package >>>> conventions, which I think is important. >>>> >>>> geir >>>> >>>> >> > -- Tim Ellison ([EMAIL PROTECTED]) IBM Java technology centre, UK.
