Oliver Deakin wrote:
George Harley wrote:
Oliver Deakin wrote:
George Harley wrote:
<SNIP!>
Here the annotation on MyTestClass applies to all of its test methods.
So what are the well-known TestNG groups that we could define for
use inside Harmony ? Here are some of my initial thoughts:
* type.impl -- tests that are specific to Harmony
So tests are implicitly API unless specified otherwise?
I'm slightly confused by your definition of impl tests as "tests
that are
specific to Harmony". Does this mean that impl tests are only
those that test classes in org.apache.harmony packages?
I thought that impl was our way of saying "tests that need to go on
the bootclasspath".
I think I just need a little clarification...
Hi Oliver,
I was using the definition of implementation-specific tests that we
currently have on the Harmony testing conventions web page. That is,
implementation-specific tests are those that are dependent on some
aspect of the Harmony implementation and would therefore not pass
when run against the RI or other conforming implementations. It's
orthogonal to the classpath/bootclasspath issue.
OK, that's what I imagined you meant. IMHO using api and impl
in this way makes the most sense (since, as you say, they do not
really relate to the classpath/bootclasspath issue).
So do we also need a pair of groups for classpath/bootclasspath
tests? I'm assuming that this is how we would handle this distinction,
rather than organising them into separate directories in the file system.
Hi Oliver,
I guess that would be possible but, given that the intended classloader
of a test is probably not going to vary much, I would have no objection
to that distinction being made in separate directories in the file
system. In other words, I see the benefits of a suite-driven system
(TestNG or whatever) as being more applicable to those attributes of
tests that we identify as being more susceptible to change. Examples
like "currently only works on platform X" or "broken on platform Y" or
"only works when run against Harmony" seem like good candidates,
especially when the time comes to grow Harmony on other platforms. Right
now, I don't see the runtime classloader fitting into that category.
Maybe I'm wrong.
* state.broken.<platform id> -- tests bust on a specific platform
* state.broken -- tests broken on every platform but we want to
decide whether or not to run from our suite configuration
* os.<platform id> -- tests that are to be run only on the
specified platform (a test could be member of more than one of these)
And the defaults for these are an unbroken state and runs on any
platform.
That makes sense...
Will the platform ids be organised in a similar way to the platform ids
we've discussed before for organisation of native code [1]?
The actual string used to identify a particular platform can be
whatever we want it to be, just so long as we are consistent. So,
yes, the ids mentioned in the referenced email would seem a good
starting point. Do we need to include a 32-bit/64-bit identifier ?
I cannot immediately think of any obvious 32/64-bit specific tests
that we
might require in the future (although Id be interested to know if anyone
can think of any!). However, if the need did arise, then I would
suggest that this is incorporated as another tag on the end of the
group name e.g. os.linux.ppc.32.
Right.
So all tests are, by default, in an all-platforms (or shared) group.
If a test fails on all Windows platforms, it is marked with
state.broken.windows.
If a test fails on Windows but only on, say, amd hardware,
it is marked state.broken.windows.amd.
Yes. Agreed.
Then when you come to run tests on your windows amd machine,
you want to include all tests in the all-platform (shared) group,
os.windows and os.windows.amd, and exclude all tests in
the state.broken, state.broken.windows and state.broken.windows.amd
groups.
Does this tally with what you were thinking?
Yes, that is the idea.
What does everyone else think ? Does such a scheme sound reasonable ?
I think so - it seems to cover our current requirements. Thanks for
coming up with this!
Thanks, but I don't see it as final yet really. It would be great to
prove the worth of this by doing a trial on one of the existing
modules, ideally something that contains tests that are
platform-specific.
Thanks for volunteering... ;)
...but seriously, do any of our modules currently contain platform
specific tests?
Have you attempted a TestNG trial on any of the modules (with or
without platform specific tests) and, if so, was it
simpler/harder/better/worse
than our current setup?
Regards,
Oliver
Yes, I believe that there are platform specific tests out there. NIO and
auth are two examples that spring to mind. My own experiments with
TestNG have so far not been with our modules but in separate projects.
Running the tests didn't seem all that different to what we currently
use but the speed with which the configuration could be altered was a
lot better, not to mention the degree of control available.
Best regards,
George
Best regards,
George
Regards,
Oliver
[1]
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/[EMAIL PROTECTED]
Thanks for reading this far.
Best regards,
George
George Harley wrote:
Hi,
Just seen Tim's note on test support classes and it really caught
my attention as I have been mulling over this issue for a little
while now. I think that it is a good time for us to return to the
topic of class library test layouts.
The current proposal [1] sets out to segment our different types
of test by placing them in different file locations. After looking
at the recent changes to the LUNI module tests (where the layout
guidelines were applied) I have a real concern that there are
serious problems with this approach. We have started down a track
of just continually growing the number of test source folders as
new categories of test are identified and IMHO that is going to
bring complexity and maintenance issues with these tests.
Consider the dimensions of tests that we have ...
API
Harmony-specific
Platform-specific
Run on classpath
Run on bootclasspath
Behaves different between Harmony and RI
Stress
...and so on...
If you weigh up all of the different possible permutations and
then consider that the above list is highly likely to be extended
as things progress it is obvious that we are eventually heading
for large amounts of related test code scattered or possibly
duplicated across numerous "hard wired" source directories. How
maintainable is that going to be ?
If we want to run different tests in different configurations then
IMHO we need to be thinking a whole lot smarter. We need to be
thinking about keeping tests for specific areas of functionality
together (thus easing maintenance); we need something quick and
simple to re-configure if necessary (pushing whole directories of
files around the place does not seem a particularly lightweight
approach); and something that is not going to potentially mess up
contributed patches when the file they patch is found to have been
recently pushed from source folder A to B.
To connect into another recent thread, there have been some posts
lately about handling some test methods that fail on Harmony and
have meant that entire test case classes have been excluded from
our test runs. I have also been noticing some API test methods
that pass fine on Harmony but fail when run against the RI. Are
the different behaviours down to errors in the Harmony
implementation ? An error in the RI implementation ? A bug in the
RI Javadoc ? Only after some investigation has been carried out do
we know for sure. That takes time. What do we do with the test
methods in the meantime ? Do we push them round the file system
into yet another new source folder ? IMHO we need a testing
strategy that enables such "problem" methods to be tracked easily
without disruption to the rest of the other tests.
A couple of weeks ago I mentioned that the TestNG framework [2]
seemed like a reasonably good way of allowing us to both group
together different kinds of tests and permit the exclusion of
individual tests/groups of tests [3]. I would like to strongly
propose that we consider using TestNG as a means of providing the
different test configurations required by Harmony. Using a
combination of annotations and XML to capture the kinds of
sophisticated test configurations that people need, and that
allows us to specify down to the individual method, has got to be
more scalable and flexible than where we are headed now.
Thanks for reading this far.
Best regards,
George
[1]
http://incubator.apache.org/harmony/subcomponents/classlibrary/testing.html
[2] http://testng.org
[3]
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200606.mbox/[EMAIL PROTECTED]
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
