David,
I share Alan's concerns about the maintainability of this, but I don't
have a better answer ( without much reorganizing/tagging ).
Out of interest, how did you come up with the lists? For example,
java/net/URLClassLoader/closetest/GetResourceAsStream.java is in
needs_jdk. I presume because of a dependency on javac. The test could be
trivially changed to remove this dependency ( I don't expect you to have
the time to do this for all tests ). Maybe, over time, with some
socializing of these find of issues, enough tests can be changed/moved
to greatly reduce the number of specific tests explicitly listed in
these groups.
I resolved some of the ftp issues Alan referred to, a while back, and I
guess this helped reduce the need to have a few more specific tests
listed in needs_jdk. This really seems like a task for the actual
component teams, with higher level guidance.
Once this gets in I'll take a look to see if it is possible to remove
some of the obvious networking tests. What is the recommended test cycle
to verify that all this is working well after making changes?
-Chris.
On 09/03/2013 04:02 AM, David Holmes wrote:
webrev:
http://cr.openjdk.java.net/~dholmes/8024140/webrev/
Similar to what was recently done on the hotspot side, this introduces
profile-based test groups for the jdk regression tests.
The primary groups are:
- jdk
- jre
- compact3
- compact2
- compact2_minimal
- compact1
- compact1_minimal
The minimal VM is only supported on compact1 and compact2.
The groups are defined hierarchically in two forms:
- The need_xxx groups list all the tests that have a dependency on a
specific profile. This is either because it tests a feature in that
profile, or the test infrastructure uses a feature in that profile.
- The primary groups are defined in terms of the other primary groups
combined with the needs_xxx groups (including and excluding them as
appropriate). For example the jre can run all tests from compact3, plus
those from needs_jre, but excluding those from need_jdk.
The bottom group defines all the actual tests to be considered, simply
by listing the top-level test directories, and then excluding all the
needs_xxx groups.
To select a group of tests you use :<group-name>
Eg to run only those tests that can run on compact1 use:
jtreg :compact1
Of course you still need to point jtreg at the right kind of runtime
image (and give it a full JDK as the compile-jdk!); and if testing the
minimal VM you need to tell jtreg to select it using -javaoptions:-minimal
These top-level groups are not as useful standalone as they are for
hotspot due to the fact that there are so many regression tests that are
problematic in one way or another. In most cases what is desired is a
way to run sub-set of tests for a given profile eg all jdk_core tests
that can run on compact1. At present this would require a new group to
be defined for each permutation and that is not scalable. I hope that in
the near future jtreg will allow you to implicitly define such ad-hoc
groups via the command-line, simply by listing groups to include and
groups to exclude.
The full jtreg group facility is only available in the most recent jtreg
builds, so you will need to grab the latest nightly build, or latest
sources.
Note: once this is in place, anyone writing regression tests will need
to be aware of whether that test is limited to certain profiles and
update the group file accordingly. Sometimes it is not the item being
tested that determines the minimum needed profile, but the test
infrastructure eg if it uses XML.
See TEST.groups for more information.
Note: The initial group definitions proposed here are not complete.
There are over 4000 tests of the "desktop" that can not be readily
executed manually and it may be that a number of these tests will
require a JDK rather than the full JRE. Our full test processes will
discover this and update the lists as needed. (Or we fix the tests to
not need full JDKs.)
Thanks,
David
