2010/8/24 Patricia Shanahan <[email protected]>
On 8/22/2010 4:57 PM, Peter Firmstone wrote:
...
Thanks Patricia, that's very helpful, I'll figure it out where I
went
wrong this week, it really shows the importance of full test
coverage.
...
I strongly agree that test coverage is important. Accordingly, I've
done
some analysis of the "ant qa.run" output.
There are 1059 test description (*.td) files that exist, and are
loaded
at
the start of "ant qa.run", but that do not seem to be run. I've
extracted
the top level categories from those files:
constraint
discoveryproviders_impl
discoveryservice
end2end
eventmailbox
export_spec
io
javaspace
jeri
joinmanager
jrmp
loader
locatordiscovery
lookupdiscovery
lookupservice
proxytrust
reliability
renewalmanager
renewalservice
scalability
security
start
txnmanager
I'm sure some of these tests are obsolete, duplicates of tests in
categories that are being run, or otherwise inappropriate, but there
does
seem to be a rich vein of tests we could mine.
The QA harness loads all .td files under the "spec" and "impl"
directories
when starting and only witholds the ones that are tagged with the
categories
that we specify from the Ant target.
Whenever a test is really obsolete or otherwise not supposed to run,
it
is
marked with a "SkipTestVerifier" in its .td file.
Most of these are genuine and should be run though.
There are more categories than the ones you mention above, for
instance:
"spec", "id", "id_spec", etc.
Also, some tests are tagged with multiple categories and as such
duplicates
can exist when assembling the list of tests to run.
The reason not all of them are run (by Hudson) now is that we give a
specific set of test categories that are known (to me) to run
smoothly.
There are many others that are not run (by default) because issue(s)
are
present with one or more of the tests in that category.
I completely agree with the fact that we should not exclude complete
test
categories because of one test failing.
What we probably should do is tag any problematic test (due to
infrastructure or other reasons) with a SkipTestVerifier for the time
being
so that it is not taken into account by the QA harness for now.
That way, we can add all test categories to the default Ant run.
However, this would take a large amount of time to run (I've tried it
once,
and killed the process after several days), which brings us to your
next
point:
Part of the problem may be time to run the tests. I'd like to propose
splitting the tests into two sets:
1. A small set that one would run in addition to the relevant tests,
whenever making a small change. It should *not* be based on skipping
complete categories, but on doing those tests from each category that
are
most likely to detect regression, especially regression due to
changes
in
other areas.
Completely agree. However, most of the QA tests are not clear unit or
regression tests. They are more integration/conformance tests that
test
the
requirements of the spec and its implementation.
Identifying the list of "right" tests to run as part of the small set
you
mention would require going through all 1059 test descriptions and
their
sources.
2. A full test set that may take a lot longer. In many projects, there
is
a
"nightly build" and a test sequence that is run against that build.
That
test sequence can take up to 24 hours to run, and should be as
complete
as
possible. Does Apache have infrastructure to support this sort of
operation?
Again, completely agree. I'm sure Apache supports this through Hudson.
We
could request to setup a second build job, doing nightly builds and
running
the whole test suite. Think this is the only way to make running the
complete QA suite automatically practical.
Are there any tests that people *know* should not run? I'm thinking
of
running the lot just to see what happens, but knowing ones that are
not
expected to work would help with result interpretation.
See above, tests of that type should have already been tagged to be
skipped
by the good people that donated this test suite.
I've noticed that usually, when a SkipTestVerifier is used in a .td
file,
someone has put some comments in there to explain why it was tagged as
such.
Patricia
