I ran a batch of the previously ignored QA tests overnight. I got 156
passes and 64 failures. This is nowhere near as bad as it sounds,
because many of the failures were clusters of related tests failing in
similar ways, suggesting a single problem affecting the base
infrastructure for the test category. Some of the failures may relate to
the known regression that Peter is going to look at this week.
Also, it is important to remember that the bugs may be in the tests, not
in the code under test. A test may be obsolete, depending on behavior
that is no longer supported.
I do think there is a good enough chance that at least one of the
failures represents a real problem, and an opportunity to improve River,
that I plan to start a background activity looking at failed tests to
see what is going on. The objective is to do one of three things for
each cluster of failures:
1. Fix River.
2. Fix the test.
3. Decide the test is unfixable, and delete it. There is no point
spending disk space, file transfer time, and test load time on tests we
are never going to run.
Running the subset I did last night took about 15 hours, but that
included a lot of timeouts.
Patricia
