In message <aanlktikh3nv56qb8hxsxfzlni9jrtu-c2crf1shel...@mail.gmail.com>,
Nathan Beyer writes:
>
> For future milestone votes, I'd like to propose that we set the
> expectation of zero test failures. Every time we do a milestone and I
> run the build and tests I find myself looking at the test failures
> (there are ALWAYS failures) and ask 'are these expected'? If the
> failures are expected, they should be excluded and the tests should
> just pass.
>
> If this proposal is workable, immediately after this upcoming release,
> all 'expected' failures will be pulled into the exclusion lists.
I was thinking this too. The problem for me is that:
1) Our test coverage isn't brilliant so we need a way to exclude
individual tests not the tests for a whole class. We have too many
excluded tests already[0].
2) The JDWP test framework seems to pass most of the time on the 5.0
code base but seems to fail quite often on the 6.0 branch. I normally
resort to doing 10 test runs and comparing the results of all runs. I
typically see every test pass on at least one run. This leads me to
believe that the improved jdwp in the 6.0 branch is merely exposing race
conditions in the test framework. It would be great to fix the test
framework so that these are avoided. If we tried to exclude tests that
fail regularly on java6 jdwp we would probably end up removing most of
the tests. ;-(
I think solving 1 is really a pre-requisite to moving forward with your
(excellent) goal. We should make that a priority after this release. I
don't care if the solution is something fancy with junit 4 annotations
or something simple like adding:
if (Support_Excludes.isExcluded()) {
return;
}
to problematic tests where the support class just looks up the
class/method name of the caller in the exclude lists. (I actually
quite like the idea of doing the exclude list processing lazily at
run time rather than generating the list with ant.)
Solving 2) is a bit more involved.
Regards,
Mark.
[0] I am sure there are more inaccurate excludes entries but a quick scan
of excluded tests that reference Fixed JIRA shows:
HARMONY-3135 Fixed/Closed
[classlib][imageio] unexpected InterruptedException after loading png-image
This appears to be fixed so the test should be removed from the exclude list
and a new JIRA opened if the problem still exists. (There are possibly some
JNI issues with this code that I suspect might still cause problems.)
HARMONY-3443 Fixed/Closed
[classlib][awt][netbeans] Container.paint() throws NPE
I don't think this JIRA actually describes the problem with the excluded
test only the issue that exposed it. A separate JIRA should probably be
open for the real issue and that used in the exclude list reference.
HARMONY-3483 Fixed/Closed
[classlib][winx64][testing] initial exclude list for win on arch x86_64 should
be created
This is a catch-all exclude JIRA. Really, the excluded tests should reference
specific JIRA that describe the reason for their exclusion.
HARMONY-3617 Fixed/Resolved
[classlib][prefs] API test
org.apache.harmony.prefs.tests.java.util.prefs.AbstractPreferencesTest failed
on Win x86_64 release build
Comment after closing on JIRA says:
Alexey, did you un-exclude the test?
So I believe this one should be removed.
HARMONY-4181 Fixed/Resolved
[drlvm][jit] Assertion `min_idx != (~(unsigned)0)' failed on Dacapo chart
benchmark on x86_64
The java.awt.geom.AreaTest test passes for me on linux/x86_64 now. So I
think this should be removed.
HARMONY-4289 Fixed/Resolved
[classlib][prefs] test
org.apache.harmony.prefs.tests.java.util.prefs.AbstractPreferencesTest should
not depends on Internet connection
This appears to be fixed so it should be removed and the JIRA closed.
HARMONY-4307 Fixed/Closed
[classlib][luni] java.io.ObjectStreamField.getType() sometimes returns null
This looks like it was fixed (in 2007!) and the exclude list needs updating.
HARMONY-5131 Fixed/Closed
[classlib][ldap]Do ldap bind operation whne LdapContextImpl initial
Looks like fixing 5131 exposed a problematic test so a specific jira is needed
to describe the issue with the test.
