1) Guitests. I'll see what I can find. Probably something minor. I
know the "tests" target runs all test classes ending in "*Test" and
ignores "AllTests", while Eclipse (and probably guitests) just runs
the AllTests classes. It's likely that one or more of the AllTests
classes is failing to include, oh, about 34 tests. :)
2) Graceful LDAP fail (inside the tests themselves). Any ideas on
how
to implement? The easy way would be to look for a localhost listener
on 4890 (where the OpenLDAP test fixture listens) and then not run
the
tests if it isn't found. Should they FAIL or PASS in that case? It
sounds like passing is the right thing to do.
3) Differences in your test pass rate versus mine. Not sure why your
"ant tests" run would produce different results than mine. I did try
running mine with a completely new, checked-out branch. Because I
can't know what changes you might have in your local branch, could
you
check out a clean copy and diff the tree versus yours? SOMETHING is
different. Also, I'd like to know what Harry and others are seeing.
Gents, any clues?
I agree that all three methods should return the same number of test
cases, and pass/fail the same ways. I also agree that tests should
be
self-contained. That was part of the rationale for the Ant script
tweaks I checked in recently.
Eclipse, by the way, hasn't been reliable for me, for testing, for a
while. I tend to exhaust memory somewhere around
JSPWikiMarkupParser.
But I haven't tried it in the last few months (i.e. before my
massive
bug-hunting campaign).
Andrew
On Tue, Oct 27, 2009 at 3:25 AM, Janne Jalkanen
<[email protected]> wrote:
Interestingly, I applied your most recent checkins applied (and
I have
small one patch to JSPWikiMarkupParserTest that I haven't
checked in).
I am running 100% clean, with no errors. Total number of tests:
1024
-- a nice round number. :) WikiEngineTest.testOldVersionVars has
been running fine for me for a while.
There's no way it should've run, unless you have some old code/
config
files
lying around. Can you check out a previous version to a clean
directory
and
see if it runs?
As a control case, I also checked out a new built from trunk, and
simply typed 'ant tests'. I used a vanilla build with absolutely
no
customizations, even to build.properties. It ran completely
clean also
except for 1 JSPWikiMarkupParserTest test (because I haven't
checked
in that fix), 1024 tests total.
Running the AllTests from Eclipse or with "ant guitests" results
in 990
test
cases. "ant tests" is the only one giving 1024 tests, and I get 12
failures
and 14 errors for it. LdapAuthorizerTest, LdapUserDatabaseTest and
XMLUserDatabaseTest all fail with all tests.
What I find odd is that guitests and tests targets should give
the same
results, since they both are run from build.xml.
The only other item causing the discrepancy would be if you
don't have
a local LDAP server running for the LDAP tests. Those should
cause, at
most, 14 failures or errors. I'll add in some code to build.xml
to set
up the LDAP fixtures and/or disable the tests if the OpenLDAP
executable isn't available.
I think it's probably a better idea to do the test directly in
the tests
itself. The JCR TCK throws a NonExecutableException when the
test case
cannot be executed (and this shows up as a passed test).
I think it's important that all three methods give the same
number of
test
cases; if the number is not reliable, it's too easy to forget to
run
certain
tests.
Also, I sometimes run all tests for a given package from within
Eclipse.
I'd
like the test cases to be self-contained enough so that I don't
have to
remember which tests are supposed to run under which conditions.
/Janne
