Sounds like we have a few issues here: 1) Guitests. I'll see what I can find. Probably something minor. I know the "tests" target runs all test classes ending in "*Test" and ignores "AllTests", while Eclipse (and probably guitests) just runs the AllTests classes. It's likely that one or more of the AllTests classes is failing to include, oh, about 34 tests. :)
2) Graceful LDAP fail (inside the tests themselves). Any ideas on how to implement? The easy way would be to look for a localhost listener on 4890 (where the OpenLDAP test fixture listens) and then not run the tests if it isn't found. Should they FAIL or PASS in that case? It sounds like passing is the right thing to do. 3) Differences in your test pass rate versus mine. Not sure why your "ant tests" run would produce different results than mine. I did try running mine with a completely new, checked-out branch. Because I can't know what changes you might have in your local branch, could you check out a clean copy and diff the tree versus yours? SOMETHING is different. Also, I'd like to know what Harry and others are seeing. Gents, any clues? I agree that all three methods should return the same number of test cases, and pass/fail the same ways. I also agree that tests should be self-contained. That was part of the rationale for the Ant script tweaks I checked in recently. Eclipse, by the way, hasn't been reliable for me, for testing, for a while. I tend to exhaust memory somewhere around JSPWikiMarkupParser. But I haven't tried it in the last few months (i.e. before my massive bug-hunting campaign). Andrew On Tue, Oct 27, 2009 at 3:25 AM, Janne Jalkanen <[email protected]> wrote: >> Interestingly, I applied your most recent checkins applied (and I have >> small one patch to JSPWikiMarkupParserTest that I haven't checked in). >> I am running 100% clean, with no errors. Total number of tests: 1024 >> -- a nice round number. :) WikiEngineTest.testOldVersionVars has >> been running fine for me for a while. > > There's no way it should've run, unless you have some old code/config files > lying around. Can you check out a previous version to a clean directory and > see if it runs? > >> As a control case, I also checked out a new built from trunk, and >> simply typed 'ant tests'. I used a vanilla build with absolutely no >> customizations, even to build.properties. It ran completely clean also >> except for 1 JSPWikiMarkupParserTest test (because I haven't checked >> in that fix), 1024 tests total. > > Running the AllTests from Eclipse or with "ant guitests" results in 990 test > cases. "ant tests" is the only one giving 1024 tests, and I get 12 failures > and 14 errors for it. LdapAuthorizerTest, LdapUserDatabaseTest and > XMLUserDatabaseTest all fail with all tests. > > What I find odd is that guitests and tests targets should give the same > results, since they both are run from build.xml. > >> The only other item causing the discrepancy would be if you don't have >> a local LDAP server running for the LDAP tests. Those should cause, at >> most, 14 failures or errors. I'll add in some code to build.xml to set >> up the LDAP fixtures and/or disable the tests if the OpenLDAP >> executable isn't available. > > I think it's probably a better idea to do the test directly in the tests > itself. The JCR TCK throws a NonExecutableException when the test case > cannot be executed (and this shows up as a passed test). > > I think it's important that all three methods give the same number of test > cases; if the number is not reliable, it's too easy to forget to run certain > tests. > > Also, I sometimes run all tests for a given package from within Eclipse. I'd > like the test cases to be self-contained enough so that I don't have to > remember which tests are supposed to run under which conditions. > > /Janne >
