Roman Kennke wrote:
Am Montag, den 07.03.2005, 21:49 -0600 schrieb Archie Cobbs:
In this thread:
http://lists.gnu.org/archive/html/classpath/2004-12/threads.html#00137
we talked about putting together somehow a list of Mauve tests that
all Classpath-based VMs should expect to pass, along with another
(complementary) list of those tests a Classpath-based VM should
expect to fail.
Sorry if I am completely naive here, but why should there be tests, that
a Classpath-based VM should be expected to fail?? Isn't the whole point
to (ideally) pass all tests?
Ideally, yes. But in practice, it isn't practical to pass all the tests ;-). What mauve does need is an easier way to highlight regressions. Currently the way people check for regressions is to do before & after mauve runs and diff the output to look for new FAILs, which is obviously pretty tedious. It would be nice if mauve could do this automatically by feeding it a file containing a list of known FAILs for a given VM, so that it would only let you know about failures that were previously expected to pass (and successes that were expected to fail!). This is complicated a little by mauve's design - there isn't a way to uniquely identify each "test" (ie each check() call), and sometimes some check()'s wont be run if a previous one failed, so fixing one bug can sometimes cause more FAILs. This means that it may not be clear if a new FAIL is actually a regression or a check that just wasn't run before. We could make the tests more course-grained from the point of view of the expected-failure machinery - ie track success/failure at the Testlet level, but that has the disadvantage of potentially not showing new regressions within a given a testlet that was already failing.
Bryce
_______________________________________________ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
