Investigating a bit more the StandardTests, I find that 7832 tests pass if I ignore base1.skip and opt1.skip whereas only ~6542 pass when skip files are used. This means that ~1300 tests that previously didn't pass (or didn't exist) are now green, that's a good news. On the other side, this means as well that they may become red again without notice as they are not currently considered.
As a consequence, I propose following: - build a "positive" list of tests that pass - run *all* the standard tests from the tests dir - report a failure if a test from the positive list fails - report a failure if a test that is not in the positive list passes I believe that it is better to work with a positive list because the tests set is not under the control of the Rhino committers and I guess that newly added tests are more likely to fail than to pass (new language features, ...). Tests that depends on OS or computer capacities still need to be handled separately. Thoughts? Cheers, Marc. -- Web: http://www.efficient-webtesting.com Blog: http://mguillem.wordpress.com Marc Guillemot wrote: > Norris Boyd wrote: >> ... >> Which tests are you referring to? I run the tests that are shared with >> SpiderMonkey using org.mozilla.javascript.drivers.JsDriver, which has >> skip lists, see http://developer.mozilla.org/En/Running_the_Rhino_tests. >> Those tests all pass for me, although I need to resync to pull down >> any new tests that have been introduced on the SpiderMonkey side. >> >> How are you running tests? Using >> org.mozilla.javascript.drivers.StandardTests? > > Exactly these tests. Running it with ant junit-all, I have most of the > time 98 failures (out of 6641 tests). On the server with CruiseControl, > I have more failures but I guess that this is only because it is a bit > more slow. This is made with the latest js tests from CVS head. > > When I run ant jsdriver-run this fails as well, but no idea how much > failures occur then. > > If I compile and run the tests in testsrc/org/mozilla/javascript/tests > (which are not included in any Ant target) from my IDE, I have 14 errors > (12 of them in DoctestsTest). > > Cheers, > Marc. _______________________________________________ dev-tech-js-engine-rhino mailing list [email protected] https://lists.mozilla.org/listinfo/dev-tech-js-engine-rhino
