Brad, Sorry for the late comment as I was on vacation last week.
The *_baseline.txt files are being written to the src/tests directory without any platform information. However, the tests may have different pass/fail behaviors when run on different platforms. There is a proto/build_* directory (one for each platform) that could be used to store the baseline files. However, I'm not sure how one would be able to checkin a baseline file then.
Tom Brad Hall wrote:
Hi, Link to CR: http://cr.opensolaris.org/~bhall/bug-2627/ Link to BUG: http://defect.opensolaris.org/bz/show_bug.cgi?id=2627 Basically, we want to be able to run the test suite against a generated baseline to see what tests differ since recent changes have been made. The baseline result set is a file that could either be checked in, or generated after you clone a repo (or both). If there is no baseline result set, one will be generated; otherwise you can re-generate the result set by giving the -g option to 'python setup.py test', or using the make test-generate target. Once the baseline is generated, all tests results will be compared against the baseline and a summary will be printed after each suite (cli, api) has been run. Two new files were added: - src/tests/pkg5unittest.py, which inherits from unittest and overrides the the testrunner and testresult classes. - src/tests/baseline.py, which deals with keeping track of the baseline result files and checking the current test results against those files. I've added a -v option to print the test output in verbose mode, and a -m option to output the results in something more machine consumable. The machinery for passing options through setup.py -> {api,cli}-complete.py is a little ugly so I welcome suggestions on that. In addition, the make test-verbose target now works. There are also some whitespace fixes in the webrev; sorry about that, it clutters the diffs but they pop up in bright red in my editor so I like to have them gone :) Anyhow, the new output looks like this (and if -v isn't specified, it will still print the dots as it used to): $ pfexec python src/setup.py test -v running test logging to /tmp/tmpoY2XSZ.pkg-test.log test_action_errors (api.t_action.TestActions) ... pass test_action_parser (api.t_action.TestActions) ... pass test_action_tostr (api.t_action.TestActions) ... pass testcatalogfmris1 (api.t_catalog.TestCatalog) ... pass testcatalogfmris2 (api.t_catalog.TestCatalog) ... pass testcatalogfmris3 (api.t_catalog.TestCatalog) ... pass testcatalogfmris4 (api.t_catalog.TestCatalog) ... pass ... snip (more passing tests) ... ... a failing test that normally passes and you just broke ... testrolllogfiles (api.t_catalog.TestUpdateLog) ... FAIL ... snip (more passing tests) ... testversionsuccessor7 (api.t_version.TestVersion) ... pass testversionsuccessor8 (api.t_version.TestVersion) ... pass testversionsuccessor9 (api.t_version.TestVersion) ... pass ====================================================================== FAIL: testrolllogfiles (api.t_catalog.TestUpdateLog) ------------------------------------------- in testrolllogfiles self.assert_(len(dl) == 2) AssertionError ---------------------------------------------------------------------- Ran 131 tests in 1.604s FAILED (failures=1) ====================================================================== BASELINE MISMATCH: The following results didn't match the baseline. ---------------------------------------------------------------------- testrolllogfiles (api.t_catalog.TestUpdateLog): fail ---------------------------------------------------------------------- Thanks, Brad _______________________________________________ pkg-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
begin:vcard fn:Tom Mueller n:Mueller;Tom org:Sun Microsystems, Inc.;Update Center Software adr:;;21915 Hillandale Dr;Elkhorn;NE;68022;USA email;internet:[EMAIL PROTECTED] title:Senior Staff Engineer tel;work:877-250-4011 tel;fax:877-250-4011 tel;home:402-916-9943 x-mozilla-html:TRUE version:2.1 end:vcard
_______________________________________________ pkg-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
