Hi,
Link to CR: http://cr.opensolaris.org/~bhall/bug-2627/
Link to BUG: http://defect.opensolaris.org/bz/show_bug.cgi?id=2627
Basically, we want to be able to run the test suite against a generated
baseline to see what tests differ since recent changes have been made.
The baseline result set is a file that could either be checked in, or
generated after you clone a repo (or both). If there is no baseline result
set, one will be generated; otherwise you can re-generate the result set by
giving the -g option to 'python setup.py test', or using the make
test-generate target. Once the baseline is generated, all tests results will
be compared against the baseline and a summary will be printed after each
suite (cli, api) has been run.
Two new files were added:
- src/tests/pkg5unittest.py, which inherits from unittest and overrides the
the testrunner and testresult classes.
- src/tests/baseline.py, which deals with keeping track of the baseline result
files and checking the current test results against those files.
I've added a -v option to print the test output in verbose mode, and a -m
option to output the results in something more machine consumable. The
machinery for passing options through setup.py -> {api,cli}-complete.py is a
little ugly so I welcome suggestions on that. In addition, the make
test-verbose target now works.
There are also some whitespace fixes in the webrev; sorry about that, it
clutters the diffs but they pop up in bright red in my editor so I like to have
them gone :)
Anyhow, the new output looks like this (and if -v isn't specified, it will
still print the dots as it used to):
$ pfexec python src/setup.py test -v
running test
logging to /tmp/tmpoY2XSZ.pkg-test.log
test_action_errors (api.t_action.TestActions) ... pass
test_action_parser (api.t_action.TestActions) ... pass
test_action_tostr (api.t_action.TestActions) ... pass
testcatalogfmris1 (api.t_catalog.TestCatalog) ... pass
testcatalogfmris2 (api.t_catalog.TestCatalog) ... pass
testcatalogfmris3 (api.t_catalog.TestCatalog) ... pass
testcatalogfmris4 (api.t_catalog.TestCatalog) ... pass
... snip (more passing tests) ...
... a failing test that normally passes and you just broke ...
testrolllogfiles (api.t_catalog.TestUpdateLog) ... FAIL
... snip (more passing tests) ...
testversionsuccessor7 (api.t_version.TestVersion) ... pass
testversionsuccessor8 (api.t_version.TestVersion) ... pass
testversionsuccessor9 (api.t_version.TestVersion) ... pass
======================================================================
FAIL: testrolllogfiles (api.t_catalog.TestUpdateLog)
-------------------------------------------
in testrolllogfiles
self.assert_(len(dl) == 2)
AssertionError
----------------------------------------------------------------------
Ran 131 tests in 1.604s
FAILED (failures=1)
======================================================================
BASELINE MISMATCH: The following results didn't match the baseline.
----------------------------------------------------------------------
testrolllogfiles (api.t_catalog.TestUpdateLog): fail
----------------------------------------------------------------------
Thanks,
Brad
_______________________________________________
pkg-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
