Changes look good to me. One question I had is what happens if you give more than one option, like -v -n for example? Or is that just a "don't do that" situation?
From what I can tell, even if you don't run in verbose mode, you still get the baseline output down at the bottom, right? Btw, I really like that you're displaying the test names as they're running. One suggestion is that in the error case, could you display the exception that was thrown after ERROR or on the next line (and maybe the same thing for fail?). I'm not suggesting the full traceback that would be shown down in the lower section, maybe just the exception or the line which caused the unexpected value/assertion failure. Would that be possible to add? These improvements look really good. Thanks, Brock Brad Hall wrote: > Hi, > > Link to CR: http://cr.opensolaris.org/~bhall/bug-2627/ > Link to BUG: http://defect.opensolaris.org/bz/show_bug.cgi?id=2627 > > Basically, we want to be able to run the test suite against a generated > baseline to see what tests differ since recent changes have been made. > > The baseline result set is a file that could either be checked in, or > generated after you clone a repo (or both). If there is no baseline result > set, one will be generated; otherwise you can re-generate the result set by > giving the -g option to 'python setup.py test', or using the make > test-generate target. Once the baseline is generated, all tests results will > be compared against the baseline and a summary will be printed after each > suite (cli, api) has been run. > > Two new files were added: > - src/tests/pkg5unittest.py, which inherits from unittest and overrides the > the testrunner and testresult classes. > - src/tests/baseline.py, which deals with keeping track of the baseline result > files and checking the current test results against those files. > > I've added a -v option to print the test output in verbose mode, and a -m > option to output the results in something more machine consumable. The > machinery for passing options through setup.py -> {api,cli}-complete.py is a > little ugly so I welcome suggestions on that. In addition, the make > test-verbose target now works. > > There are also some whitespace fixes in the webrev; sorry about that, it > clutters the diffs but they pop up in bright red in my editor so I like to > have > them gone :) > > Anyhow, the new output looks like this (and if -v isn't specified, it will > still print the dots as it used to): > > $ pfexec python src/setup.py test -v > running test > logging to /tmp/tmpoY2XSZ.pkg-test.log > test_action_errors (api.t_action.TestActions) ... pass > test_action_parser (api.t_action.TestActions) ... pass > test_action_tostr (api.t_action.TestActions) ... pass > testcatalogfmris1 (api.t_catalog.TestCatalog) ... pass > testcatalogfmris2 (api.t_catalog.TestCatalog) ... pass > testcatalogfmris3 (api.t_catalog.TestCatalog) ... pass > testcatalogfmris4 (api.t_catalog.TestCatalog) ... pass > > ... snip (more passing tests) ... > > ... a failing test that normally passes and you just broke ... > > testrolllogfiles (api.t_catalog.TestUpdateLog) ... FAIL > > ... snip (more passing tests) ... > > testversionsuccessor7 (api.t_version.TestVersion) ... pass > testversionsuccessor8 (api.t_version.TestVersion) ... pass > testversionsuccessor9 (api.t_version.TestVersion) ... pass > > ====================================================================== > FAIL: testrolllogfiles (api.t_catalog.TestUpdateLog) > ------------------------------------------- > in testrolllogfiles > self.assert_(len(dl) == 2) > AssertionError > > ---------------------------------------------------------------------- > Ran 131 tests in 1.604s > > FAILED (failures=1) > > ====================================================================== > BASELINE MISMATCH: The following results didn't match the baseline. > ---------------------------------------------------------------------- > testrolllogfiles (api.t_catalog.TestUpdateLog): fail > ---------------------------------------------------------------------- > > Thanks, > Brad > _______________________________________________ > pkg-discuss mailing list > [email protected] > http://mail.opensolaris.org/mailman/listinfo/pkg-discuss > _______________________________________________ pkg-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
