On Wed 23 Jul 2008 at 04:33PM, Brad Hall wrote:
> Hi,
>
> Link to CR: http://cr.opensolaris.org/~bhall/bug-2627/
> Link to BUG: http://defect.opensolaris.org/bz/show_bug.cgi?id=2627
>
> Basically, we want to be able to run the test suite against a generated
> baseline to see what tests differ since recent changes have been made.
Brad, I'm going to be looking at this more carefully, but here are
my first reactions. Overall, this functionality is great and is
going to be a huge win. I really appreciate you doing it.
So will you be checking in a baseline? [Hopefully? :)]
I was wondering if perhaps the baseline should be in some sort of
text-tool-processable format? (I was thinking one line per test). I
was thinking that people might want to be able to review changes which
have been made to it by using 'diff'. I know that may be more of a pain
than a 'pkl'.
> $ pfexec python src/setup.py test -v
> running test
> logging to /tmp/tmpoY2XSZ.pkg-test.log
> test_action_errors (api.t_action.TestActions) ... pass
> test_action_parser (api.t_action.TestActions) ... pass
> test_action_tostr (api.t_action.TestActions) ... pass
> testcatalogfmris1 (api.t_catalog.TestCatalog) ... pass
> testcatalogfmris2 (api.t_catalog.TestCatalog) ... pass
> testcatalogfmris3 (api.t_catalog.TestCatalog) ... pass
> testcatalogfmris4 (api.t_catalog.TestCatalog) ... pass
For human consumption, I'm wondering if it would be
helpful just to show the complete path to the test in
order:
api.t_catalog.TestCatalog.testcatalogfmris2 pass
I also have a preference for eliminating "..." sequences, as
I personally find them more distracting than whitespace. Maybe
it's just me.
Another option would be to do more of a tabular form:
api.t_action.TestActions:
test_action_errors pass
test_action_parser pass
test_action_tostr pass
api.t_catalog.TestCatalog:
testcatalogfmris1 pass
...
This would let you set column widths, etc.
In pkg5unittest.startTest(), make sure to flush the stream... that way
if the test hangs, we know what test we're running. There
may be other places where a flush is also useful, but I'll leave
it to you to find them.
> ... a failing test that normally passes and you just broke ...
>
> testrolllogfiles (api.t_catalog.TestUpdateLog) ... FAIL
So as we talked about yesterday-- there are four cases. Can
we get example output for each?
Result Baseline Output
Pass Pass ??
Pass Fail ??
Fail Pass ??
Fail Fail ??
> FAILED (failures=1)
Would it be possible to update this "(failures=1)" output to reflect
the four cases listed above?
-dp
--
Daniel Price - Solaris Kernel Engineering - [EMAIL PROTECTED] - blogs.sun.com/dp
_______________________________________________
pkg-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
