On Wed, Dec 22, 2010 at 3:13 PM, Marius Gedminas mar...@gedmin.as wrote:
It is reported near the beginning of the output (highlighted in red, to
stand out), mentioned in the summary at the end to ensure you don't
miss it, and the test runner exits with a non-zero status code (I hope;
if it doesn't, that's a bug).
Yes, the status code is correctly set - I wasn't complaining about
that, the runner works fine in our CI system - but I usually don't
pipe || while running tests at the console.
Also, there's a bit of impedence between 0 failures and 0 errors
and the exit status 1. I mean, the first time it happened i thought
WTF?. I mean: it's very easy to associate red with something bad
occurred, I like that, and I'd like that to be extended.
Maybe something like
Ran A tests with B failures, C errors and D other problems
would satisfy us all.
That's an interesting perspective.
Note that even when there are failures, the number of tests and the
number of seconds are highlighted in green. (The colours there are
mainly to make the numbers stand out so they're easier to notice in the
output.)
Yes, you're right, it's just the number of failures that turns red, if any.
What I'd probably like is to have failed imports and syntax errors to
be counted as errors, while files defining no tests just as warnings.
probably not what it want.
What's it?
it = I , sorry, my typo.
- I need to dig to the top in order to get the traceback; other
frameworks, like twisted's own trial, print all the tracebacks at the
bottom of the test run for easy debugging;
Having used other frameworks, I appreciate zope.testrunner's eagerness
to show me the traceback at once, so I can start examining an error
without having to stare at an F in the middle of a sea of dots and
wonder what it might be, while waiting 20 minutes for the rest of the
test suite to finish running.
I realize zope.testrunner layering system is designed to run unit,
integration and maybe acceptance tests, which can be pretty consuming.
I usually run mostly unit tests, taking less than 2 seconds.
By the way you're describing nose behaviour, I think; twisted trial is
better at this; normally it just outputs the errors at the end, but if
the -e switch is passed, it prints the tracebacks both ASAP *and* at
the end. I guess a similar approach, using a switch, could be employed
in zope.testrunner.
Then again I agree that having to scroll back to the first traceback is
a necessity that's a bit bothersome. I don't think printing the
tracebacks at the end of the run would help, in case there were multiple
tracebacks -- you'd want the first one anyway; the others likely being
caused by it. Also, tracebacks tend to be long, requiring me to scroll
anyway.
If I'm running unit tests there's no certain connection between
tracebacks, and most probably there's no definite first. Again,
probably a configurable switch would make everybody happy :-)
- while an import issue is a serious fact - meaning the test can't be
run, and should be reported, a test module which does not define any
test could just issue a warning - it could just be a placeholder test,
or a not-yet-finished test case, and should not be a blocking issue.
An import error could be a placeholder as well. What makes a module
with no tests different? If you added a module, it's reasonable to
assume you added it for a reason, and what other reason could there be
other than for it to have tests?
An import error is surely an error; if it can't be imported it can't
work in any way. The file might define 100 tests, and those are not
run.
A syntax error is surely an error as well; if written with the correct
syntax there might be tests inside, and they won't get executed.
A file defining no tests may be an error, but you can never be sure;
hence I think you should tell the user but you shouldn't fail by
default, IMHO.
NB:
if any of my proposal is accepted, I'm willing to contribute a patch,
I just don't want to start coding for something that might later just
be trashed.
--
Alan Franzoni
--
contact me at pub...@[mysurname].eu
___
Zope-Dev maillist - Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )
