Martin Rubey wrote:
> Waldek Hebisch <[email protected]> writes:
>
> > Attached is a patch that adds support for expected failures to
> > testing framework. IMHO currently testing framework has
> > several problem but this was probably the worst one.
>
> I'll try not to take this personal. I always thought that my package
> was quite an improvement about *no* package
>
Sure. Maybe we should say more about achivements (and current
framework is one of them). But certainly we need to discuss
problems, without that it is hard to improve things...
> > Namely, without such support adding tests for known bugs was
> > inpractical: known failures were reported exactly the same as new
> > ones, so they would make checking for regressions significantly
> > harder.
>
> Great!
>
> > The patch adds buch of xf* routines parallel to existing
> > ones -- the xf* routines expect failure, so they increment
> > separate counters for expected failures.
>
> please rename the internal routines
>
> *PasInc
>
> the
>
> *PassInc
>
Will do.
> > I also added a mulitline summary at the end, which hopefully will be
> > clearer than existing one.
>
> I assume that what you add is a grand total over all testcases - or is
> it a grand total over all testsuites? Please make this clearer in the
> output.
>
>
> =============================================================================
> testsuite | testcases: failed (total) | tests: failed (total)
> bugs2009 0 (19) 0 (69)
> unexpected failures: 0
> expected failures: 5
> unexpected passes: 0
> total tests: 69
>
I want total of all tests: if one test of a testcase is expected
failure and say three other pass classifying the whole testcase
as expected failure would be (IMHO) misleading. So I want
summary which ignores testsuite/testcase boundaries. Note that
my plan is to have 'make check' rule in Makefiles which gives a
single compact result summarizing all tests. Since we run each
test file separately the first step is to get summaries for
files. Later we can add some (probably awk) script to collect
data from files and produce final numbers. My main concern
is that numbers are easy to parse for programs, I hope that
human will have little need to look at them. We can add
a line and extra header to make clear that the new summary
is separate from the old one:
...
==========================================================
File summary.
unexpected failures: 0
expected failures: 5
unexpected passes: 0
total tests: 69
--
Waldek Hebisch
[email protected]
--
You received this message because you are subscribed to the Google Groups
"FriCAS - computer algebra system" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/fricas-devel?hl=en.
