Am Montag, 23. November 2015 um 16:38:20, schrieb Guenter Milde <mi...@users.sf.net>
...
> >> Up to here, I hope we can agree.
>
> > Partly. I made the comments where I think it should be more precise.
>
> Which of the follwing is true?
>
> a) whether a combination of document, output format, and scripted changes
> is tested or ignored is decided at configuration time,
a.)
> b) this cannot be changed (add or skip certain combinations)
> when running the test suite,
b.)
But you can select anything from the created tests.
> c) it is not possible to list "ignored tests"?
>
c.)
Not possible. Totally unknown to ctest.
So a), b) and c) are true.
...
>> suspended is subset of reverted (and therefore inverted).
>
> > nonstandard is subset of all exports (but ignored)
>
> I want suspended tests to be *ignored*, because running
> them does not tell us anything about regressions.
> It doesn't make sense to expect a defined result here.
>
> IMO, suspended test should become a subset of "ignored" - not run normally
> but only on special request.
>
Now it's time to make something clear.
Possible tests which make no sense are grouped into a set _I_ named 'ignored'.
Test which always fail (independent of system they are running on) are grouped
in a set _I_ named 'reverted'.
Nonstandard seems to be uncontroversial.
Reverted tests which are expected to fail for a long time are grouped into a
set _I_ named 'suspended'
That way these tests could be run less often.
This all is probably not good naming.
Maybe you could propose different names?
> > Nonstandard test should not be part of reverted.
> > (and cannot be, even if regex from revertedTests signals otherwise)
> > It doesn't make sense to expect defined result here.
>
> True.
...
> > So after update of, say luatex, we could use 'ctest -R "dvi3|pdf5"' to
> > check if any relevant tests changed its compilation result.
>
> Does this mean, with 'ctest -R texF' you can also test *ignored* tests?
>
> How does this relate to "Everything ignored cannot be tested."?
Of course not. Ctest does not know ignored possible tests.
...
> > That is not possible. All messages go to
> > Testing/Temporary/LastTest.log. One has to extract them from there
> > after the test(s) is/are run.
>
> OK. But it should be possible to do this extraction in a post-processing
> script or a wrapper.
Sure.
...
> > Yes, suspended/reverted tests are waiting for success.
>
> Not all reverted tests are waiting for success. There are combinations of
> document + output format where we must ensure that export fails (because
> we know for sure it cannot work and otherwise this would be a hidden
> failure).
True. With 'waiting' I meant we want to know when something changes.
> > You didn't mention tests which works on some systems but not on others.
>
> I don't want any output about them after a "normal" test run.
So do I.
> Günter
Sorry for not being good enough in making good and understandable names.
Kornel
signature.asc
Description: This is a digitally signed message part.
