On 2015-11-05, Kornel Benko wrote:
> Am Donnerstag, 5. November 2015 um 12:43:12, schrieb Guenter Milde
> <mi...@users.sf.net>
...
>> > (from here on it matches revertedTests)
>> iv. Prepend INVERTED... to testname. Revert test verification.
>> > Now calling ctest with '-L export' selects tests with label 'export',
>> > thus skipping suspended tests. The same is valid for '-L reverted'.
>> > The other possibilities, e.g. using '-R' parameter, does only check for
>> > testnames. But the suspended tests must also have a name, so we cannot
>> > skip them automatically.
>> Then, with a regexp not starting with SUSPENDED, -R would not include
>> suspended tests, right?
> No, the tests do not have any special naming. As they are a subset of
> revertedTests their name starts with 'INVERTED'.
> Of course, it could be arranged.
I was still explaining how my concept of "suspension" could work.
Arranging "suspended" tests to start with SUSPENDED would ensure the -R
parameter normally skips them as well...
> But one can use e.g.
> #ctest -R someregex -LE suspended
> to get the desired.
OK
...
>> However, this is basically what you currently do with "inverted tests"
>> there is basically just a different naming scheme.
>> The "measure for deviation from a clean state" would be the number of
>> "suspended" tests with my proposal and "inverted" tests in yours.
>> The only difference are export tests we want to return an error.
>> How are they managed in your scheme?
> I am not sure I understand.
> ATM export tests are expected to pass. All other tests are coverer be
> revertedTests. We (Scott and me) would add them to revertedTests.
I see that there is one more ambiguity:
* I divide LyX testing into
unit tests:
testing specific functions and classes
currently missing
functional tests:
testing processing (import/export) against a known reference
currently implemented as tex2lyx tests
export tests:
testing the export status of sample documents (most of them taken from
documentation and examples, i.e. with a different main purpose).
* In contrast, the label "export" inside the "automatic export test framework"
applies only to a subset of test cases.
The others are "inverted/reverted" or "ignored".
-> "export test" can be either the complete set of the "autotests" or just
the subset where we currently expect the export to pass.
I understand that this naming is historically motivated und "natural" for
the ones that work with the automatic tests from start. However, it makes
understanding what is going on complicated for newbies.
The next problem is, that we must be very careful when speaking about a
test that fails:
-> inverted tests pass, when the export fails (i.e. returns an error)
So I'll try to re-state my question:
I would like to have a simple number measuring the "cleanness" of LyX
code and shipped documents. For this, I propose to use the number
of test cases that do not give the "correct" result.
With the current implementation, how can I distinguish:
1. correct results:
a) tests where the export should succeed and really does
b) tests where the export fails for "good resons" (i.e. correct error
reporting for "impossible" actions)
2. wrong results:
a) tests where the export should succeed but fails
b) tests where the export does not report an error but we know it should
(because the exported document is corrupt).
As far as I understand, "revertedTests" reflect the currently expected
result, not the desired result. The number of inverted tests includes
1b) and 2a).
Conclusion:
The result of running the tests is good at telling about regressions or
"surfaced bugs" (i.e. old problems that lead to export errors after changing
either code or sample documents).
However, neither the number of "failing tests", nor the number of
"inverted" tests can be used to tell how "good" LyX currently is.
Günter
