On 2015-11-02, Kornel Benko wrote:
> Am 2. November 2015 um 08:36:05, schrieb Guenter Milde <mi...@users.sf.net>
>> On 2015-11-02, Scott Kostyshak wrote:
>> > On Sun, Nov 01, 2015 at 10:36:17PM +0100, Kornel Benko wrote:
>> >> Am 1. November 2015 um 21:27:11, schrieb Guenter Milde
>> >> > Could we introduce a new status "suspended" - meaning just skip the
>> >> > test until a known bug is solved as we know the result is
>> >> > insignificant until then?
>> >> We already have such (specified in file "ignoredTests"). But as this
>> >> tests are never executed, nobody cares for them anymore.
>> >> The tests here are such, that we know, we never resolve them.
But even here, there are two cases:
a) cases that fail for a good reason and should always fail
(e.g. export document with non-TeX fonts using 8-bit LaTeX)
b) cases that should not fail but do (for reasons we cannot change).
While a) should be "permanently inverted" (to give a warning if a change
makes the export pass but we expect the result to be faulty), b) is
correctly placed in "ignored" -- the test case does not help detect LyX
buts in any way.
>> Suggestion:
>> Specified in file "suspendedTests") with the reason for suspending
>> (bug report, commit that turned hidden problems into export failure, ...)
>> These tests are normally skipped, but they are not forgotten.
>> The tests here are such, that we know, we can resolve them but their
>> failure is a minor issue that can be postponed (comparable to enhancement
>> requests in Trac).
>> Candidates for "suspended" tests are
>> * "fragile" documents (many packages, ERT, heavy preamble, ...),
>> * "fragile" export routes (XeTeX/LuaTeX with TeX fonts),
>> * non-default export routes
>> and especially combinations of these.
> OK, here is my suggestion
> 1.) We add the appropriate tests into revertedTests
Why? This would undo one important benefit of "suspended" tests:
>> Suspending instead of reverting also frees us from the need to reassess
>> them if a change in the "real life documents" or a fix makes them pass
>> again. Instead, they could/should be revised at a time we have fixed
>> major known problems and no release pressure...
I'd only add the test to "revertedTests", too, if it is established that
the correct result for this test is a failure.
> The content in suspendedTests may be (in our case) e.g.
> pdf4_texF
> 1.a)
> If a test is to be inverted, we check suspendedTest,
> and if there, we ignore the testcase.
Here, I'd change the priority:
* with a normal run, all suspended tests are ignored,
* with "run suspended tests" or "include suspended tests",
suspended test are run, taking into consideration their
"inversion state".
> or
> 1.b)
> Such a test may get a label "suspended" instead of
> "export", so that 'ctest -L export' will be clean, but
> we also have the possibility to use 'ctest -L
> suspended'.
> This does neither effect non-inverted nor ignored.
My suggestion would affect non-inverted tests (the ones with the label
"suspended").
This means for failing test you will have 3 options:
1. if failing is the expected outcome: invert
2. if failing for permanent reasons that are none of our business: ignore
3. if failing for minor reasons or a known bug: suspend
Motivation:
Some bugs only lead to failures depending on document content. We do
"road testing" with real life documents (manuals, examples). Work on the
documents could easily change the export status of tests without any
progress on fixing the underlying problem! (Examples: adding/removing a
character not available with non-default export route, adding non-ASCII
character to the PDF Header Information with "inputenc=ascii",
Writing my name as G\"unter in the PDF Header Information before solving
#9830, ...)
However, editing manuals or examples should not require immediate
revision of test cases using this document if the basic problem is not
solved.
OTOH, if bugs that triggered suspension of some test cases are solved,
it is time to revisit the set of suspended tests, reactivate the ones
that pass and re-label the ones that still fail.
> I prefer 1.b.
Whatever works best in praxi.
Günter
