On 2016-09-13, Kornel Benko wrote: >> >> * Do you still need the "suspendeTests"? What for?
>> > Yes, we need them. This tests will not be executed with the call 'ctest >> > -L export'. >> suspendeTests were introduced for the "less urgent" test cases like files in >> the attic and export with Xe/LuaTeX and 8-bit TeX fonts when there were >100 >> test errors. > My understanding is that they are suspended because of difficult to > solve, therefore we do not try to tests them every time. Both. Looking in suspendedTests, I see two patterns: XeTeX/LuaTeX with 8-bit fonts and attic-docs. I suggest the following comments: # XeTeX/LuaTeX with TeX fonts is rarely used and not properly # supported by many packages. -> Failures are mostly due to LaTeX problems. .*(|de|es|fr|he|hu|id|ko|ru|uk|zh_CN)/.*(dvi3|pdf4|pdf5)_texF # Files in the attic are kept for reference and conversion tests. # Failures are usually due to new package versions and "wontfix". .*/attic/(eu|id|it|pl|sk)_.* >> In the meantime, we know the particular problem for most of the suspended >> tests. >> Some were fixed. Others require inversion (we know the problem is a >> wontfix or a bug on trac). >> How many of the suspendedTests are still failing? > 73% tests passed, 21 tests failed out of 79 (this means 58 failing) This means 21 "fail to fail", does it? > So, 21 of them could be removed from suspiciousTests. Fine. Can you do this or send me a list of these tests? >> Do we really still need the suspension or could we lift it? >> If we want to keep it, it would be important to have a way to clearly define >> tests that are "suspended" AND "inverted". > ATM, we have it. All suspended are inverted. I just read again in Development.lyx: * only inverted tests are checked for suspension * they are inverted but get a different label Maybe this would become clearer for the "unintiated", if the name would include "inverted", e.g. - Test #375: SUSPENDED.ERT_export/export/utf8-plain-with-tex-fonts_dvi3_texF + Test #375: SUSPENDED.INVERTED.ERT_export/export/utf8-plain-with-tex-fonts_dvi3_texF This way, I can see that the specific pattern is in invertedTests under the sublabel ERT. OTOH, it may be confusing if the name has "inverted" but there is no inverted label. One option would be to give the "suspended" tag in addition to "inverted" and deselect suspended tests with -LE suspended, e.g ctest -N -L manuals -LE suspended Günter
