Re: suspended ctests

[Scott Kostyshak]

On Wed, Sep 21, 2016 at 01:02:07PM +, Guenter Milde wrote:

> > It was not me alone who built the test system. Still, I don’t think it
> > is complicated.
> 
> For me, it is still far too complicated. It took half a year to
> understand things like
> 
>   Edit "invertedTests" to fix a "suspended" test that fails
>   because the export of the test document works, ...
> 
> or the table in Development.lyx "4.3.1.10 Export test filtering".
> (See the alternative table in the note below for the
> simplification that could be achieved.)

I can see what you mean. I think that the tests are still evolving. It's
possible that we will be able to smooth them out and make them more
accessible. I think it might have been necessary to have gone through
the (complicated?) stages that we have, in order to learn certain
lessons. But I think we'll eventually end up with as system that will be
easier to describe and understand by newcommers. Who knows, maybe
someday creating and running tests will even be fun! (I know, I'm
dreaming now...)

> Thanks for you patience,

And for yours!

Scott


signature.asc
Description: PGP signature

Re: suspended ctests

[Guenter Milde]

Dear Kornel,

thank you for the reply.

On 2016-09-20, Kornel Benko wrote:
> Am 19. September 2016 um 06:36:49, schrieb Guenter Milde 
>> On 2016-09-16, Kornel Benko wrote:

...

>> > This is my alias for export:
>> >ctest -j12 -timeout 300 -L export -E "xhtml|lyx16|lyx21|lyx22"

...

>> ... the alias might easily be changed to:

>>ctest -j12 -timeout 300 -L export -LE "texissues|attic"
>>  -E "xhtml|lyx16|lyx21|lyx22"

> And why remove attic? Do we not want to support lyx-file written with
> old lyx? If yes, then we should test IMHO.

The alias only excludes attic-files that are also excluded in the current
alias, because they have the label "suspended:attic" and no label
"export":

#> ctest -N -L attic
 Test project /usr/local/src/lyxtest
  Test #638: SUSPENDED.ATTIC_export/doc/attic/eu_UserGuide_pdf2
  Test #642: SUSPENDED.ATTIC_export/doc/attic/eu_UserGuide_pdf5_texF
  Test #643: SUSPENDED.ATTIC_export/doc/attic/eu_UserGuide_pdf5_systemF
  Test #652: SUSPENDED.ATTIC_export/doc/attic/id_UserGuide_dvi3_systemF
  Test #657: SUSPENDED.ATTIC_export/doc/attic/id_UserGuide_pdf4_systemF
  Test #659: SUSPENDED.ATTIC_export/doc/attic/id_UserGuide_pdf5_systemF
  Test #675: SUSPENDED.ATTIC_export/doc/attic/it_Customization_pdf5_systemF
  Test #684: SUSPENDED.ATTIC_export/doc/attic/it_UserGuide_dvi3_systemF
  Test #689: SUSPENDED.ATTIC_export/doc/attic/it_UserGuide_pdf4_systemF
  Test #720: SUSPENDED.ATTIC_export/doc/attic/sk_UserGuide_pdf4_texF


>> >>   +1 filters not only 58 but 80 "wontfix" inverted tests

> "wontfix" definitely belongs to suspendedTests or ignoredTests.

To be more accurate, these 80 tests include, "wontfix", "later" and
"upstream" issues.

* Moving patterns from invertedTests to ignoredTests is an option
  for real "wontfix" cases -> let's discuss this separately.

* "suspendedTests" was introduced to get a first priorisation of the
  >100 test failures after reporting missing characters as error via
  "orthogonal" regular expressions.

  Labeling all "later" and "upstream" issues as "suspended" would require
  duplicate entries in invertedTests and suspendedTests. (And also
  require to remove these entries from 2 places once the export works
  again...). 
  
  This is why I think the "suspendedTests" file is obsoleted by the
  sublabel mechanism.

>> >>   +4 makes the test system less complicated.

> It was not me alone who built the test system. Still, I don’t think it
> is complicated.

For me, it is still far too complicated. It took half a year to
understand things like

  Edit "invertedTests" to fix a "suspended" test that fails
  because the export of the test document works, ...

or the table in Development.lyx "4.3.1.10 Export test filtering".
(See the alternative table in the note below for the
simplification that could be achieved.)

Thanks for you patience,

Günter

Re: suspended ctests

[Kornel Benko]

Am Montag, 19. September 2016 um 06:36:49, schrieb Guenter Milde 

> On 2016-09-16, Kornel Benko wrote:
> 
> > [-- Type: text/plain, Encoding: quoted-printable --]
> 
> > Am Freitag, 16. September 2016 um 19:28:04, schrieb Guenter Milde 
> > 
> >> On 2016-09-13, Guenter Milde wrote:
> >> > On 2016-09-13, Kornel Benko wrote:
> 
> >> >>> >> * Do you still need the "suspendeTests"? What for?
> 
> >> >>> > Yes, we need them. This tests will not be executed with the call
> >> >>> > 'ctest
> >> >>> > -L export'.
> 
> >> ...
> 
> >> >>> In the meantime, we know the particular problem for most of the 
> >> >>> suspended
> >> >>> tests.
> >> >>> Some were fixed. Others require inversion (we know the problem is a
> >> >>> wontfix or a bug on trac).
> 
> >> Therefore, I'll reword the question:
> 
> >> Do you call `ctest -L export` regularely?
> 
> > This is my alias for export:
> > ctest -j12 -timeout 300 -L export -E "xhtml|lyx16|lyx21|lyx22"
> 
> >> Could you imagine calling `ctest -L export -LE "texissues|attic"` instead?
> 
> > Sure, but if you think texissues should not be tested, make them part
> > of suspended.
> 
> -1 this would mean duplicating all 83 lines of the "texissues" section in
>invertedTests into suspendedTests

Maybe. I counted 20 entries.

> -1 this would not allow me to get rid of the suspendedTests filter.   
> 
> OTOH, the alias might easily be changed to:
> 
>ctest -j12 -timeout 300 -L export -E "xhtml|lyx16|lyx21|lyx22" -LE 
> "texissues|attic"

And why remove attic? Do we not want to support lyx-file written with old lyx?
If yes, then we should test IMHO.

> >>   +1 filters not only 58 but 80 "wontfix" inverted tests

"wontfix" definitely belongs to suspendedTests or ignoredTests.

> >>   +4 makes the test system less complicated.
> 

It was not me alone who built the test system. Still, I don’t think it is 
complicated.

> Günter

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: suspended ctests

[Guenter Milde]

On 2016-09-16, Kornel Benko wrote:

> [-- Type: text/plain, Encoding: quoted-printable --]

> Am Freitag, 16. September 2016 um 19:28:04, schrieb Guenter Milde 
> 
>> On 2016-09-13, Guenter Milde wrote:
>> > On 2016-09-13, Kornel Benko wrote:

>> >>> >> * Do you still need the "suspendeTests"? What for?

>> >>> > Yes, we need them. This tests will not be executed with the call
>> >>> > 'ctest
>> >>> > -L export'.

>> ...

>> >>> In the meantime, we know the particular problem for most of the suspended
>> >>> tests.
>> >>> Some were fixed. Others require inversion (we know the problem is a
>> >>> wontfix or a bug on trac).

>> Therefore, I'll reword the question:

>> Do you call `ctest -L export` regularely?

> This is my alias for export:
>   ctest -j12 -timeout 300 -L export -E "xhtml|lyx16|lyx21|lyx22"

>> Could you imagine calling `ctest -L export -LE "texissues|attic"` instead?

> Sure, but if you think texissues should not be tested, make them part
> of suspended.

-1 this would mean duplicating all 83 lines of the "texissues" section in
   invertedTests into suspendedTests
   
-1 this would not allow me to get rid of the suspendedTests filter.   

OTOH, the alias might easily be changed to:

   ctest -j12 -timeout 300 -L export -E "xhtml|lyx16|lyx21|lyx22" -LE 
"texissues|attic"

>>   +1 filters not only 58 but 80 "wontfix" inverted tests

>>   +4 makes the test system less complicated.


Günter

Re: suspended ctests

[Kornel Benko]

Am Freitag, 16. September 2016 um 19:28:04, schrieb Guenter Milde 

> On 2016-09-13, Guenter Milde wrote:
> > On 2016-09-13, Kornel Benko wrote:
> 
> >>> >> * Do you still need the "suspendeTests"? What for?
> 
> >>> > Yes, we need them. This tests will not be executed with the call 'ctest
> >>> > -L export'.
> 
> ...
> 
> >>> In the meantime, we know the particular problem for most of the suspended
> >>> tests.
> >>> Some were fixed. Others require inversion (we know the problem is a
> >>> wontfix or a bug on trac).
> 
> Therefore, I'll reword the question:
> 
> Do you call `ctest -L export` regularely?

This is my alias for export:
ctest -j12 -timeout 300 -L export -E "xhtml|lyx16|lyx21|lyx22"

> Could you imagine calling `ctest -L export -LE "texissues|attic"` instead?

Sure, but if you think texissues should not be tested, make them part of 
suspended.

>   -1 more to type
>   
>   +1 filters not only 58 but 80 "wontfix" inverted tests
>   
>   +4 makes the test system less complicated.
> 
> Do you call `ctest -L inverted` regularely?

Sometimes, yes.

> Could you imagine calling `ctest -L inverted -LE "texissues|attic"` instead?
> 
> ...
> 
> >> My understanding is that they are suspended because of difficult to
> >> solve, therefore we do not try to tests them every time.
> 
> However, the "usual full run" (without -L) does test them.

Sure, I never claimed otherwise.

> How often is -L used without also -R to select specific tests via regexp?

Here? Almost always. That was one reason to invent/use labels.

> Günter

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: suspended ctests

[Guenter Milde]

On 2016-09-13, Guenter Milde wrote:
> On 2016-09-13, Kornel Benko wrote:

>>> >> * Do you still need the "suspendeTests"? What for?

>>> > Yes, we need them. This tests will not be executed with the call 'ctest
>>> > -L export'.

...

>>> In the meantime, we know the particular problem for most of the suspended
>>> tests.
>>> Some were fixed. Others require inversion (we know the problem is a
>>> wontfix or a bug on trac).

Therefore, I'll reword the question:

Do you call `ctest -L export` regularely?

Could you imagine calling `ctest -L export -LE "texissues|attic"` instead?

  -1 more to type
  
  +1 filters not only 58 but 80 "wontfix" inverted tests
  
  +4 makes the test system less complicated.

Do you call `ctest -L inverted` regularely?

Could you imagine calling `ctest -L inverted -LE "texissues|attic"` instead?

...

>> My understanding is that they are suspended because of difficult to
>> solve, therefore we do not try to tests them every time.

However, the "usual full run" (without -L) does test them.
How often is -L used without also -R to select specific tests via regexp?


Günter

suspended ctests

[Guenter Milde]

On 2016-09-13, Kornel Benko wrote:

>> >> * Do you still need the "suspendeTests"? What for?

>> > Yes, we need them. This tests will not be executed with the call 'ctest
>> > -L export'.

>> suspendeTests were introduced for the "less urgent" test cases like files in
>> the attic and export with Xe/LuaTeX and 8-bit TeX fonts when there were >100
>> test errors.

> My understanding is that they are suspended because of difficult to
> solve, therefore we do not try to tests them every time.

Both. 

Looking in suspendedTests, I see two patterns: XeTeX/LuaTeX with 8-bit fonts
and attic-docs.

I suggest the following comments:

  # XeTeX/LuaTeX with TeX fonts is rarely used and not properly 
  # supported by many packages. -> Failures are mostly due to LaTeX problems.
  .*(|de|es|fr|he|hu|id|ko|ru|uk|zh_CN)/.*(dvi3|pdf4|pdf5)_texF
  
  # Files in the attic are kept for reference and conversion tests.
  # Failures are usually due to new package versions and "wontfix".
  .*/attic/(eu|id|it|pl|sk)_.*


>> In the meantime, we know the particular problem for most of the suspended
>> tests.
>> Some were fixed. Others require inversion (we know the problem is a
>> wontfix or a bug on trac).

>> How many of the suspendedTests are still failing?

> 73% tests passed, 21 tests failed out of 79 (this means 58 failing)

This means 21 "fail to fail", does it?

> So, 21 of them could be removed from suspiciousTests.

Fine. Can you do this or send me a list of these tests?

>> Do we really still need the suspension or could we lift it?

>> If we want to keep it, it would be important to have a way to clearly define
>> tests that are "suspended" AND "inverted".

> ATM, we have it. All suspended are inverted.

I just read again in Development.lyx: 

* only inverted tests are checked for suspension
* they are inverted but get a different label

Maybe this would become clearer for the "unintiated", if the
name would include "inverted", e.g.

-  Test #375: SUSPENDED.ERT_export/export/utf8-plain-with-tex-fonts_dvi3_texF
+  Test #375: 
SUSPENDED.INVERTED.ERT_export/export/utf8-plain-with-tex-fonts_dvi3_texF

This way, I can see that the specific pattern is in
invertedTests under the sublabel ERT.

OTOH, it may be confusing if the name has "inverted" but there is no
inverted label.

One option would be to give the "suspended" tag in addition to "inverted"
and deselect suspended tests with  -LE suspended, e.g

  ctest -N -L manuals -LE suspended


Günter