Re: status update on the export tests

[Kornel Benko]

Am Donnerstag, 5. November 2015 um 16:56:32, schrieb Guenter Milde 

> On 2015-11-05, Kornel Benko wrote:
> > Am Donnerstag, 5. November 2015 um 12:43:12, schrieb Guenter Milde 
> > 
> 

...

> > No, the tests do not have any special naming. As they are a subset of
> > revertedTests their name starts with 'INVERTED'.
> > Of course, it could be arranged.
> 
> I was still explaining how my concept of "suspension" could work.

Ah, sorry.

> Arranging "suspended" tests to start with SUSPENDED would ensure the -R
> parameter normally skips them as well...
> 
> > But one can use e.g.
> > #ctest -R someregex -LE  suspended
> > to get the desired.
> 
> OK
> 
> ...
> 
> >> However, this is basically what you currently do with "inverted tests"
> >> there is basically just a different naming scheme.
> 
> >> The "measure for deviation from a clean state" would be the number of
> >> "suspended" tests with my proposal and "inverted" tests in yours.
> 
> >> The only difference are export tests we want to return an error.
> >> How are they managed in your scheme?
> 
> > I am not sure I understand.
> 
> > ATM export tests are expected to pass. All other tests are coverer be
> > revertedTests. We (Scott and me) would add them to revertedTests.
> 
> I see that there is one more ambiguity:
> 
> * I divide LyX testing into
> 
>   unit tests: 
> testing specific functions and classes 
>   
> currently missing

We will see, how it evolves.
  
>   functional tests:
> testing processing (import/export) against a known reference
>   
> currently implemented as tex2lyx tests

There is also a label "lyx2lyx", but I agree that are only rudimentary tests.

>   export tests: 
> testing the export status of sample documents (most of them taken from
> documentation and examples, i.e. with a different main purpose).
> 
> * In contrast, the label "export" inside the "automatic export test framework"
>   applies only to a subset of test cases. 
>   The others are "inverted/reverted" or "ignored".
>   
> -> "export test" can be either the complete set of the "autotests" or just
>the subset where we currently expect the export to pass.
> 
> I understand that this naming is historically motivated und "natural" for
> the ones that work with the automatic tests from start. However, it makes
> understanding what is going on complicated for newbies.
> 
>
> The next problem is, that we must be very careful when speaking about a
> test that fails:
> 
> -> inverted tests pass, when the export fails (i.e. returns an error)
> 
> 
> So I'll try to re-state my question:
> 
> I would like to have a simple number measuring the "cleanness" of LyX
> code and shipped documents. For this, I propose to use the number
> of test cases that do not give the "correct" result.
> 
> With the current implementation, how can I distinguish:
> 
> 1. correct results:
>
>a) tests where the export should succeed and really does
>
>b) tests where the export fails for "good resons" (i.e. correct error
>   reporting for "impossible" actions)

Yes.

> 2. wrong results:
>
>a) tests where the export should succeed but fails

There is no such test known. But of course, might exist. If it fails, then 
something in
our preparing lyx file for export is wrong (perl script 
useSystemFonts.pl+lyxStatus.pm)

>b) tests where the export does not report an error but we know it should
>   (because the exported document is corrupt).

This is the hard part. There are so many tests, that no one had time (or 
pleasure) to check them all visually.

> As far as I understand, "revertedTests" reflect the currently expected
> result, not the desired result.  The number of inverted tests includes
> 1b) and 2a).
> 
> Conclusion:
> 
> The result of running the tests is good at telling about regressions or
> "surfaced bugs" (i.e. old problems that lead to export errors after changing
> either code or sample documents).

Yes.

> However, neither the number of "failing tests", nor the number of
> "inverted" tests can be used to tell how "good" LyX currently is.
> 

Yes.

> Günter

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: status update on the export tests

[Guenter Milde]

On 2015-11-05, Kornel Benko wrote:
> Am Donnerstag, 5. November 2015 um 12:43:12, schrieb Guenter Milde 
> 

...

>> > (from here on it matches revertedTests)
>>   iv. Prepend INVERTED... to testname. Revert test verification.

>> > Now calling ctest with '-L export' selects tests with label 'export',
>> > thus skipping suspended tests. The same is valid for '-L reverted'.

>> > The other possibilities, e.g. using '-R' parameter, does only check for
>> > testnames. But the suspended tests must also have a name, so we cannot
>> > skip them automatically.

>> Then, with a regexp not starting with SUSPENDED, -R would not include
>> suspended tests, right?

> No, the tests do not have any special naming. As they are a subset of
> revertedTests their name starts with 'INVERTED'.
> Of course, it could be arranged.

I was still explaining how my concept of "suspension" could work.
Arranging "suspended" tests to start with SUSPENDED would ensure the -R
parameter normally skips them as well...

> But one can use e.g.
>   #ctest -R someregex -LE  suspended
> to get the desired.

OK

...

>> However, this is basically what you currently do with "inverted tests"
>> there is basically just a different naming scheme.

>> The "measure for deviation from a clean state" would be the number of
>> "suspended" tests with my proposal and "inverted" tests in yours.

>> The only difference are export tests we want to return an error.
>> How are they managed in your scheme?

> I am not sure I understand.

> ATM export tests are expected to pass. All other tests are coverer be
> revertedTests. We (Scott and me) would add them to revertedTests.

I see that there is one more ambiguity:

* I divide LyX testing into

  unit tests: 
testing specific functions and classes 
  
currently missing
  
  functional tests:
testing processing (import/export) against a known reference
  
currently implemented as tex2lyx tests

  export tests: 
testing the export status of sample documents (most of them taken from
documentation and examples, i.e. with a different main purpose).

* In contrast, the label "export" inside the "automatic export test framework"
  applies only to a subset of test cases. 
  The others are "inverted/reverted" or "ignored".
  
-> "export test" can be either the complete set of the "autotests" or just
   the subset where we currently expect the export to pass.

I understand that this naming is historically motivated und "natural" for
the ones that work with the automatic tests from start. However, it makes
understanding what is going on complicated for newbies.

   
The next problem is, that we must be very careful when speaking about a
test that fails:

-> inverted tests pass, when the export fails (i.e. returns an error)


So I'll try to re-state my question:

I would like to have a simple number measuring the "cleanness" of LyX
code and shipped documents. For this, I propose to use the number
of test cases that do not give the "correct" result.

With the current implementation, how can I distinguish:

1. correct results:
   
   a) tests where the export should succeed and really does
   
   b) tests where the export fails for "good resons" (i.e. correct error
  reporting for "impossible" actions)
   
2. wrong results:
   
   a) tests where the export should succeed but fails

   b) tests where the export does not report an error but we know it should
  (because the exported document is corrupt).

As far as I understand, "revertedTests" reflect the currently expected
result, not the desired result.  The number of inverted tests includes
1b) and 2a).

Conclusion:

The result of running the tests is good at telling about regressions or
"surfaced bugs" (i.e. old problems that lead to export errors after changing
either code or sample documents).

However, neither the number of "failing tests", nor the number of
"inverted" tests can be used to tell how "good" LyX currently is.


Günter

Re: status update on the export tests

[Kornel Benko]

Am Donnerstag, 5. November 2015 um 12:43:12, schrieb Guenter Milde 

> Dear Kornel,
> 
> thank you for the explanation. I come a bit nearer to a generic
> understanding now.
> 
> On 2015-11-05, Kornel Benko wrote:
> > Am Donnerstag, 5. November 2015 um 07:34:47, schrieb Guenter Milde 
> > 
> 
> > ...
> >> > It is suspended _only_ if you select testcases with the '-L' parameter.
> 
> >> OK. My idea was that suspended testcases are skipped by default.
> 
> > Let me sketch our use of '-L' parameter.
> 
> > We have a number of lyx-files under lib/doc, lib/examples etc directories.
> > There are export types like lyx16, xhtml, pdf, dvi etc
> > TeX or non TeX fonts.
> > For each combination we create a testname containing a hint to
> > a.) the relative path of the lyx file (without extension)
> > b.) export type (restricted to the specified output format(in 
> > lyx-file), if not 'default')
> > c.) TeX or non TeX font (texF, systemF)
> 
> (Is there a documentation of the testnames? If not, could this be added to
> Development.lyx, maybe or in a README for the test machinery? With examples?)
> 

I was too lazy. (But probably I try to omit documenting because of lack of 
enough English knowledge)

> Then: for my concept of "suspension":
> 
> > If the name
> 
> > i. matches regex in ignoreTests, the testcase is discarded. Handle to
> >next testname.
> 
>   ii. matches regex in suspendedTests: Prepend SUSPENDED to testname. If
>   there can be more than one label, the test gets the label "suspended".
> 
>   iii. Not matching revertedTests, the test gets the label "export".
>Handle next testname.
>   
> > (from here on it matches revertedTests)
>   iv. Prepend INVERTED... to testname. Revert test verification.
>   
> 
> > Now calling ctest with '-L export' selects tests with label 'export',
> > thus skipping suspended tests. The same is valid for '-L reverted'.
> 
> > The other possibilities, e.g. using '-R' parameter, does only check for
> > testnames. But the suspended tests must also have a name, so we cannot
> > skip them automatically.
> 
> Then, with a regexp not starting with SUSPENDED, -R would not include
> suspended tests, right?

No, the tests do not have any special naming. As they are a subset of 
revertedTests their name
starts with 'INVERTED'.
Of course, it could be arranged.
But one can use e.g.
#ctest -R someregex -LE  suspended
to get the desired.

> > This should answer also your remark in a previous post (
> > > So what do you propose for such tests (84% of export tests)?
> > > ctest -L export -N | wc => 3719
> > > ctest -L export -N |egrep '/(doc|examples)/'| wc => 3153
> > > 3153 / 3719 => 84.78%
> 
> >>I don't have any clue which tests are hidden behind these commands.
> 
> This means we have about 3000 export tests with "real life" documents in doc 
> and examples?
> 
> Basically, I would look at currently failing tests 
> (i.e. return value != expected return value) and
> 
> * invert/uninvert test based on the "correct" return value

I expect the suspended to pass (120 cases). If some of them fails (good news), 
it will be the minority
listed by ctest.
This is really easier to check.

> * suspend test failing for a known reason, so they are not run nor reported
>   by a "normal" run.
>   
> However, this is basically what you currently do with "inverted tests" there
> is basically just a different naming scheme.
> 
> The "measure for deviation from a clean state" would be the number of
> "suspended" tests with my proposal and "inverted" tests in yours.
> 
> The only difference are export tests we want to return an error.
> How are they managed in your scheme?

I am not sure I understand.
ATM export tests are expected to pass. All other tests are coverer be 
revertedTests.
We (Scott and me) would add them to revertedTests.

> Günter

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: status update on the export tests

[Guenter Milde]

Dear Kornel,

thank you for the explanation. I come a bit nearer to a generic
understanding now.

On 2015-11-05, Kornel Benko wrote:
> Am Donnerstag, 5. November 2015 um 07:34:47, schrieb Guenter Milde 
> 

> ...
>> > It is suspended _only_ if you select testcases with the '-L' parameter.

>> OK. My idea was that suspended testcases are skipped by default.

> Let me sketch our use of '-L' parameter.

> We have a number of lyx-files under lib/doc, lib/examples etc directories.
> There are export types like lyx16, xhtml, pdf, dvi etc
> TeX or non TeX fonts.
> For each combination we create a testname containing a hint to
>   a.) the relative path of the lyx file (without extension)
>   b.) export type (restricted to the specified output format(in 
> lyx-file), if not 'default')
>   c.) TeX or non TeX font (texF, systemF)

(Is there a documentation of the testnames? If not, could this be added to
Development.lyx, maybe or in a README for the test machinery? With examples?)


Then: for my concept of "suspension":

> If the name

> i. matches regex in ignoreTests, the testcase is discarded. Handle to
>next testname.

  ii. matches regex in suspendedTests: Prepend SUSPENDED to testname. If
  there can be more than one label, the test gets the label "suspended".

  iii. Not matching revertedTests, the test gets the label "export".
   Handle next testname.
  
> (from here on it matches revertedTests)
  iv. Prepend INVERTED... to testname. Revert test verification.
  

> Now calling ctest with '-L export' selects tests with label 'export',
> thus skipping suspended tests. The same is valid for '-L reverted'.

> The other possibilities, e.g. using '-R' parameter, does only check for
> testnames. But the suspended tests must also have a name, so we cannot
> skip them automatically.

Then, with a regexp not starting with SUSPENDED, -R would not include
suspended tests, right?


> This should answer also your remark in a previous post (
>   > So what do you propose for such tests (84% of export tests)?
>   > ctest -L export -N | wc => 3719
>   > ctest -L export -N |egrep '/(doc|examples)/'| wc => 3153
>   > 3153 / 3719 => 84.78%

>>  I don't have any clue which tests are hidden behind these commands.

This means we have about 3000 export tests with "real life" documents in doc 
and examples?

Basically, I would look at currently failing tests 
(i.e. return value != expected return value) and

* invert/uninvert test based on the "correct" return value

* suspend test failing for a known reason, so they are not run nor reported
  by a "normal" run.
  
However, this is basically what you currently do with "inverted tests" there
is basically just a different naming scheme.

The "measure for deviation from a clean state" would be the number of
"suspended" tests with my proposal and "inverted" tests in yours.

The only difference are export tests we want to return an error.
How are they managed in your scheme?

Günter

Re: status update on the export tests

[Kornel Benko]

Am Donnerstag, 5. November 2015 um 07:34:47, schrieb Guenter Milde 

...
> > It is suspended _only_ if you select testcases with the '-L' parameter.
> 
> OK. My idea was that suspended testcases are skipped by default.

Let me sketch our use of '-L' parameter.
This should answer also your remark in a previous post (
> So what do you propose for such tests (84% of export tests)?
> ctest -L export -N | wc => 3719
> ctest -L export -N |egrep '/(doc|examples)/'| wc => 3153
> 3153 / 3719 => 84.78%

I don't have any clue which tests are hidden behind these commands. The
general recipe is:
)

We have a number of lyx-files under lib/doc, lib/examples etc directories.
There are export types like lyx16, xhtml, pdf, dvi etc
TeX or non TeX fonts.
For each combination we create a testname containing a hint to
a.) the relative path of the lyx file (without extension)
b.) export type (restricted to the specified output format(in 
lyx-file), if not 'default')
c.) TeX or non TeX font (texF, systemF)
Then:
If the name
i. matches regex in ignoreTests, the testcase is discarded. 
Handle to next testname.
ii. Not matching revertedTests, the test gets the label 
"export". Handle next testname.
(from here on it matches revertedTests)
iii. Prepend INVERTED... to testname. Revert test verification.
iv.  Not matching suspendedTests, the test gets the label 
"reverted". Handle next testname.
(test matches suspendedTests)
v. test gets the label "suspended".

Now calling ctest with '-L export' selects tests with label 'export', thus 
skipping suspended tests.
The same is valid for '-L reverted'.

The other possibilities, e.g. using '-R' parameter, does only check for 
testnames. But the suspended tests
must also have a name, so we cannot skip them automatically.

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: status update on the export tests

[Guenter Milde]

On 2015-11-04, Kornel Benko wrote:
> Am Mittwoch, 4. November 2015 um 19:27:06, schrieb Guenter Milde 
> 
>> On 2015-11-04, Kornel Benko wrote:
>> > Am Mittwoch, 4. November 2015 um 15:36:45, schrieb Guenter Milde 
>> > 

>> ...

>> >> Not only, with "suspending" I also mean "The outcome is of no value
>> >> for finding new bugs or regressions until someone solves the known
>> >> bug ...".

>> >> However, we usually know what the outcome should be if the bug is
>> >> solved: if the expected outcome is "pass", this test should not be
>> >> inverted.

>> > Here we disagree. Matter of taste I suppose. For me the test fails
>> > _now_. We don't care now (because we know what's going on etc.).
>> > Therefore the test is to be inverted as to not catch unwanted
>> > attention.

>> I still do not understand the reasoning. It will not catch attention if
>> it is suspended, that is why it should be suspended.

> It is suspended _only_ if you select testcases with the '-L' parameter.

OK. My idea was that suspended testcases are skipped by default.

...

> Sure, but look into suspendedTests, there is only 1 regex (for now, I
> know) selecting from the inverted tests.

Maybe it is indeed a matter of taste. The advantage of your approach is,
that it is a small change to the exisisting/previous setup. As you are doing
the work, you shall have the final say. Just two thoughts:

* In your implementeation, the 1 regex (for now) rather points to 
  "fragile" tests, some of which are "suspended" (i.e. temporarily
  ignored).
  
  Could you name the file "fragileTests" reduce confusion?

* How can we distinguish a "good" inversion (the test should fail) from a
  "bad" inversion (the test should pass but currently does not).


Günter

Re: status update on the export tests

[Kornel Benko]

Am Mittwoch, 4. November 2015 um 19:27:06, schrieb Guenter Milde 

> On 2015-11-04, Kornel Benko wrote:
> > Am Mittwoch, 4. November 2015 um 15:36:45, schrieb Guenter Milde 
> > 
> 
> ...
> 
> >> Not only, with "suspending" I also mean "The outcome is of no value for
> >> finding new bugs or regressions until someone solves the known bug ...".
> 
> > OK.
> 
> >> However, we usually know what the outcome should be if the bug is
> >> solved: if the expected outcome is "pass", this test should not be
> >> inverted.
> 
> > Here we disagree. Matter of taste I suppose. For me the test fails
> > _now_. We don't care now (because we know what's going on etc.).
> > Therefore the test is to be inverted as to not catch unwanted
> > attention.
> 
> I still do not understand the reasoning. It will not catch attention if
> it is suspended, that is why it should be suspended.

It is suspended _only_ if you select testcases with the '-L' parameter.

> OTOH, if a test that should pass but does not is inverted & suspended (2
> actions), we need to uninvert and to unsuspend (again 2 actions) once the
> problem causing the failure is solved.

No, we need only uninvert. 'Suspend' has no effect or non-inverted tests.

> In contrast, if the "inversion status" matches the expected test result, we
> can run suspended tests from time to time and "unsuspend" the tests that now
> give the expected output.

No, only uninvert.

> Also, when looking at inverted tests, we do not know whether this is a 
> "good" inversion (the test should fail) or a "bad" inversion (the test
> should pass but currently does not).
> 

Sure, but look into suspendedTests, there is only 1 regex (for now, I know) 
selecting
from the inverted tests.

> Günter
> 

Kornel 

signature.asc
Description: This is a digitally signed message part.

Re: status update on the export tests

[Guenter Milde]

On 2015-11-04, Kornel Benko wrote:
> Am Mittwoch, 4. November 2015 um 15:36:45, schrieb Guenter Milde 
> 

...

>> Not only, with "suspending" I also mean "The outcome is of no value for
>> finding new bugs or regressions until someone solves the known bug ...".

> OK.

>> However, we usually know what the outcome should be if the bug is
>> solved: if the expected outcome is "pass", this test should not be
>> inverted.

> Here we disagree. Matter of taste I suppose. For me the test fails
> _now_. We don't care now (because we know what's going on etc.).
> Therefore the test is to be inverted as to not catch unwanted
> attention.

I still do not understand the reasoning. It will not catch attention if
it is suspended, that is why it should be suspended.

OTOH, if a test that should pass but does not is inverted & suspended (2
actions), we need to uninvert and to unsuspend (again 2 actions) once the
problem causing the failure is solved.

In contrast, if the "inversion status" matches the expected test result, we
can run suspended tests from time to time and "unsuspend" the tests that now
give the expected output.

Also, when looking at inverted tests, we do not know whether this is a 
"good" inversion (the test should fail) or a "bad" inversion (the test
should pass but currently does not).


Günter

Re: status update on the export tests

[Kornel Benko]

Am Mittwoch, 4. November 2015 um 15:36:45, schrieb Guenter Milde 

> On 2015-11-04, Kornel Benko wrote:
> 
> > [-- Type: text/plain, Encoding: 7bit --]
> 
> > Am Mittwoch, 4. November 2015 um 14:36:19, schrieb Vincent van Ravesteijn 
> > 
> >> > In my view, suspension is orthogonal to reversion:
> >> >
> >> > - normal:  we want the test to pass
> >> >   revert:  we want the test to fail
> 
> >> You mean: revert is that "it is known to fail", but we haven't fixed it 
> >> yet.
> 
> > Yes. We hope to do it eventually.
> 
> We have also tests that correctly fail:
> "it is known to fail", there is nothing to fix, and we must ensure that it
> continues to fail.
> The third point is, what prevents us from ignoring this test.
> 
> >> > - normal:  run the test
> >> >   suspend: skip the test temporarily
> >> >   ignore:  skip the test permanently
> >> >
> 
> >> Suspending means: "The outcome is noisy, so skip it until someone
> >> looks into it and makes the test better."
> 
> > Sort of, if we are careful enough about which test should go there.
> 
> Not only, with "suspending" I also mean "The outcome is of no value for
> finding new bugs or regressions until someone solves the known bug ...".

OK.

> However, we usually know what the outcome should be if the bug is solved: if
> the expected outcome is "pass", this test should not be inverted.
> 

Here we disagree. Matter of taste I suppose.
For me the test fails _now_. We don't care now (because we know what's going on 
etc.).
Therefore the test is to be inverted as to not catch unwanted attention.

> > For the actual committed cmake build, suspended test means:
> > a.) Test is one of export tests
> > b.) Test is failing, and therefore it is part of revertedTests
> > c.) In 'normal' use like 'ctest -L export' it is hidden
> > d.) Test gets the ctest-label 'suspended'
> > You can run all suspended tests with 'ctest -L suspended'.
> > Or, for a known testname (say xyzzy) you can use 'ctest -R xyzzy'.
> 
> >> Ignore: Skipping a test permanently is the same as just removing the
> >> test.. ??
> 
> > This is effectively the same here. Because the testcase should be
> > ignored (it is part of ignoredTests) it will not be added with
> > add_test(). E.g. it is not known to ctest.
> 
> However, as the test rules are something like: "test exporting all our
> manuals to PDF (pdflatex)", we need to specify cases where a manual is
> known not to compile, e.g. because it relies on non-TeX fonts or special
> features of LuaTeX.

I was describing the machinery how our ctests currently works. Not politics, 
that is how we organize
our config files (ignoredTests, revertedTests, suspendedTests).

> Günter
> 

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: status update on the export tests

[Guenter Milde]

On 2015-11-04, Kornel Benko wrote:

> [-- Type: text/plain, Encoding: 7bit --]

> Am Mittwoch, 4. November 2015 um 14:36:19, schrieb Vincent van Ravesteijn 
> 
>> > In my view, suspension is orthogonal to reversion:
>> >
>> > - normal:  we want the test to pass
>> >   revert:  we want the test to fail

>> You mean: revert is that "it is known to fail", but we haven't fixed it yet.

> Yes. We hope to do it eventually.

We have also tests that correctly fail:
"it is known to fail", there is nothing to fix, and we must ensure that it
continues to fail.
The third point is, what prevents us from ignoring this test.

>> > - normal:  run the test
>> >   suspend: skip the test temporarily
>> >   ignore:  skip the test permanently
>> >

>> Suspending means: "The outcome is noisy, so skip it until someone
>> looks into it and makes the test better."

> Sort of, if we are careful enough about which test should go there.

Not only, with "suspending" I also mean "The outcome is of no value for
finding new bugs or regressions until someone solves the known bug ...".
However, we usually know what the outcome should be if the bug is solved: if
the expected outcome is "pass", this test should not be inverted.


> For the actual committed cmake build, suspended test means:
>   a.) Test is one of export tests
>   b.) Test is failing, and therefore it is part of revertedTests
>   c.) In 'normal' use like 'ctest -L export' it is hidden
>   d.) Test gets the ctest-label 'suspended'
> You can run all suspended tests with 'ctest -L suspended'.
> Or, for a known testname (say xyzzy) you can use 'ctest -R xyzzy'.

>> Ignore: Skipping a test permanently is the same as just removing the
>> test.. ??

> This is effectively the same here. Because the testcase should be
> ignored (it is part of ignoredTests) it will not be added with
> add_test(). E.g. it is not known to ctest.

However, as the test rules are something like: "test exporting all our
manuals to PDF (pdflatex)", we need to specify cases where a manual is
known not to compile, e.g. because it relies on non-TeX fonts or special
features of LuaTeX.

Günter

Re: status update on the export tests

[Kornel Benko]

Am Mittwoch, 4. November 2015 um 14:36:19, schrieb Vincent van Ravesteijn 

> > In my view, suspension is orthogonal to reversion:
> >
> > - normal:  we want the test to pass
> >   revert:  we want the test to fail
> 
> You mean: revert is that "it is known to fail", but we haven't fixed it yet.

Yes. We hope to do it eventually.

> >
> > - normal:  run the test
> >   suspend: skip the test temporarily
> >   ignore:  skip the test permanently
> >
> 
> Suspending means: "The outcome is noisy, so skip it until someone
> looks into it and makes the test better."

Sort of, if we are careful enough about which test should go there.

For the actual committed cmake build, suspended test means:
a.) Test is one of export tests
b.) Test is failing, and therefore it is part of revertedTests
c.) In 'normal' use like 'ctest -L export' it is hidden
d.) Test gets the ctest-label 'suspended'
You can run all suspended tests with 'ctest -L suspended'.
Or, for a known testname (say xyzzy) you can use 'ctest -R xyzzy'.

> Ignore: Skipping a test permanently is the same as just removing the test.. ??

This is effectively the same here. Because the testcase should be ignored (it 
is part of ignoredTests)
it will not be added with add_test(). E.g. it is not known to ctest.

> Vincent

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: status update on the export tests

[Vincent van Ravesteijn]

> In my view, suspension is orthogonal to reversion:
>
> - normal:  we want the test to pass
>   revert:  we want the test to fail

You mean: revert is that "it is known to fail", but we haven't fixed it yet.

>
> - normal:  run the test
>   suspend: skip the test temporarily
>   ignore:  skip the test permanently
>

Suspending means: "The outcome is noisy, so skip it until someone
looks into it and makes the test better."

Ignore: Skipping a test permanently is the same as just removing the test.. ??

Vincent

Re: status update on the export tests

[Guenter Milde]

On 2015-11-04, Kornel Benko wrote:
> Am Mittwoch, 4. November 2015 um 09:30:03, schrieb Guenter Milde 
> 
>> On 2015-11-02, Kornel Benko wrote:
>> > Am 2. November 2015 um 08:36:05, schrieb Guenter Milde 
>> >> On 2015-11-02, Scott Kostyshak wrote:
>> >> > On Sun, Nov 01, 2015 at 10:36:17PM +0100, Kornel Benko wrote:
>> >> >> Am 1. November 2015 um 21:27:11, schrieb Guenter Milde

>> >> >> > Could we introduce a new status "suspended" - meaning just skip the
>> >> >> > test until a known bug is solved as we know the result is
>> >> >> > insignificant until then?
...
>> >> Candidates for "suspended" tests are 

>> >> * "fragile" documents (many packages, ERT, heavy preamble, ...), 
>> >> * "fragile" export routes (XeTeX/LuaTeX with TeX fonts), 
>> >> * non-default export routes 
>> >> and especially combinations of these.


>> > OK, here is my suggestion
>> > 1.) We add the appropriate tests into revertedTests

>> Why? This would undo one important benefit of "suspended" tests:

> I was not clear about this. These tests will not be run with e.g.
> 'ctest -L export' (see the '-L' param). In this sense we get 'clean'
> export tests.

May be I was not clear either: 

In my view, suspension is orthogonal to reversion:

- normal:  we want the test to pass
  revert:  we want the test to fail

- normal:  run the test
  suspend: skip the test temporarily
  ignore:  skip the test permanently

In other words: while "normal" vs. "reverted" will be used to specify the
expected test result, ignore and suspend will be used if we established
the reason why a test currently gives the wrong result but cannot
immediately solve the issue.

Then we have 

* suspended reverted tests: should fail but currently passes, e.g.
wrong output/dataloss that does not lead to error.

* suspended normal tests: should pass but currently fails, e.g.
  ru/Intro.lyx with XeTeX+TeX-fonts

With the benefit:

>> >>   Suspending instead of reverting frees us from the need to
>> >>   reassess them if a change in the "real life documents" or a fix
>> >>   makes them pass again. Instead, they could/should be revised at a
>> >>   time we have fixed major known problems and no release
>> >>   pressure...


...

> So what do you propose for such tests (84% of export tests)?
> ctest -L export -N | wc => 3719
> ctest -L export -N |egrep '/(doc|examples)/'| wc => 3153
> 3153 / 3719 => 84.78%

I don't have any clue which tests are hidden behind these commands. The
general recipe is:

* inspect the file,
* run the export command "by hand",
* try to find the reason for the test failure result

If the current "inversion state" does not match the expected test outcome
(pass/fail), correct it.

If it is "easyfix", fix.
If fixing »the right way« takes time, suspend (stating the established
reason).

The idea is, to de-couple the test analysis (from failure to bug report)
from the repair of found problems. This will hopefully allow us 
to get the test suite clean without hackish quick-fixes.

> For now committed 1.b at 538228d

Thanks

Günter

Re: status update on the export tests

[Kornel Benko]

Am Mittwoch, 4. November 2015 um 09:30:03, schrieb Guenter Milde 

> On 2015-11-02, Kornel Benko wrote:
> > Am 2. November 2015 um 08:36:05, schrieb Guenter Milde 
> >> On 2015-11-02, Scott Kostyshak wrote:
> >> > On Sun, Nov 01, 2015 at 10:36:17PM +0100, Kornel Benko wrote:
> >> >> Am 1. November 2015 um 21:27:11, schrieb Guenter Milde
> 
> >> >> > Could we introduce a new status "suspended" - meaning just skip the
> >> >> > test until a known bug is solved as we know the result is
> >> >> > insignificant until then?
> 
> >> >> We already have such (specified in file "ignoredTests"). But as this
> >> >> tests are never executed, nobody cares for them anymore.
> >> >> The tests here are such, that we know, we never resolve them.
> 
> But even here, there are two cases: 
> 
> a) cases that fail for a good reason and should always fail 
>(e.g. export document with non-TeX fonts using 8-bit LaTeX)
>
> b) cases that should not fail but do (for reasons we cannot change).
> 
> While a) should be "permanently inverted" (to give a warning if a change
> makes the export pass but we expect the result to be faulty), b) is
> correctly placed in "ignored" -- the test case does not help detect LyX
> buts in any way.
> 
> 
> >> Suggestion:
> 
> >>   Specified in file "suspendedTests") with the reason for suspending
> >>   (bug report, commit that turned hidden problems into export failure, ...)
> 
> >>   These tests are normally skipped, but they are not forgotten.
> 
> >>   The tests here are such, that we know, we can resolve them but their
> >>   failure is a minor issue that can be postponed (comparable to enhancement
> >>   requests in Trac).
> 
> 
> >> Candidates for "suspended" tests are 
> 
> >> * "fragile" documents (many packages, ERT, heavy preamble, ...), 
> >> * "fragile" export routes (XeTeX/LuaTeX with TeX fonts), 
> >> * non-default export routes 
> 
> >> and especially combinations of these.
> 
> 
> > OK, here is my suggestion
> > 1.) We add the appropriate tests into revertedTests
> 
> Why? This would undo one important benefit of "suspended" tests:

I was not clear about this. These tests will not be run with e.g. 'ctest -L 
export' (see the '-L' param).
In this sense we get 'clean' export tests.

> >>   Suspending instead of reverting also frees us from the need to reassess
> >>   them if a change in the "real life documents" or a fix makes them pass
> >>   again. Instead, they could/should be revised at a time we have fixed
> >>   major known problems and no release pressure...
> 
> I'd only add the test to "revertedTests", too, if it is established that
> the correct result for this test is a failure.
> 
> > The content in suspendedTests may be (in our case) e.g.
> > pdf4_texF
> 
> > 1.a)
> > If a test is to be inverted, we check suspendedTest,
> > and if there, we ignore the testcase.
> 
> Here, I'd change the priority: 
>   
>   * with a normal run, all suspended tests are ignored,
>   
>   * with "run suspended tests" or "include suspended tests",
> suspended test are run, taking into consideration their
> "inversion state".
> 
> > or
> > 1.b)
> > Such a test may get a label "suspended" instead of
> > "export", so that 'ctest -L export' will be clean, but
> > we also have the possibility to use 'ctest -L
> > suspended'.
> 
> > This does neither effect non-inverted nor ignored.
> 
> My suggestion would affect non-inverted tests (the ones with the label
> "suspended").
> 
> This means for failing test you will have 3 options:
> 
> 1. if failing is the expected outcome: invert
> 2. if failing for permanent reasons that are none of our business: ignore
> 3. if failing for minor reasons or a known bug: suspend
> 
> Motivation: 
> 
> Some bugs only lead to failures depending on document content. We do
> "road testing" with real life documents (manuals, examples). Work on the
> documents could easily change the export status of tests without any
> progress on fixing the underlying problem! (Examples: adding/removing a
> character not available with non-default export route, adding non-ASCII
> character to the PDF Header Information with "inputenc=ascii",
> Writing my name as G\"unter in the PDF Header Information before solving
> #9830, ...)
> 
> However, editing manuals or examples should not require immediate
> revision of test cases using this document if the basic problem is not
> solved.

So what do you propose for such tests (84% of export tests)?
ctest -L export -N | wc => 3719
ctest -L export -N |egrep '/(doc|examples)/'| wc => 3153
3153 / 3719 => 84.78%

> OTOH, if bugs that triggered suspension of some test cases are solved,
> it is time to revisit the set of suspended tests, reactivate the ones
> that pass and re-label the ones that still fail.
> 
> 
> 
> > I prefer 1.b.
> 
> Whatever works best 

Re: status update on the export tests

[Kornel Benko]

Am Montag, 2. November 2015 um 08:39:59, schrieb Guenter Milde 

> On 2015-11-02, Scott Kostyshak wrote:
> 
> > Good point. On the other hand, if we make regex changes then it is
> > harder to revert them. Problems have come up before of us
> > misconstructing the regexes.
> 
> If possible, make separate regexp patterns for separate "fail reasons".
> Then, we just have to delete one regexp if the cause for the inversion is
> gone...

I don't understand. There is no such reference in testcase-names.

> Günter

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: status update on the export tests

[Guenter Milde]

On 2015-11-02, Kornel Benko wrote:
> Am 2. November 2015 um 08:36:05, schrieb Guenter Milde 
>> On 2015-11-02, Scott Kostyshak wrote:
>> > On Sun, Nov 01, 2015 at 10:36:17PM +0100, Kornel Benko wrote:
>> >> Am 1. November 2015 um 21:27:11, schrieb Guenter Milde

>> >> > Could we introduce a new status "suspended" - meaning just skip the
>> >> > test until a known bug is solved as we know the result is
>> >> > insignificant until then?

>> >> We already have such (specified in file "ignoredTests"). But as this
>> >> tests are never executed, nobody cares for them anymore.
>> >> The tests here are such, that we know, we never resolve them.

But even here, there are two cases: 

a) cases that fail for a good reason and should always fail 
   (e.g. export document with non-TeX fonts using 8-bit LaTeX)
   
b) cases that should not fail but do (for reasons we cannot change).

While a) should be "permanently inverted" (to give a warning if a change
makes the export pass but we expect the result to be faulty), b) is
correctly placed in "ignored" -- the test case does not help detect LyX
buts in any way.


>> Suggestion:

>>   Specified in file "suspendedTests") with the reason for suspending
>>   (bug report, commit that turned hidden problems into export failure, ...)

>>   These tests are normally skipped, but they are not forgotten.

>>   The tests here are such, that we know, we can resolve them but their
>>   failure is a minor issue that can be postponed (comparable to enhancement
>>   requests in Trac).


>> Candidates for "suspended" tests are 

>> * "fragile" documents (many packages, ERT, heavy preamble, ...), 
>> * "fragile" export routes (XeTeX/LuaTeX with TeX fonts), 
>> * non-default export routes 

>> and especially combinations of these.


> OK, here is my suggestion
> 1.) We add the appropriate tests into revertedTests

Why? This would undo one important benefit of "suspended" tests:

>>   Suspending instead of reverting also frees us from the need to reassess
>>   them if a change in the "real life documents" or a fix makes them pass
>>   again. Instead, they could/should be revised at a time we have fixed
>>   major known problems and no release pressure...

I'd only add the test to "revertedTests", too, if it is established that
the correct result for this test is a failure.

> The content in suspendedTests may be (in our case) e.g.
>   pdf4_texF

>   1.a)
>   If a test is to be inverted, we check suspendedTest,
>   and if there, we ignore the testcase.

Here, I'd change the priority: 

* with a normal run, all suspended tests are ignored,

* with "run suspended tests" or "include suspended tests",
  suspended test are run, taking into consideration their
  "inversion state".

> or
>   1.b)
>   Such a test may get a label "suspended" instead of
>   "export", so that 'ctest -L export' will be clean, but
>   we also have the possibility to use 'ctest -L
>   suspended'.

> This does neither effect non-inverted nor ignored.

My suggestion would affect non-inverted tests (the ones with the label
"suspended").

This means for failing test you will have 3 options:

1. if failing is the expected outcome: invert
2. if failing for permanent reasons that are none of our business: ignore
3. if failing for minor reasons or a known bug: suspend

Motivation: 

Some bugs only lead to failures depending on document content. We do
"road testing" with real life documents (manuals, examples). Work on the
documents could easily change the export status of tests without any
progress on fixing the underlying problem! (Examples: adding/removing a
character not available with non-default export route, adding non-ASCII
character to the PDF Header Information with "inputenc=ascii",
Writing my name as G\"unter in the PDF Header Information before solving
#9830, ...)

However, editing manuals or examples should not require immediate
revision of test cases using this document if the basic problem is not
solved.

OTOH, if bugs that triggered suspension of some test cases are solved,
it is time to revisit the set of suspended tests, reactivate the ones
that pass and re-label the ones that still fail.



> I prefer 1.b.

Whatever works best in praxi.

Günter

Re: status update on the export tests

[Scott Kostyshak]

On Mon, Nov 02, 2015 at 09:36:02PM +0100, Kornel Benko wrote:
> Am Montag, 2. November 2015 um 21:31:12, schrieb Kornel Benko 
> > Am Montag, 2. November 2015 um 11:21:09, schrieb Kornel Benko 
> > 
> > > Am Montag, 2. November 2015 um 08:36:05, schrieb Guenter Milde 
> > > 
> > > > Dear Scott and Kornel,
> > > > 
> > > > On 2015-11-02, Scott Kostyshak wrote:
> > > > > On Sun, Nov 01, 2015 at 10:36:17PM +0100, Kornel Benko wrote:
> > > > >> Am Sonntag, 1. November 2015 um 21:27:11, schrieb Guenter Milde 
> > > > >> 
> > 
> > ...
> > 
> > > > Candidates for "suspended" tests are 
> > > > 
> > > > * "fragile" documents (many packages, ERT, heavy preamble, ...), 
> > > > * "fragile" export routes (XeTeX/LuaTeX with TeX fonts), 
> > > > * non-default export routes 
> > > > 
> > > > and especially combinations of these.
> > > > 
> > > > 
> > > > Günter
> > > >   
> > > 
> > > OK, here is my suggestion
> > > 1.) We add the appropriate tests into revertedTests
> > > 
> > > The content in suspendedTests may be (in our case) e.g.
> > >   pdf4_texF
> > > 
> > >   1.a)
> > >   If a test is to be inverted, we check suspendedTest, and if 
> > > there, we ignore the testcase.
> > > 
> > > or
> > >   1.b)
> > >   Such a test may became a label "suspended" instead of "export", 
> > > so that
> > >   'ctest -L export' will be clean, but we also have the 
> > > possibility to
> > >   use 'ctest -L suspended'.
> > > 
> > > This does neither effect non-inverted nor ignored.
> > > 
> > > I prefer 1.b.
> > > 
> > 
> > Apparently nobody cares?
> > The attached works here.
> > 
> 
> Forgotten this one in the last post.
> Therefore I resend all.

I don't have much of an opinion on this. Part of me thinks that it adds
extra complication but on the other hand I understand the reasoning. If
you and Günter think it makes sense, then since no one else has an
opinion I would say go ahead. But please document the suspended tests,
what they mean, how to interpret, etc. in Development.lyx. I don't have
enough of an understanding to document it.

Thanks for your work on this! Soon hopefully the tests will be in a good
state.

Scott

> 
>   Kornel

> diff --git a/development/cmake/modules/LyXMacros.cmake 
> b/development/cmake/modules/LyXMacros.cmake
> index 2d20cb7..ea16243 100644
> --- a/development/cmake/modules/LyXMacros.cmake
> +++ b/development/cmake/modules/LyXMacros.cmake
> @@ -322,7 +322,7 @@ macro(settestlabel testname)
>  endmacro()
>  
>  macro(setmarkedtestlabel testname reverted)
> -  if(reverted)
> +  if(${reverted})
>  settestlabel(${testname} "reverted" ${ARGN})
>else()
>  settestlabel(${testname} ${ARGN})

> diff --git a/development/autotests/ExportTests.cmake 
> b/development/autotests/ExportTests.cmake
> index 50b4aa4..c3367a3 100644
> --- a/development/autotests/ExportTests.cmake
> +++ b/development/autotests/ExportTests.cmake
> @@ -137,8 +137,26 @@ endmacro()
>  
>  loadTestList(revertedTests revertedTests)
>  loadTestList(ignoredTests ignoredTests)
> +loadTestList(suspendedTests suspendedTests)
>  
> -foreach(libsubfolder doc examples templates)
> +macro(handlesuspended TestName reverted testlabel)
> +  set(mylabel ${testlabel})
> +  set(myreverted ${reverted})
> +  if (${reverted})
> +# check suspension only for reverted tests
> +findexpr(tfound TestName suspendedTests)
> +if (tfound)
> +  set(mylabel "suspended")
> +  set(myreverted 0) # if test is to be suspended, remove the 'reverted' 
> flag
> +endif()
> +  endif()
> +  setmarkedtestlabel(${TestName} ${myreverted} ${mylabel})
> +endmacro()
> +
> +# preparing to add e.g. development/mathmacro to the foreach() loop
> +foreach(libsubfolderx lib/doc lib/examples lib/templates)
> +  set(LIBSUB_SRC_DIR "${TOP_SRC_DIR}/${libsubfolderx}")
> +  string(REGEX REPLACE "^(lib|development)/" "" libsubfolder 
> "${libsubfolderx}")
>set(LIBSUB_SRC_DIR "${TOP_SRC_DIR}/lib/${libsubfolder}")
>file(GLOB_RECURSE lyx_files RELATIVE "${LIBSUB_SRC_DIR}" 
> "${LIBSUB_SRC_DIR}/*.lyx")
>list(SORT lyx_files)
> @@ -173,7 +191,7 @@ foreach(libsubfolder doc examples templates)
>  -DTOP_SRC_DIR=${TOP_SRC_DIR}
>  -DPERL_EXECUTABLE=${PERL_EXECUTABLE}
>  -P "${TOP_SRC_DIR}/development/autotests/export.cmake")
> -  setmarkedtestlabel(${TestName} ${reverted} "export")
> +  handlesuspended(${TestName} ${reverted} "export") # checking for 
> suspended lyx16 exports
>  endif()
>  if(LYX_PYTHON_EXECUTABLE)
># For use of lyx2lyx we need the python executable
> @@ -188,7 +206,7 @@ foreach(libsubfolder doc examples templates)
>"-DLYX_TESTS_USERDIR=${LYX_TESTS_USERDIR}"
>"-DLYXFILE=${LIBSUB_SRC_DIR}/${f}.lyx"
>-P "${TOP_SRC_DIR}/development/autotests/lyx2lyxtest.cmake")
> -setmarkedtestlabel(${TestName} ${reverted} "lyx2lyx")
> +handlesuspended(${TestName} ${reverted} "lyx2lyx")
>endif()
>  endif()
>  set(TestNa

Re: status update on the export tests

[Kornel Benko]

Am Montag, 2. November 2015 um 21:31:12, schrieb Kornel Benko 
> Am Montag, 2. November 2015 um 11:21:09, schrieb Kornel Benko 
> > Am Montag, 2. November 2015 um 08:36:05, schrieb Guenter Milde 
> > 
> > > Dear Scott and Kornel,
> > >
> > > On 2015-11-02, Scott Kostyshak wrote:
> > > > On Sun, Nov 01, 2015 at 10:36:17PM +0100, Kornel Benko wrote:
> > > >> Am Sonntag, 1. November 2015 um 21:27:11, schrieb Guenter Milde 
> > > >> 
>
> ...
>
> > > Candidates for "suspended" tests are
> > >
> > > * "fragile" documents (many packages, ERT, heavy preamble, ...),
> > > * "fragile" export routes (XeTeX/LuaTeX with TeX fonts),
> > > * non-default export routes
> > >
> > > and especially combinations of these.
> > >
> > >
> > > Günter
> > >
> >
> > OK, here is my suggestion
> > 1.) We add the appropriate tests into revertedTests
> >
> > The content in suspendedTests may be (in our case) e.g.
> > pdf4_texF
> >
> > 1.a)
> > If a test is to be inverted, we check suspendedTest, and if 
> > there, we ignore the testcase.
> >
> > or
> > 1.b)
> > Such a test may became a label "suspended" instead of "export", 
> > so that
> > 'ctest -L export' will be clean, but we also have the 
> > possibility to
> > use 'ctest -L suspended'.
> >
> > This does neither effect non-inverted nor ignored.
> >
> > I prefer 1.b.
> >
>
> Apparently nobody cares?
> The attached works here.
>

Forgotten this one in the last post.
Therefore I resend all.

Korneldiff --git a/development/cmake/modules/LyXMacros.cmake b/development/cmake/modules/LyXMacros.cmake
index 2d20cb7..ea16243 100644
--- a/development/cmake/modules/LyXMacros.cmake
+++ b/development/cmake/modules/LyXMacros.cmake
@@ -322,7 +322,7 @@ macro(settestlabel testname)
 endmacro()

 macro(setmarkedtestlabel testname reverted)
-  if(reverted)
+  if(${reverted})
 settestlabel(${testname} "reverted" ${ARGN})
   else()
 settestlabel(${testname} ${ARGN})
diff --git a/development/autotests/ExportTests.cmake b/development/autotests/ExportTests.cmake
index 50b4aa4..c3367a3 100644
--- a/development/autotests/ExportTests.cmake
+++ b/development/autotests/ExportTests.cmake
@@ -137,8 +137,26 @@ endmacro()

 loadTestList(revertedTests revertedTests)
 loadTestList(ignoredTests ignoredTests)
+loadTestList(suspendedTests suspendedTests)

-foreach(libsubfolder doc examples templates)
+macro(handlesuspended TestName reverted testlabel)
+  set(mylabel ${testlabel})
+  set(myreverted ${reverted})
+  if (${reverted})
+# check suspension only for reverted tests
+findexpr(tfound TestName suspendedTests)
+if (tfound)
+  set(mylabel "suspended")
+  set(myreverted 0) # if test is to be suspended, remove the 'reverted' flag
+endif()
+  endif()
+  setmarkedtestlabel(${TestName} ${myreverted} ${mylabel})
+endmacro()
+
+# preparing to add e.g. development/mathmacro to the foreach() loop
+foreach(libsubfolderx lib/doc lib/examples lib/templates)
+  set(LIBSUB_SRC_DIR "${TOP_SRC_DIR}/${libsubfolderx}")
+  string(REGEX REPLACE "^(lib|development)/" "" libsubfolder "${libsubfolderx}")
   set(LIBSUB_SRC_DIR "${TOP_SRC_DIR}/lib/${libsubfolder}")
   file(GLOB_RECURSE lyx_files RELATIVE "${LIBSUB_SRC_DIR}" "${LIBSUB_SRC_DIR}/*.lyx")
   list(SORT lyx_files)
@@ -173,7 +191,7 @@ foreach(libsubfolder doc examples templates)
 -DTOP_SRC_DIR=${TOP_SRC_DIR}
 -DPERL_EXECUTABLE=${PERL_EXECUTABLE}
 -P "${TOP_SRC_DIR}/development/autotests/export.cmake")
-  setmarkedtestlabel(${TestName} ${reverted} "export")
+  handlesuspended(${TestName} ${reverted} "export") # checking for suspended lyx16 exports
 endif()
 if(LYX_PYTHON_EXECUTABLE)
   # For use of lyx2lyx we need the python executable
@@ -188,7 +206,7 @@ foreach(libsubfolder doc examples templates)
   "-DLYX_TESTS_USERDIR=${LYX_TESTS_USERDIR}"
   "-DLYXFILE=${LIBSUB_SRC_DIR}/${f}.lyx"
   -P "${TOP_SRC_DIR}/development/autotests/lyx2lyxtest.cmake")
-setmarkedtestlabel(${TestName} ${reverted} "lyx2lyx")
+handlesuspended(${TestName} ${reverted} "lyx2lyx")
   endif()
 endif()
 set(TestName "check_load/${libsubfolder}/${f}")
@@ -202,7 +220,7 @@ foreach(libsubfolder doc examples templates)
 -DPARAMS_DIR=${TOP_SRC_DIR}/development/autotests
 -DWORKDIR=${CMAKE_CURRENT_BINARY_DIR}/${LYX_HOME}
 -P "${TOP_SRC_DIR}/development/autotests/check_load.cmake")
-  setmarkedtestlabel(${TestName} ${reverted} "load")
+  handlesuspended(${TestName} ${reverted} "load")
   set_tests_properties(${TestName} PROPERTIES RUN_SERIAL ON)
 endif()
 getoutputformats("${LIBSUB_SRC_DIR}/${f}.lyx" formatlist)
@@ -234,7 +252,7 @@ foreach(libsubfolder doc examples templates)
 -DTOP_SRC_DIR=${TOP_SRC_DIR}
 -DPERL_EXECUTABLE=${PERL_EXECUTABLE}
 -P "${TOP_SRC_DIR}/development/autotests/export.cmake")
-  setmarkedtestlabel(${TestName} ${reverted

Re: status update on the export tests

[Kornel Benko]

Am Montag, 2. November 2015 um 11:21:09, schrieb Kornel Benko 
> Am Montag, 2. November 2015 um 08:36:05, schrieb Guenter Milde 
> 
> > Dear Scott and Kornel,
> >
> > On 2015-11-02, Scott Kostyshak wrote:
> > > On Sun, Nov 01, 2015 at 10:36:17PM +0100, Kornel Benko wrote:
> > >> Am Sonntag, 1. November 2015 um 21:27:11, schrieb Guenter Milde 
> > >> 

...

> > Candidates for "suspended" tests are
> >
> > * "fragile" documents (many packages, ERT, heavy preamble, ...),
> > * "fragile" export routes (XeTeX/LuaTeX with TeX fonts),
> > * non-default export routes
> >
> > and especially combinations of these.
> >
> >
> > Günter
> >
>
> OK, here is my suggestion
> 1.) We add the appropriate tests into revertedTests
>
> The content in suspendedTests may be (in our case) e.g.
>   pdf4_texF
>
>   1.a)
>   If a test is to be inverted, we check suspendedTest, and if 
> there, we ignore the testcase.
>
> or
>   1.b)
>   Such a test may became a label "suspended" instead of "export", 
> so that
>   'ctest -L export' will be clean, but we also have the 
> possibility to
>   use 'ctest -L suspended'.
>
> This does neither effect non-inverted nor ignored.
>
> I prefer 1.b.
>

Apparently nobody cares?
The attached works here.

Korneldiff --git a/development/autotests/ExportTests.cmake b/development/autotests/ExportTests.cmake
index 50b4aa4..c3367a3 100644
--- a/development/autotests/ExportTests.cmake
+++ b/development/autotests/ExportTests.cmake
@@ -137,8 +137,26 @@ endmacro()

 loadTestList(revertedTests revertedTests)
 loadTestList(ignoredTests ignoredTests)
+loadTestList(suspendedTests suspendedTests)

-foreach(libsubfolder doc examples templates)
+macro(handlesuspended TestName reverted testlabel)
+  set(mylabel ${testlabel})
+  set(myreverted ${reverted})
+  if (${reverted})
+# check suspension only for reverted tests
+findexpr(tfound TestName suspendedTests)
+if (tfound)
+  set(mylabel "suspended")
+  set(myreverted 0) # if test is to be suspended, remove the 'reverted' flag
+endif()
+  endif()
+  setmarkedtestlabel(${TestName} ${myreverted} ${mylabel})
+endmacro()
+
+# preparing to add e.g. development/mathmacro to the foreach() loop
+foreach(libsubfolderx lib/doc lib/examples lib/templates)
+  set(LIBSUB_SRC_DIR "${TOP_SRC_DIR}/${libsubfolderx}")
+  string(REGEX REPLACE "^(lib|development)/" "" libsubfolder "${libsubfolderx}")
   set(LIBSUB_SRC_DIR "${TOP_SRC_DIR}/lib/${libsubfolder}")
   file(GLOB_RECURSE lyx_files RELATIVE "${LIBSUB_SRC_DIR}" "${LIBSUB_SRC_DIR}/*.lyx")
   list(SORT lyx_files)
@@ -173,7 +191,7 @@ foreach(libsubfolder doc examples templates)
 -DTOP_SRC_DIR=${TOP_SRC_DIR}
 -DPERL_EXECUTABLE=${PERL_EXECUTABLE}
 -P "${TOP_SRC_DIR}/development/autotests/export.cmake")
-  setmarkedtestlabel(${TestName} ${reverted} "export")
+  handlesuspended(${TestName} ${reverted} "export") # checking for suspended lyx16 exports
 endif()
 if(LYX_PYTHON_EXECUTABLE)
   # For use of lyx2lyx we need the python executable
@@ -188,7 +206,7 @@ foreach(libsubfolder doc examples templates)
   "-DLYX_TESTS_USERDIR=${LYX_TESTS_USERDIR}"
   "-DLYXFILE=${LIBSUB_SRC_DIR}/${f}.lyx"
   -P "${TOP_SRC_DIR}/development/autotests/lyx2lyxtest.cmake")
-setmarkedtestlabel(${TestName} ${reverted} "lyx2lyx")
+handlesuspended(${TestName} ${reverted} "lyx2lyx")
   endif()
 endif()
 set(TestName "check_load/${libsubfolder}/${f}")
@@ -202,7 +220,7 @@ foreach(libsubfolder doc examples templates)
 -DPARAMS_DIR=${TOP_SRC_DIR}/development/autotests
 -DWORKDIR=${CMAKE_CURRENT_BINARY_DIR}/${LYX_HOME}
 -P "${TOP_SRC_DIR}/development/autotests/check_load.cmake")
-  setmarkedtestlabel(${TestName} ${reverted} "load")
+  handlesuspended(${TestName} ${reverted} "load")
   set_tests_properties(${TestName} PROPERTIES RUN_SERIAL ON)
 endif()
 getoutputformats("${LIBSUB_SRC_DIR}/${f}.lyx" formatlist)
@@ -234,7 +252,7 @@ foreach(libsubfolder doc examples templates)
 -DTOP_SRC_DIR=${TOP_SRC_DIR}
 -DPERL_EXECUTABLE=${PERL_EXECUTABLE}
 -P "${TOP_SRC_DIR}/development/autotests/export.cmake")
-  setmarkedtestlabel(${TestName} ${reverted} "export")
+  handlesuspended(${TestName} ${reverted} "export") # check for suspended pdf/dvi exports
 endif()
   endforeach()
 endforeach()
INV.*pdf4_texF


signature.asc
Description: This is a digitally signed message part.

Re: status update on the export tests

[Kornel Benko]

Am Montag, 2. November 2015 um 08:36:05, schrieb Guenter Milde 

> Dear Scott and Kornel,
> 
> On 2015-11-02, Scott Kostyshak wrote:
> > On Sun, Nov 01, 2015 at 10:36:17PM +0100, Kornel Benko wrote:
> >> Am Sonntag, 1. November 2015 um 21:27:11, schrieb Guenter Milde 
> >> 
> 
> >> > Could we introduce a new status "suspended" - meaning just skip the
> >> > test until a known bug is solved as we know the result is
> >> > insignificant until then?
> 
> > As Kornel mentions, we have "ignoredTests" for this.
> 
> Not exactly, see below.
> 
> >> We already have such (specified in file "ignoredTests"). But as this
> >> tests are never executed, nobody cares for them anymore.
> >> The tests here are such, that we know, we never resolve them.
> 
> > I agree. We should not use ignored tests for temporary issues. Once we
> > put something in ignoredTests chances are strong that we will forget
> > about them for a long time.
> 
> This is why I said "suspended": 
> 
> Suggestion:
> 
>   Specified in file "suspendedTests") with the reason for suspending
>   (bug report, commit that turned hidden problems into export failure, ...)
>   
>   These tests are normally skipped, but they are not forgotten.
>   
>   The tests here are such, that we know, we can resolve them but their
>   failure is a minor issue that can be postponed (comparable to enhancement
>   requests in Trac).
>   
>   Suspending instead of reverting also frees us from the need to reassess
>   them if a change in the "real life documents" or a fix makes them pass
>   again. Instead, they could/should be revised at a time we have fixed
>   major known problems and no release pressure...
>   
>   
> Candidates for "suspended" tests are 
> 
> * "fragile" documents (many packages, ERT, heavy preamble, ...), 
> * "fragile" export routes (XeTeX/LuaTeX with TeX fonts), 
> * non-default export routes 
> 
> and especially combinations of these.
> 
> 
> Günter
>   

OK, here is my suggestion
1.) We add the appropriate tests into revertedTests

The content in suspendedTests may be (in our case) e.g.
pdf4_texF

1.a)
If a test is to be inverted, we check suspendedTest, and if 
there, we ignore the testcase.

or
1.b)
Such a test may became a label "suspended" instead of "export", 
so that
'ctest -L export' will be clean, but we also have the 
possibility to
use 'ctest -L suspended'.

This does neither effect non-inverted nor ignored.

I prefer 1.b.

Kornel


signature.asc
Description: This is a digitally signed message part.

Re: status update on the export tests

[Guenter Milde]

On 2015-11-02, Scott Kostyshak wrote:

> Good point. On the other hand, if we make regex changes then it is
> harder to revert them. Problems have come up before of us
> misconstructing the regexes.

If possible, make separate regexp patterns for separate "fail reasons".
Then, we just have to delete one regexp if the cause for the inversion is
gone...

Günter

Re: status update on the export tests

[Guenter Milde]

Dear Scott and Kornel,

On 2015-11-02, Scott Kostyshak wrote:
> On Sun, Nov 01, 2015 at 10:36:17PM +0100, Kornel Benko wrote:
>> Am Sonntag, 1. November 2015 um 21:27:11, schrieb Guenter Milde 
>> 

>> > Could we introduce a new status "suspended" - meaning just skip the
>> > test until a known bug is solved as we know the result is
>> > insignificant until then?

> As Kornel mentions, we have "ignoredTests" for this.

Not exactly, see below.

>> We already have such (specified in file "ignoredTests"). But as this
>> tests are never executed, nobody cares for them anymore.
>> The tests here are such, that we know, we never resolve them.

> I agree. We should not use ignored tests for temporary issues. Once we
> put something in ignoredTests chances are strong that we will forget
> about them for a long time.

This is why I said "suspended": 

Suggestion:

  Specified in file "suspendedTests") with the reason for suspending
  (bug report, commit that turned hidden problems into export failure, ...)
  
  These tests are normally skipped, but they are not forgotten.
  
  The tests here are such, that we know, we can resolve them but their
  failure is a minor issue that can be postponed (comparable to enhancement
  requests in Trac).
  
  Suspending instead of reverting also frees us from the need to reassess
  them if a change in the "real life documents" or a fix makes them pass
  again. Instead, they could/should be revised at a time we have fixed
  major known problems and no release pressure...
  
  
Candidates for "suspended" tests are 

* "fragile" documents (many packages, ERT, heavy preamble, ...), 
* "fragile" export routes (XeTeX/LuaTeX with TeX fonts), 
* non-default export routes 

and especially combinations of these.


Günter
  
  
  

Re: status update on the export tests

[Scott Kostyshak]

On Sun, Nov 01, 2015 at 10:31:32PM +, Guenter Milde wrote:
> On 2015-11-01, Scott Kostyshak wrote:
> > On Sun, Nov 01, 2015 at 07:37:41PM +, Guenter Milde wrote:
> >> On 2015-10-30, Kornel Benko wrote:
> >> > Am Freitag, 30. Oktober 2015 um 19:23:02, schrieb Scott Kostyshak 
> >> > 
> 
> >> > Besides we have ATM about 200 failing export test cases.
> 
> >> Too many to be helpfull.
> 
> > I don't understand this logic.
> 
> For me, the tests are meant to find problems with LyX code.
> Also, they should ensure that commits do not break anything important.
> 
> With 200 failing test cases, the test suite starts to resemble the issue
> tracker: we can not ensure there is no known bug in LyX, nor can we
> ensure that the test suite runs without failure.
> 
> We are classifying bug reports and start with the important ones.
> 
> The same should be done for failing export tests. In order to concentrate
> on the important stuff, we need a way to filter less important tests
> (maybe just do not run them.) But like the less important bug reports, we
> should not remove them altogether without convincing reason.

I see what you mean. Thanks for explaining this. I agree.

> >> How many of these are for the obscure combination of XeTeX and TeX fonts?
> >> While there is a use case for LuaTeX and TeX fonts, I can't see a reason to
> >> use XeTeX instead of pdflatex with TeX fonts!
> 
> > The use case that I have in mind is simply that I expect our users might
> > use such a combination. If we don't want to support the combination, we
> > should output a useful error message explaining why, or just disable the
> > combination. If not, users will use it. 
> 
> Or treat the combination XeTeX + TeX fonts as a non-standard use case
> that we do not prevent but discourage.

OK but how do we discourage? In the documentation? What about one of
those warning boxes that has the checkbox "don't show this warning
again". So the first time a user tries to compile with XeTeX and TeX
fonts it warns them "It is not useful to use XeTeX and TeX fonts. You
might prefer to use PDF (pdfTeX) or use PDF (XeTeX) but use system fonts
by going to Document > Settings > Fonts.

> This is similar to, e.g. the "language default (without inputenc)" option
> for the inputencoding: this option fails inevitably without additional user
> input in the LaTeX preamble.

OK but I think it is a stranger case that a user goes into the settings
and changes that than just click on the button for PDF (XeTeX).

> > Many LyX users do not have the understanding that you do. They do not
> > even understand what it means to use non-TeX fonts. They might just
> > read something about XeTeX and start using it by clicking on our button
> > that says PDF (XeTeX). 
> 
> The "inviting" toolbar button starting this obscure and bad-supported
> export route is the real problem.

Ah yes I agree with this.

> This is, basically, what I want in Ticket #9744: the toolbar button would
> call XeTeX with Unicode fonts by default.

I see. That does seem like a good idea then.

> > The purpose of the tests is just to prevent regressions. Sure, maybe it
> > is true that something was working by luck before, but I still think it
> > is reasonable to try to preserve it. 
> 
> Restoring behaviour that was half-broken before and will be of no use
> later (except for ill guided one) should not be a precondition for
> getting the test suite in a usable shape again.
> 
> The follow-up fixes led to even more "regressions" (or to surfacing of
> hidden problems).

I see. I did not know it was a similar situation.

> >> Solving all issues that arise from this combination is diverting
> >> attention and resources from more important tasks.
> 
> > Agreed.
> 
> >> Also, we identified some generic problems with this combination that are
> >> not solvable in the short term: third party packages well as documents
> >> not prepared for this use case.
> 
> >> Just reverting failing XeTeX export tests for the moment would allow us to
> >> concentrate on the remaining failing test and get the test suite in a
> >> usable state again.
> 
> > Fine with me. Kornel, OK with you? We could create a bug report so that
> > we do not forget about the tests. Then we could just invert all of the
> > pdf4_texF tests that are failing. I have in mind just pasting the long
> > list of individual tests, rather than using regexes.
> 
> Actually, I'd rather suspend than revert - whether they fail or pass has
> currently no predictive value regarding regression or progress in solving
> the underlying problems.

I see your reasoning now. I will start a new thread about testing where
we can discuss this.

> I am all for the additon of special test cases. 
> We should find a way to specify the to-be-tested export route(s) (special
> naming or sub-directories, say) for these test documents.

I think we all like these special test cases. Hopefully we can implement
more of them to avoid many of the fragile tests we have.

Sc

Re: status update on the export tests

[Scott Kostyshak]

On Sun, Nov 01, 2015 at 09:39:24PM +0100, Kornel Benko wrote:

> I have to be convinced.

Fair. I think that whenever tests fail the burden is on those who wish
to invert or temporarily ignore the tests. I am not proud to be on the
dark side of wanting to invert the tests, but I think that is where I am
right now because I think that we need to get clean output so we can
catch future regressions.

> It is not OK with me, see my other post.
> Remark to regexes:
> 1.) it is easier for us to read such a file
> 2.) but is takes longer time to interpret.
> 
> #ctest -j12 -R 'export/.*(pdf|dvi)' -N  | wc
>   3085
> ATM, we have 59 lines to identify 300 inverted tests.
> Adding the new 200 we have 500 individual lines.
> That means we have to check (500/2) * 3085 combinations
> to determine which test is inverted and which not.

Good point. On the other hand, if we make regex changes then it is
harder to revert them. Problems have come up before of us
misconstructing the regexes. This is easily fixed though. I might add
the lines locally just to see how many clock seconds it increases the
cmake call.

Scott

Re: status update on the export tests

[Scott Kostyshak]

On Sun, Nov 01, 2015 at 11:05:06PM +0100, Kornel Benko wrote:

> My argument is not that they do not belong into the set of reverted, but I'd 
> like them
> to be checked first.
> They will be included with the appropriate comment.

+1 

Scott

Re: status update on the export tests

[Scott Kostyshak]

On Sun, Nov 01, 2015 at 09:48:28PM +, Guenter Milde wrote:
> On 2015-11-01, Kornel Benko wrote:

> > Most of these tests were working some time ago. 
> 
> Many of them "by chance": not failing but with incorrect output (Missing
> characters, wrong characters) or "fragile" (working, but would fail by
> addition of one non-ASCII character, say).

Agreed. I don't think "by chance" correctly describes the missing
characters situation, but only the "fragile" documents.

> > We already have many inverted test.
> 
> This also adds to the impression, that this is an area where test
> failures are to be expected for many reasons. I.e. the signal to noise ratio
> is rather bad for XeTeX+TeX fonts and we would be better of without these
> tests.

This might be. I admit I have very little knowledge of XeTeX + TeX
fonts.

> > I am strongly against such policy. First one has to check if the reason is 
> > really
> > babel/polyglossia conflict.
> 
> There are many more reasons, mhchem, babel language files, font problems, ...
> 
> The combination XeTeX + TeX-fonts is not supported by many packages,
> including standard LaTeX packages like "inputenc"! OTOH, it is so obscure
> that it is really not worth wasting much time to work around all the
> problems. Rather treat/document it as: avoid if possible, use at your own
> risk.

I am starting to be open to this idea. If it is true that performing a
simple, correct operation in a document leads to a significant chance of
it breaking compilation of XeTeX with TeX fonts, then it is true that
this could cause a lot of noise. On one hand, our manuals were pretty
darn complicated and compiled just fine with TeX fonts. This might be
partly due to 'luck' but since they use so many packages and ERT and
weird constructs, that would be a lot of luck. I guess the main issue is
the missing characters.

Scott

Re: status update on the export tests

[Scott Kostyshak]

On Sun, Nov 01, 2015 at 10:36:17PM +0100, Kornel Benko wrote:
> Am Sonntag, 1. November 2015 um 21:27:11, schrieb Guenter Milde 
> 

> > Could we introduce a new status "suspended" - meaning just skip the test
> > until a known bug is solved as we know the result is insignificant until 
> > then?

As Kornel mentions, we have "ignoredTests" for this.

> > This way, known problems will not mask new problems.

I agree with this goal. The current situation is a problem because like
you say it might mask new problems. I agree with Kornel that ignored
tests are not the solution. But I think that inverting the tests might
be a temporary solution. The only way to get clean working tests would
be to rever the commits and I don't think that is the right action for
this situation.

> We already have such (specified in file "ignoredTests"). But as this tests 
> are never executed,
> nobody cares for them anymore.
> The tests here are such, that we know, we never resolve them.
> Example:
> We write a lyx file for odt/word/whatever output only. There is sense to 
> expect
> valid latex.

I agree. We should not use ignored tests for temporary issues. Once we
put something in ignoredTests chances are strong that we will forget
about them for a long time.

Scott

Re: status update on the export tests

[Scott Kostyshak]

On Sun, Nov 01, 2015 at 10:07:02PM +0100, Georg Baum wrote:
> Kornel Benko wrote:
> 
> > Does it mean 'delay 2.2 alpha'?
> 
> I don't think so. 2.1 has language nesting bugs as well, and the more simple 
> cases do work. But we should probably mention that there are known language 
> nesting bugs in the release notes. It is alpha after all (and I still hope 
> to get some safe improvement for the beta).

I agree with Georg. I think these issues should not block alpha. It
might be reasonable to have a policy of "no failing tests" for beta in
the future. This doesn't mean that we fix all bugs. It just means that
every test failure was looked at. Maybe we looked at a test and decided
it is OK for it to fail so we invert it and thus it no longer fails.

Scott

Re: status update on the export tests

[Scott Kostyshak]

On Sun, Nov 01, 2015 at 09:14:53PM +0100, Kornel Benko wrote:
> Am Sonntag, 1. November 2015 um 19:37:41, schrieb Guenter Milde 
> 
> > On 2015-10-30, Kornel Benko wrote:

> > Solving all issues that arise from this combination is diverting attention
> > and ressources from more important tasks. 
> 
> Most of these tests were working some time ago. We already have many inverted 
> test.
> I am strongly against such policy. First one has to check if the reason is 
> really
> babel/polyglossia conflict.

I agree that we should check before reverting. Actually I don't think
babel/polyglossia plays a role in most of the failing tests. There are
only a few that fail because of this (you can see this by reverting the
polyglossia commit, edd37de8). Georg has already done a good job of
improving things and only a few affected tests remain.

The main issue is the missing characters commit, I think. And this is a
tricky issue because the commit that broke the tests was actually a good
commit---it did not introduce a bug. What to do in such a case? I'm not
sure. Ideally we would have a separate branch where we try to address
the tests (either fixing the underlying bugs or fixing the .lyx files or
inverting the tests). This way the tests on master continue to be clean
and it is easy to see if there are any regressions. But I fear this is
too much of an inconvenience to people who already are annoyed by the
tests. 

> > Also, we identified some generic problems with this combination that are
> > not solvable in the short term: third party packages well as documents
> > not prepared for this use case.
> > 
> > Just reverting failing Xetex export tests for the moment would allow us to
> > concentrfate on the remaining failing test and get the test suite in a
> > usable state again.
> 
> Optimist (I mean 'usable state').
> I am strongly against such policy. First one has to check if the reason is 
> really
> babel/polyglossia conflict.
> There are already too many tests inverted, no one cares anymore.

Actually the last few weeks have seen the most interest in the tests
yet. Other developers are trying to run the tests, which is already an
improvement. I would say this is the height of interest in the export
tests :)

Scott

Re: status update on the export tests

[Guenter Milde]

On 2015-11-01, Scott Kostyshak wrote:
> On Sun, Nov 01, 2015 at 07:37:41PM +, Guenter Milde wrote:
>> On 2015-10-30, Kornel Benko wrote:
>> > Am Freitag, 30. Oktober 2015 um 19:23:02, schrieb Scott Kostyshak 
>> > 

>> > Besides we have ATM about 200 failing export test cases.

>> Too many to be helpfull.

> I don't understand this logic.

For me, the tests are meant to find problems with LyX code.
Also, they should ensure that commits do not break anything important.

With 200 failing test cases, the test suite starts to resemble the issue
tracker: we can not ensure there is no known bug in LyX, nor can we
ensure that the test suite runs without failure.

We are classifying bug reports and start with the important ones.

The same should be done for failing export tests. In order to concentrate
on the important stuff, we need a way to filter less important tests
(maybe just do not run them.) But like the less important bug reports, we
should not remove them altogether without convincing reason.

...

>> How many of these are for the obscure combination of XeTeX and TeX fonts?
>> While there is a use case for LuaTeX and TeX fonts, I can't see a reason to
>> use XeTeX instead of pdflatex with TeX fonts!

> The use case that I have in mind is simply that I expect our users might
> use such a combination. If we don't want to support the combination, we
> should output a useful error message explaining why, or just disable the
> combination. If not, users will use it. 

Or treat the combination XeTeX + TeX fonts as a non-standard use case
that we do not prevent but discourage.

This is similar to, e.g. the "language default (without inputenc)" option
for the inputencoding: this option fails inevitably without additional user
input in the LaTeX preamble.

> Many LyX users do not have the understanding that you do. They do not
> even understand what it means to use non-TeX fonts. They might just
> read something about XeTeX and start using it by clicking on our button
> that says PDF (XeTeX). 

The "inviting" toolbar button starting this obscure and bad-supported
export route is the real problem.

This is, basically, what I want in Ticket #9744: the toolbar button would
call XeTeX with Unicode fonts by default.

> The purpose of the tests is just to prevent regressions. Sure, maybe it
> is true that something was working by luck before, but I still think it
> is reasonable to try to preserve it. 

Restoring behaviour that was half-broken before and will be of no use
later (except for ill guided one) should not be a precondition for
getting the test suite in a usable shape again.


> So I defend the tests in general. The current situation is a special
> case. You fixed a bug (the missing characters bug) 

I just reported it but did not fix it.

> and that is why so many tests are failing. I do not imagine such a
> situation coming up often.

The follow-up fixes led to even more "regressions" (or to surfacing of
hidden problems).

>> Solving all issues that arise from this combination is diverting
>> attention and resources from more important tasks.

> Agreed.

>> Also, we identified some generic problems with this combination that are
>> not solvable in the short term: third party packages well as documents
>> not prepared for this use case.

>> Just reverting failing XeTeX export tests for the moment would allow us to
>> concentrate on the remaining failing test and get the test suite in a
>> usable state again.

> Fine with me. Kornel, OK with you? We could create a bug report so that
> we do not forget about the tests. Then we could just invert all of the
> pdf4_texF tests that are failing. I have in mind just pasting the long
> list of individual tests, rather than using regexes.

Actually, I'd rather suspend than revert - whether they fail or pass has
currently no predictive value regarding regression or progress in solving
the underlying problems.

I am all for the additon of special test cases. 
We should find a way to specify the to-be-tested export route(s) (special
naming or sub-directories, say) for these test documents.

Günter

Re: status update on the export tests

[Kornel Benko]

Am Sonntag, 1. November 2015 um 21:48:28, schrieb Guenter Milde 

> On 2015-11-01, Kornel Benko wrote:
> > Am Sonntag, 1. November 2015 um 19:37:41, schrieb Guenter Milde 
> > 
> >> On 2015-10-30, Kornel Benko wrote:
> 
> ...
> 
> >> > Besides we have ATM about 200 failing export test cases.
> 
> ...
> 
> >> How many of these are for the obscure combination of Xetex and Tex fonts?
> >> While there is a use case for LuaTeX and TeX fonts, I can't see a reason to
> >> use Xetex instead of pdflatex with TeX fonts!
> 
> > 316 tests are XeTex + system font, 52 (not inverted) of them fail, 42 
> > inverted
> > 316 tests are XeTex + tex font, 12 (not inverted) of them fail, 54 inverted
> 
> >> Solving all issues that arise from this combination is diverting attention
> >> and ressources from more important tasks. 
> 
> > Most of these tests were working some time ago. 
> 
> Many of them "by chance": not failing but with incorrect output (Missing
> characters, wrong characters) or "fragile" (working, but would fail by
> addition of one non-ASCII character, say).

Sure, but we have to check, ...
It is not done by simply declaring them to be inverted.

> > We already have many inverted test.
> 
> This also adds to the impression, that this is an area where test
> failures are to be expected for many reasons. I.e. the signal to noise ratio
> is rather bad for XeTeX+TeX fonts and we would be better of without these
> tests.
> 
> > I am strongly against such policy. First one has to check if the reason is 
> > really
> > babel/polyglossia conflict.
> 
> There are many more reasons, mhchem, babel language files, font problems, ...

Yes, and I am grateful too for what you have done in this direction.

> The combination XeTeX + TeX-fonts is not supported by many packages,
> including standard LaTeX packages like "inputenc"! OTOH, it is so obscure
> that it is really not worth wasting much time to work around all the
> problems. Rather treat/document it as: avoid if possible, use at your own
> risk.

My argument is not that they do not belong into the set of reverted, but I'd 
like them
to be checked first.
They will be included with the appropriate comment.

> Günter

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: status update on the export tests

[Guenter Milde]

On 2015-11-01, Kornel Benko wrote:
> Am Sonntag, 1. November 2015 um 19:37:41, schrieb Guenter Milde 
> 
>> On 2015-10-30, Kornel Benko wrote:

...

>> > Besides we have ATM about 200 failing export test cases.

...

>> How many of these are for the obscure combination of Xetex and Tex fonts?
>> While there is a use case for LuaTeX and TeX fonts, I can't see a reason to
>> use Xetex instead of pdflatex with TeX fonts!

> 316 tests are XeTex + system font, 52 (not inverted) of them fail, 42 inverted
> 316 tests are XeTex + tex font, 12 (not inverted) of them fail, 54 inverted

>> Solving all issues that arise from this combination is diverting attention
>> and ressources from more important tasks. 

> Most of these tests were working some time ago. 

Many of them "by chance": not failing but with incorrect output (Missing
characters, wrong characters) or "fragile" (working, but would fail by
addition of one non-ASCII character, say).

> We already have many inverted test.

This also adds to the impression, that this is an area where test
failures are to be expected for many reasons. I.e. the signal to noise ratio
is rather bad for XeTeX+TeX fonts and we would be better of without these
tests.

> I am strongly against such policy. First one has to check if the reason is 
> really
> babel/polyglossia conflict.

There are many more reasons, mhchem, babel language files, font problems, ...

The combination XeTeX + TeX-fonts is not supported by many packages,
including standard LaTeX packages like "inputenc"! OTOH, it is so obscure
that it is really not worth wasting much time to work around all the
problems. Rather treat/document it as: avoid if possible, use at your own
risk.


Günter

Re: status update on the export tests

[Kornel Benko]

Am Sonntag, 1. November 2015 um 21:27:11, schrieb Guenter Milde 

> On 2015-11-01, Georg Baum wrote:
> > Kornel Benko wrote:
> 
> >> Optimist (I mean 'usable state').
> >> I am strongly against such policy. First one has to check if the reason is
> >> really babel/polyglossia conflict.
> >> There are already too many tests inverted, no one cares anymore.
> 
> > We have a chicken and egg problem here. I started to work on the language 
> > nesting stuff, and fixing this without introducing regressions is 
> > impossible 
> > without creating specific test cases first. So, for this particular area of 
> > the code, the tests are already unusable, and we cannot get it into a good 
> > state again without usable tests...
> 
> > In principle I am with Kornel here, but in the current situation I think 
> > the 
> > only option we have is to set up more specific tests, fix the code one bug 
> > at a time, and then begin to look at the automatic export tests again. In 
> > the mean time, I don't care too much whether the failing tests are inverted 
> > or not.
> 
> Could we introduce a new status "suspended" - meaning just skip the test
> until a known bug is solved as we know the result is insignificant until then?
> 
> This way, known problems will not mask new problems.

We already have such (specified in file "ignoredTests"). But as this tests are 
never executed,
nobody cares for them anymore.
The tests here are such, that we know, we never resolve them.
Example:
We write a lyx file for odt/word/whatever output only. There is sense to expect
valid latex.

> Günter

Kornel



signature.asc
Description: This is a digitally signed message part.

Re: status update on the export tests

[Guenter Milde]

On 2015-11-01, Georg Baum wrote:
> Kornel Benko wrote:

>> Optimist (I mean 'usable state').
>> I am strongly against such policy. First one has to check if the reason is
>> really babel/polyglossia conflict.
>> There are already too many tests inverted, no one cares anymore.

> We have a chicken and egg problem here. I started to work on the language 
> nesting stuff, and fixing this without introducing regressions is impossible 
> without creating specific test cases first. So, for this particular area of 
> the code, the tests are already unusable, and we cannot get it into a good 
> state again without usable tests...

> In principle I am with Kornel here, but in the current situation I think the 
> only option we have is to set up more specific tests, fix the code one bug 
> at a time, and then begin to look at the automatic export tests again. In 
> the mean time, I don't care too much whether the failing tests are inverted 
> or not.

Could we introduce a new status "suspended" - meaning just skip the test
until a known bug is solved as we know the result is insignificant until then?

This way, known problems will not mask new problems.

Günter

Re: status update on the export tests

[Georg Baum]

Kornel Benko wrote:

> Am Sonntag, 1. November 2015 um 21:46:00, schrieb Georg Baum
> 
>> 
>> We have a chicken and egg problem here. I started to work on the language
>> nesting stuff, and fixing this without introducing regressions is
>> impossible without creating specific test cases first. So, for this
>> particular area of the code, the tests are already unusable, and we
>> cannot get it into a good state again without usable tests...
> 
> It is not only you. If our tests were usable at that time, then it would
> have been easy to check and set the appropriate tests in reverted drawer.

Definitely.

>> In principle I am with Kornel here, but in the current situation I think
>> the only option we have is to set up more specific tests, fix the code
>> one bug at a time, and then begin to look at the automatic export tests
>> again. In the mean time, I don't care too much whether the failing tests
>> are inverted or not.
> 
> 
> Does it mean 'delay 2.2 alpha'?

I don't think so. 2.1 has language nesting bugs as well, and the more simple 
cases do work. But we should probably mention that there are known language 
nesting bugs in the release notes. It is alpha after all (and I still hope 
to get some safe improvement for the beta).


Georg

Re: status update on the export tests

[Kornel Benko]

Am Sonntag, 1. November 2015 um 21:46:00, schrieb Georg Baum 

> Kornel Benko wrote:
> 
> > Optimist (I mean 'usable state').
> > I am strongly against such policy. First one has to check if the reason is
> > really babel/polyglossia conflict.
> > There are already too many tests inverted, no one cares anymore.
> 
> We have a chicken and egg problem here. I started to work on the language 
> nesting stuff, and fixing this without introducing regressions is impossible 
> without creating specific test cases first. So, for this particular area of 
> the code, the tests are already unusable, and we cannot get it into a good 
> state again without usable tests...

It is not only you. If our tests were usable at that time, then it would have 
been easy to
check and set the appropriate tests in reverted drawer.

> In principle I am with Kornel here, but in the current situation I think the 
> only option we have is to set up more specific tests, fix the code one bug 
> at a time, and then begin to look at the automatic export tests again. In 
> the mean time, I don't care too much whether the failing tests are inverted 
> or not.


Does it mean 'delay 2.2 alpha'?

> Georg

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: status update on the export tests

[Georg Baum]

Kornel Benko wrote:

> Optimist (I mean 'usable state').
> I am strongly against such policy. First one has to check if the reason is
> really babel/polyglossia conflict.
> There are already too many tests inverted, no one cares anymore.

We have a chicken and egg problem here. I started to work on the language 
nesting stuff, and fixing this without introducing regressions is impossible 
without creating specific test cases first. So, for this particular area of 
the code, the tests are already unusable, and we cannot get it into a good 
state again without usable tests...

In principle I am with Kornel here, but in the current situation I think the 
only option we have is to set up more specific tests, fix the code one bug 
at a time, and then begin to look at the automatic export tests again. In 
the mean time, I don't care too much whether the failing tests are inverted 
or not.


Georg

Re: status update on the export tests

[Kornel Benko]

Am Sonntag, 1. November 2015 um 15:00:25, schrieb Scott Kostyshak 

> On Sun, Nov 01, 2015 at 07:37:41PM +, Guenter Milde wrote:
> > On 2015-10-30, Kornel Benko wrote:
> > > Am Freitag, 30. Oktober 2015 um 19:23:02, schrieb Scott Kostyshak 
> > > 
> > > Besides we have ATM about 200 failing export test cases.
> > 
> > Too many to be helpfull.
> 
> I don't understand this logic.
> 
> > > The summary contains lines like:
> > >   Label Time Summary:
> > >   export  = 59316.83 sec (3753 tests)
> > >   key =   0.26 sec (1 test)
> > >   reverted= 5631.52 sec (312 tests)
> > 
> > > Even if we label some tests, the summary does not specify
> > > how mane tests went wrong for a specified label.
> > 
> > How many of these are for the obscure combination of Xetex and Tex fonts?
> > While there is a use case for LuaTeX and TeX fonts, I can't see a reason to
> > use Xetex instead of pdflatex with TeX fonts!
> 
> The use case that I have in mind is simply that I expect our users might
> use such a combination. If we don't want to support the combination, we
> should output a useful error message explaining why, or just disable the
> combination. If not, users will use it. Many LyX users do not have the
> understanding that you do. They do not even understand what it means to
> use non-TeX fonts. They might just read something about XeTeX and start
> using it by clicking on our button that says PDF (XeTeX). The purpose of
> the tests is just to prevent regressions. Sure, maybe it is true that
> something was working by luck before, but I still think it is reasonable
> to try to preserve it. So I defend the tests in general. The current
> situation is a special case. You fixed a bug (the missing characters
> bug) and that is why so many tests are failing. I do not imagine such a
> situation coming up often.
> 
> > Solving all issues that arise from this combination is diverting attention
> > and ressources from more important tasks.
> 
> Agreed.
> 
> > Also, we identified some generic problems with this combination that are
> > not solvable in the short term: third party packages well as documents
> > not prepared for this use case.
> > 
> > Just reverting failing Xetex export tests for the moment would allow us to
> > concentrfate on the remaining failing test and get the test suite in a
> > usable state again.
> 
> Fine with me. Kornel, OK with you? We could create a bug report so that
> we do not forget about the tests. Then we could just invert all of the
> pdf4_texF tests that are failing. I have in mind just pasting the long
> list of individual tests, rather than using regexes.

I have to be convinced. It is not OK with me, see my other post.
Remark to regexes:
1.) it is easier for us to read such a file
2.) but is takes longer time to interpret.

#ctest -j12 -R 'export/.*(pdf|dvi)' -N  | wc
3085
ATM, we have 59 lines to identify 300 inverted tests.
Adding the new 200 we have 500 individual lines.
That means we have to check (500/2) * 3085 combinations
to determine which test is inverted and which not.

ATM, we have 59 lines to identify 300 inverted tests.

> Scott

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: status update on the export tests

[Kornel Benko]

Am Sonntag, 1. November 2015 um 19:37:41, schrieb Guenter Milde 

> On 2015-10-30, Kornel Benko wrote:
> > Am Freitag, 30. Oktober 2015 um 19:23:02, schrieb Scott Kostyshak 
> > 
> >> On Fri, Oct 30, 2015 at 11:11:52PM +, Guenter Milde wrote:
> >> > On 2015-10-28, Scott Kostyshak wrote:
> >> > > On Mon, Oct 26, 2015 at 03:33:22PM +, Guenter Milde wrote:
> >> > 
> >> > ...
> >> > 
> >> > >> However, if the exit status of an export test changes (from fail
> >> > >> to pass or vice versa), we should check whether this is due to a
> >> > >> new bug, a fix or just exposing previously hidden problems.
> >> > 
> >> > > Agreed. In fact, sometimes it is an improvement when a test fails.
> >> > > When I check manually, it was passing before but showing garbled
> >> > > text in a PDF output. Now it might fail with a clear message of
> >> > > "language xyz not supported". It is always good to check manually
> >> > > why something fails and if it passes if the PDF output is good.
> >> > 
> >> > And, instead of re-establishing this with every status change, we
> >> > could have "tags" for inverted tests, distinguishing:
> >> > 
> >> > * failure because of known permanent incompatiblity (good failure)
> >> >   e.g. lyx -e latex ... for a document using non-TeX fonts
> >> > 
> >> > * failure because of ERT or preamble code (not LyX's fault)
> >> > 
> >> > * failure because of upstream bugs
> >> > 
> >> > * failure because of known LyX bugs
> >> > 
> >> > * failure for unknown reason with non-standard export route
> >> >   e.g. XeTeX and TeX-fonts
> 
> >> Yes this would be nice. Right now I just try to put that information
> >> as a comment for why we invert a test, but it would be nice to have
> >> that information more easily available in the test summary.
> 
> > I don't know how such an info can go to a summary.
> 
> We could have separate files for the tests that we revert for different
> reasons. Or use a set of keywords.
> 
> > Besides we have ATM about 200 failing export test cases.
> 
> Too many to be helpfull.
> 
> > The summary contains lines like:
> > Label Time Summary:
> > export  = 59316.83 sec (3753 tests)
> > key =   0.26 sec (1 test)
> > reverted= 5631.52 sec (312 tests)
> 
> > Even if we label some tests, the summary does not specify
> > how mane tests went wrong for a specified label.
> 
> How many of these are for the obscure combination of Xetex and Tex fonts?
> While there is a use case for LuaTeX and TeX fonts, I can't see a reason to
> use Xetex instead of pdflatex with TeX fonts!

316 tests are XeTex + system font, 52 (not inverted) of them fail, 42 inverted
316 tests are XeTex + tex font, 12 (not inverted) of them fail, 54 inverted
 
> Solving all issues that arise from this combination is diverting attention
> and ressources from more important tasks. 

Most of these tests were working some time ago. We already have many inverted 
test.
I am strongly against such policy. First one has to check if the reason is 
really
babel/polyglossia conflict.

> Also, we identified some generic problems with this combination that are
> not solvable in the short term: third party packages well as documents
> not prepared for this use case.
> 
> Just reverting failing Xetex export tests for the moment would allow us to
> concentrfate on the remaining failing test and get the test suite in a
> usable state again.

Optimist (I mean 'usable state').
I am strongly against such policy. First one has to check if the reason is 
really
babel/polyglossia conflict.
There are already too many tests inverted, no one cares anymore.

> Günter
> 

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: status update on the export tests

[Scott Kostyshak]

On Sun, Nov 01, 2015 at 07:37:41PM +, Guenter Milde wrote:
> On 2015-10-30, Kornel Benko wrote:
> > Am Freitag, 30. Oktober 2015 um 19:23:02, schrieb Scott Kostyshak 
> > 
> > Besides we have ATM about 200 failing export test cases.
> 
> Too many to be helpfull.

I don't understand this logic.

> > The summary contains lines like:
> > Label Time Summary:
> > export  = 59316.83 sec (3753 tests)
> > key =   0.26 sec (1 test)
> > reverted= 5631.52 sec (312 tests)
> 
> > Even if we label some tests, the summary does not specify
> > how mane tests went wrong for a specified label.
> 
> How many of these are for the obscure combination of Xetex and Tex fonts?
> While there is a use case for LuaTeX and TeX fonts, I can't see a reason to
> use Xetex instead of pdflatex with TeX fonts!

The use case that I have in mind is simply that I expect our users might
use such a combination. If we don't want to support the combination, we
should output a useful error message explaining why, or just disable the
combination. If not, users will use it. Many LyX users do not have the
understanding that you do. They do not even understand what it means to
use non-TeX fonts. They might just read something about XeTeX and start
using it by clicking on our button that says PDF (XeTeX). The purpose of
the tests is just to prevent regressions. Sure, maybe it is true that
something was working by luck before, but I still think it is reasonable
to try to preserve it. So I defend the tests in general. The current
situation is a special case. You fixed a bug (the missing characters
bug) and that is why so many tests are failing. I do not imagine such a
situation coming up often.

> Solving all issues that arise from this combination is diverting attention
> and ressources from more important tasks.

Agreed.

> Also, we identified some generic problems with this combination that are
> not solvable in the short term: third party packages well as documents
> not prepared for this use case.
> 
> Just reverting failing Xetex export tests for the moment would allow us to
> concentrfate on the remaining failing test and get the test suite in a
> usable state again.

Fine with me. Kornel, OK with you? We could create a bug report so that
we do not forget about the tests. Then we could just invert all of the
pdf4_texF tests that are failing. I have in mind just pasting the long
list of individual tests, rather than using regexes.

Scott

Re: status update on the export tests

[Guenter Milde]

On 2015-10-30, Kornel Benko wrote:
> Am Freitag, 30. Oktober 2015 um 19:23:02, schrieb Scott Kostyshak 
> 
>> On Fri, Oct 30, 2015 at 11:11:52PM +, Guenter Milde wrote:
>> > On 2015-10-28, Scott Kostyshak wrote:
>> > > On Mon, Oct 26, 2015 at 03:33:22PM +, Guenter Milde wrote:
>> > 
>> > ...
>> > 
>> > >> However, if the exit status of an export test changes (from fail
>> > >> to pass or vice versa), we should check whether this is due to a
>> > >> new bug, a fix or just exposing previously hidden problems.
>> > 
>> > > Agreed. In fact, sometimes it is an improvement when a test fails.
>> > > When I check manually, it was passing before but showing garbled
>> > > text in a PDF output. Now it might fail with a clear message of
>> > > "language xyz not supported". It is always good to check manually
>> > > why something fails and if it passes if the PDF output is good.
>> > 
>> > And, instead of re-establishing this with every status change, we
>> > could have "tags" for inverted tests, distinguishing:
>> > 
>> > * failure because of known permanent incompatiblity (good failure)
>> >   e.g. lyx -e latex ... for a document using non-TeX fonts
>> > 
>> > * failure because of ERT or preamble code (not LyX's fault)
>> > 
>> > * failure because of upstream bugs
>> > 
>> > * failure because of known LyX bugs
>> > 
>> > * failure for unknown reason with non-standard export route
>> >   e.g. XeTeX and TeX-fonts

>> Yes this would be nice. Right now I just try to put that information
>> as a comment for why we invert a test, but it would be nice to have
>> that information more easily available in the test summary.

> I don't know how such an info can go to a summary.

We could have separate files for the tests that we revert for different
reasons. Or use a set of keywords.

> Besides we have ATM about 200 failing export test cases.

Too many to be helpfull.

> The summary contains lines like:
>   Label Time Summary:
>   export  = 59316.83 sec (3753 tests)
>   key =   0.26 sec (1 test)
>   reverted= 5631.52 sec (312 tests)

> Even if we label some tests, the summary does not specify
> how mane tests went wrong for a specified label.

How many of these are for the obscure combination of Xetex and Tex fonts?
While there is a use case for LuaTeX and TeX fonts, I can't see a reason to
use Xetex instead of pdflatex with TeX fonts!

Solving all issues that arise from this combination is diverting attention
and ressources from more important tasks. 

Also, we identified some generic problems with this combination that are
not solvable in the short term: third party packages well as documents
not prepared for this use case.

Just reverting failing Xetex export tests for the moment would allow us to
concentrfate on the remaining failing test and get the test suite in a
usable state again.

Günter

Re: status update on the export tests

[Scott Kostyshak]

On Sat, Oct 31, 2015 at 12:14:39PM +0100, Jürgen Spitzmüller wrote:
> Am Mittwoch 28 Oktober 2015, 11:31:38 schrieb Scott Kostyshak:
> > > It may be that we need a new keyword for lib/languages, to tag
> > > languages where Babel is superior to Polyglossia also with non-TeX fonts.
> > > 
> > > Then, the "automatic" setting (which is also the default "default") could
> > > take into account that some languages should use Babel also with non-TeX
> > > fonts even if Polyglossia is available for them.
> > 
> > Jürgen what do you think about this idea?
> 
> Yes, this could work, but it is certainly not 2.2 stuff.

Sounds good. I created a ticket for milestone 2.3:
http://www.lyx.org/trac/ticket/9832
I don't think we would want to change between Polyglossia and Babel
during 2.2.x (except I suppose if one is completely incompatible with a
language).

Scott

Re: status update on the export tests

[Jürgen Spitzmüller]

Am Mittwoch 28 Oktober 2015, 11:31:38 schrieb Scott Kostyshak:
> > It may be that we need a new keyword for lib/languages, to tag
> > languages where Babel is superior to Polyglossia also with non-TeX fonts.
> > 
> > Then, the "automatic" setting (which is also the default "default") could
> > take into account that some languages should use Babel also with non-TeX
> > fonts even if Polyglossia is available for them.
> 
> Jürgen what do you think about this idea?

Yes, this could work, but it is certainly not 2.2 stuff.

Jürgen

Re: status update on the export tests

[Kornel Benko]

Am Freitag, 30. Oktober 2015 um 19:23:02, schrieb Scott Kostyshak 

> On Fri, Oct 30, 2015 at 11:11:52PM +, Guenter Milde wrote:
> > On 2015-10-28, Scott Kostyshak wrote:
> > > On Mon, Oct 26, 2015 at 03:33:22PM +, Guenter Milde wrote:
> > 
> > ...
> > 
> > >> However, if the exit status of an export test changes (from fail to
> > >> pass or vice versa), we should check whether this is due to a new bug,
> > >> a fix or just exposing previously hidden problems.
> > 
> > > Agreed. In fact, sometimes it is an improvement when a test fails. When
> > > I check manually, it was passing before but showing garbled text in a
> > > PDF output. Now it might fail with a clear message of "language xyz not
> > > supported". It is always good to check manually why something fails and
> > > if it passes if the PDF output is good.
> > 
> > And, instead of re-establishing this with every status change, we could
> > have "tags" for inverted tests, distinguishing:
> > 
> > * failure because of known permanent incompatiblity (good failure)
> >   e.g. lyx -e latex ... for a document using non-TeX fonts
> > 
> > * failure because of ERT or preamble code (not LyX's fault)
> > 
> > * failure because of upstream bugs
> > 
> > * failure because of known LyX bugs
> > 
> > * failure for unknown reason with non-standard export route
> >   e.g. XeTeX and TeX-fonts
> 
> Yes this would be nice. Right now I just try to put that information as
> a comment for why we invert a test, but it would be nice to have that
> information more easily available in the test summary.

I don't know how such an info can go to a summary.

Besides we have ATM about 200 failing export test cases.
The summary contains lines like:
Label Time Summary:
export  = 59316.83 sec (3753 tests)
key =   0.26 sec (1 test)
reverted= 5631.52 sec (312 tests)

Even if we label some tests, the summary does not specify
how mane tests went wrong for a specified label.

> Scott

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: status update on the export tests

[Scott Kostyshak]

On Fri, Oct 30, 2015 at 11:11:52PM +, Guenter Milde wrote:
> On 2015-10-28, Scott Kostyshak wrote:
> > On Mon, Oct 26, 2015 at 03:33:22PM +, Guenter Milde wrote:
> 
> ...
> 
> >> However, if the exit status of an export test changes (from fail to
> >> pass or vice versa), we should check whether this is due to a new bug,
> >> a fix or just exposing previously hidden problems.
> 
> > Agreed. In fact, sometimes it is an improvement when a test fails. When
> > I check manually, it was passing before but showing garbled text in a
> > PDF output. Now it might fail with a clear message of "language xyz not
> > supported". It is always good to check manually why something fails and
> > if it passes if the PDF output is good.
> 
> And, instead of re-establishing this with every status change, we could
> have "tags" for inverted tests, distinguishing:
> 
> * failure because of known permanent incompatiblity (good failure)
>   e.g. lyx -e latex ... for a document using non-TeX fonts
> 
> * failure because of ERT or preamble code (not LyX's fault)
> 
> * failure because of upstream bugs
> 
> * failure because of known LyX bugs
> 
> * failure for unknown reason with non-standard export route
>   e.g. XeTeX and TeX-fonts

Yes this would be nice. Right now I just try to put that information as
a comment for why we invert a test, but it would be nice to have that
information more easily available in the test summary.

Scott

Re: status update on the export tests

[Guenter Milde]

On 2015-10-28, Scott Kostyshak wrote:
> On Mon, Oct 26, 2015 at 03:33:22PM +, Guenter Milde wrote:

...

>> However, if the exit status of an export test changes (from fail to
>> pass or vice versa), we should check whether this is due to a new bug,
>> a fix or just exposing previously hidden problems.

> Agreed. In fact, sometimes it is an improvement when a test fails. When
> I check manually, it was passing before but showing garbled text in a
> PDF output. Now it might fail with a clear message of "language xyz not
> supported". It is always good to check manually why something fails and
> if it passes if the PDF output is good.

And, instead of re-establishing this with every status change, we could
have "tags" for inverted tests, distinguishing:

* failure because of known permanent incompatiblity (good failure)
  e.g. lyx -e latex ... for a document using non-TeX fonts

* failure because of ERT or preamble code (not LyX's fault)

* failure because of upstream bugs

* failure because of known LyX bugs

* failure for unknown reason with non-standard export route
  e.g. XeTeX and TeX-fonts
  

Günter  

Re: status update on the export tests

[Scott Kostyshak]

On Thu, Oct 29, 2015 at 10:36:57PM +0100, Georg Baum wrote:
> Unit tests are almost always worth the effort. When I implement one, it does 
> almost always find a bug right away. See also the check_lstrings failure 
> Vincent reported and http://www.lyx.org/trac/ticket/9799. Having these bugs 
> found by users is a much worse alternative.
> 
> The problem we have with unit tests is that we do not have a good 
> infrastructure: It is too much overhead to add one. But once this is 
> improved, it would be really worth it to add lots of unit tests.

Note that Vincent has already done some work on this. See:

https://www.mail-archive.com/search?l=mid&q=CAO7dr0gJu1hz%2Bmg7c1N74DH-bqiEAaY0tkS4wK%2BFUEEMmFBFSw%40mail.gmail.com

or for a quick link:
http://git.lyx.org/?p=developers/vfr/lyx.git;a=shortlog;h=refs/heads/tests

Scott

Re: status update on the export tests

[Georg Baum]

Scott Kostyshak wrote:

> On Mon, Oct 26, 2015 at 03:33:22PM +, Guenter Milde wrote:
>
>> I think we have to distinguish several test methods:
>> 
>> a) automatic export tests with lots of "real life" documents (manuals,
>>templates, examples)
>> 
>> b) functional tests: import/export of a complete test document and
>> comparing
>>it to the expected output
>>
>> c) unit tests: test of certain features or sub-units.
>> 
>> Unit tests (c) are currently not implemented, AFAIK.

We have some, but they are not really documented. See the test 
subdirectories in src/ and below.

> There has been some little talk of implementing them but nothing in the
> end was done. From what I understand, it would be a huge change and it
> wasn't clear whether it was worth the effort and if everyone would use
> the tests.

Unit tests are almost always worth the effort. When I implement one, it does 
almost always find a bug right away. See also the check_lstrings failure 
Vincent reported and http://www.lyx.org/trac/ticket/9799. Having these bugs 
found by users is a much worse alternative.

The problem we have with unit tests is that we do not have a good 
infrastructure: It is too much overhead to add one. But once this is 
improved, it would be really worth it to add lots of unit tests.


Georg

Re: status update on the export tests

[Scott Kostyshak]

On Tue, Oct 27, 2015 at 10:46:58AM +0100, Stephan Witt wrote:
> 
> Yes, I understood this. But for the first try I didn't want to run it in 
> parallel.
> I've observed that e.g. kpsewhich is used heavily while testing and one call 
> took ~10ms.
> Perhaps there is some tuning possible?

My guess is that there's a lot of tuning to be done. Kornel has already
improved a lot (thanks to him we can run tests in parallel) but I'm
guessing that you're right that some tweaks could improve things.

> But I'd guess, there is a significant overhead
> for the first time to generate all the required pieces like compiled fonts 
> etc.
> To check this now I'm starting the test again.

Ah interesting. I would be interested in whether the second run was
faster. I find that 11 hours is very slow even with just one core. If it
is expected to be slow on the first run, it would be nice to document
this (I'm currently working on a patch for Development.lyx).

> 1st question: wouldn't it be useful to split the tests into groups and
> enable the tests which should pass on a basic system automatically?
> How could this be done? I don't think it's good to have to enable the
> export tests explicitly.

Yes this would be nice. In addition to Kornel's point that it increases
configuration time, it increases the time it takes to run the tests,
which is understandably a reason why many do not run the tests.

> 2nd question: which tests should succeed with a "normal" TeX installation?

Would be good to know. If you give me an idea of what you mean by a
"normal" TeX installation I can tell you (I'll set up a virtual
machine). Would a full TeX Live installation be what you have in mind or
do you mean something more basic?

Scott

Re: status update on the export tests

[Scott Kostyshak]

On Mon, Oct 26, 2015 at 09:24:23AM +, Guenter Milde wrote:
> On 2015-10-26, Scott Kostyshak wrote:
> 
> > OK. I just want to make sure we define "there are no regressions". These
> > tests passed before edd37de8. So it is quite possible that documents
> > that users have that compile with LyX 2.1.x suddenly fail to compile
> > with LyX 2.2.0 because of edd37de8. In this sense, there is a
> > regression.
> 
> > So there is a trade-off: the bad side is that there is the possibility
> > that user's documents suddenly fail to compile. The good side is that we
> > now encourage the use of polyglossia by default which apparently has
> > advantages over babel. Jürgen knowingly made this decision and he knows
> > more than I do to be able to judge this tradeoff so I hesitantly accept
> > it.
> 
> Polyglossia has been the better (and sometimes only) choice for use with
> Xe/LuaTeX Unicode fonts for a long time, but Babel is catching up in the
> last years.

Good to know.

> It may be that we need a new keyword for lib/languages, to tag
> languages where Babel is superior to Polyglossia also with non-TeX fonts.
>
> Then, the "automatic" setting (which is also the default "default") could
> take into account that some languages should use Babel also with non-TeX
> fonts even if Polyglossia is available for them.

Jürgen what do you think about this idea?

> Could this prevent some of the regressions?

Yes, but possibly only because there are underlying bugs in our language
handling. Once those are fixed, I'm not sure if this would fix current
regressions or not. The following are the regressions (as indicated by
the tests) that I'm not sure will be fixed once LyX's language handling
is fixed. In other words, I'm not sure if these fail because we use
polyglossia incorrectly or because there is no way to use polyglossia
correctly in these situations:

export/examples/seminar_pdf5_systemF
export/examples/fr/seminar_pdf5_systemF
export/doc/fr/UserGuide_pdf5_systemF
export/examples/fr/Foils_pdf5_systemF
# kluwer gives the following error with polyglossia:
# LaTeX Error: Command \ifdef already defined.
export/templates/kluwer_pdf5_systemF

Regardless of fixing regressions, it still seems like a good idea though.

> (We need to look carefully, not
> only if the relevant documents compile without error, but also if the
> exported document is OK.)

Agreed.

Scott

Re: status update on the export tests

[Scott Kostyshak]

On Mon, Oct 26, 2015 at 03:33:22PM +, Guenter Milde wrote:
> On 2015-10-26, Liviu Andronic wrote:
> > On Mon, Oct 26, 2015 at 10:24 AM, Guenter Milde  wrote:
> >> On 2015-10-26, Scott Kostyshak wrote:
> 
> ...
> 
> >> Could this prevent some of the regressions? (We need to look carefully, not
> >> only if the relevant documents compile without error, but also if the
> >> exported document is OK.)
> 
> > By exported document do you mean .tex or .pdf? If it is .tex, would it
> > be a good idea to check whether the latest export is identical to a
> > reference .tex generated when creating the test; if not, display a
> > diff.

There are similar things Kornel and I have discussed. We would like to
have tests that do what you suggest for lyx2lyx. For example, we first
export to .tex, then we convert the .lyx file to an older version then
convert it back and export to .tex again and see if the files differ.
This is nice because it is quicker than compilation, and as you say it
might catch bugs that relying on exit code would not.

> > Simply relying on the exit code seems like an easy way to miss
> > non-error generating regressions...

Agreed. 

> I think we have to distinguish several test methods:
> 
> a) automatic export tests with lots of "real life" documents (manuals,
>templates, examples)
> 
> b) functional tests: import/export of a complete test document and comparing
>it to the expected output
>
> c) unit tests: test of certain features or sub-units.
> 
> Unit tests (c) are currently not implemented, AFAIK.

There has been some little talk of implementing them but nothing in the
end was done. From what I understand, it would be a huge change and it
wasn't clear whether it was worth the effort and if everyone would use
the tests.

> The tex2lyx tests are "functional tests" (b), where we keep track of the
> expected output. They include also export tests (in the "round trip" suite).
> Here, we have to manually check and update the "expected" output documents
> discriminating between intended changes and regressions/bug-indicators.
> 
> For a), it would be a pain to keep track of and update all the output
> documents, because this would not only be required for different export
> routines but also for changes in the input docs. However, if the exit
> status of a test changes (from fail to pass or vice versa), we should
> check whether this is due to a new bug, a fix or just exposing previously
> hidden problems.

Agreed. In fact, sometimes it is an improvement when a test fails. When
I check manually, it was passing before but showing garbled text in a
PDF output. Now it might fail with a clear message of "language xyz not
supported". It is always good to check manually why something fails and
if it passes if the PDF output is good.

Scott

Re: status update on the export tests

[Guenter Milde]

On 2015-10-27, Georg Baum wrote:
> Guenter Milde wrote:

>> A proper fix would be to use lib/unicodesymbols for the content of the
>> hypersetup.

> It has been discussed in the past to use lib/unicodesymbols replacements for 
> the preamble, but there was some important reason not to do it (which I 
> forgot unfortunately). Searching the list archives or trac should for 
> unicodesymbols and preamble should hopefully find the discussion.

OK. For XeTeX, this is not required, as the PDF-Info does not use TeX-fonts
and hence utf8 works fine.

For "latexencoding"="ascii", we need one of the tree options:

a) \usepackage[ascii]{inputenc}
   (and \inputencoding{utf8} for the PDF-info).
   
   +1 simple
   -1 breaks documents that clash with inputenc

b) use lib/unicodesymbols

   +1 no loading of "inputenc" package, as currently
   +1 support for more characters than \inputenc{utf8} with hyperref
  (e.g. polytonic Greek).
   -1 pdf strings expect a different input, 
  many lib/unicodesymbols replacements do not work!
  
c) tell the user to use LICR in the preamble, e.g.

  PDF title: Gru\ss aus Dresden
  
   +1 consistent with the idea: preamble is "raw" (kind of)
   +1 simple (for us)
   -1 fails for \" (Gr\"u\ss e) 
  (different bug: any ASCII-" terminates the string!)
  
Günter  

Re: status update on the export tests

[Georg Baum]

Guenter Milde wrote:

> A proper fix would be to use lib/unicodesymbols for the content of the
> hypersetup.

It has been discussed in the past to use lib/unicodesymbols replacements for 
the preamble, but there was some important reason not to do it (which I 
forgot unfortunately). Searching the list archives or trac should for 
unicodesymbols and preamble should hopefully find the discussion.


Georg

Re: status update on the export tests

[Guenter Milde]

On 2015-10-27, Stephan Witt wrote:
> Am 27.10.2015 um 13:42 schrieb Kornel Benko :

>> Am Dienstag, 27. Oktober 2015 um 13:29:35, schrieb Stephan Witt 
>> 
>>> One interesting example of a failure is as follows:

>>> ==
>>> 350/4621 Testing: export/doc/Math_pdf4_texF

>> It is simply not compilable with xelatex. At least here.


> Because of this:
> $ file lib/doc/Math.lyx
> lib/doc/Math.lyx: UTF-8 Unicode English text

> ...
> \pdf_author "LyX Team, Uwe Stöhr"
> ...

Mind, that this is also currently a problem with "normal" latex or pdflatex,
if you set the "latex encoding" to "ASCII".

The problem is, that the hypersetup is wrapped in:

  \inputencoding{utf8}
  ...
  \inputencoding{ascii}

which leads to;

! Undefined control sequence.
l.35 \inputencoding
   {utf8}

! LaTeX Error: Missing \begin{document}.

See the LaTeX manual or LaTeX Companion for explanation.
Type  H   for immediate help.
 ...  
  
l.35 \inputencoding{u
 tf8}


> This cannot be converted from UTF-8 to ASCII.

Of course it can:

\pdf_author "LyX Team, Uwe St\"ohr"

(Hyperref supports the LaTeX Internal Character Representation (LICR).)

If I export the "ascii"-version to latex, change as above and comment out
the  \inputencoding changes, the file compiles fine here.

A proper fix would be to use lib/unicodesymbols for the content of the
hypersetup.

Günter

Re: status update on the export tests

[Stephan Witt]

Am 27.10.2015 um 13:42 schrieb Kornel Benko :

> Am Dienstag, 27. Oktober 2015 um 13:29:35, schrieb Stephan Witt 
> 
>> One interesting example of a failure is as follows:
>> 
>> ==
>> 350/4621 Testing: export/doc/Math_pdf4_texF
> 
> It is simply not compilable with xelatex. At least here.


Because of this:
$ file lib/doc/Math.lyx
lib/doc/Math.lyx: UTF-8 Unicode English text

...
\pdf_author "LyX Team, Uwe Stöhr"
...

This cannot be converted from UTF-8 to ASCII.

Stephan

Re: status update on the export tests

[Kornel Benko]

Am Dienstag, 27. Oktober 2015 um 13:29:35, schrieb Stephan Witt 

> One interesting example of a failure is as follows:
> 
> ==
> 350/4621 Testing: export/doc/Math_pdf4_texF

It is simply not compilable with xelatex. At least here.

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: status update on the export tests

[Stephan Witt]

Am 27.10.2015 um 11:15 schrieb Kornel Benko :

> Am Dienstag, 27. Oktober 2015 um 10:46:58, schrieb Stephan Witt 
> 
>> Am 27.10.2015 um 10:13 schrieb Kornel Benko :
>> 
>>> Am Dienstag, 27. Oktober 2015 um 00:48:44, schrieb Stephan Witt 
>>> 
 Am 26.10.2015 um 12:18 schrieb Kornel Benko :
 
> Am Montag, 26. Oktober 2015 um 11:43:07, schrieb Stephan Witt 
> 
>> Am 26.10.2015 um 09:46 schrieb Kornel Benko :
>> 
>>> They are started, because otherwise you would not see 'out of 199'.
>>> What was your exact test command?
>> 
>> This is the command:
>> 
>> $ (cd /Users/stephan/git/lyx-build/cmake/2.2.0dev;ctest -C Debug)
> 
> Wow ... never used it this way ...
> But then I wonder, why only 199 tests ...
> Apparently you have not configured cmake for export tests (cmake ... 
> -DLYX_ENABLE_EXPORT_TESTS=ON …)
 
 Yes, this make a huge difference - now I've 4621 tests to run :)
>>> 
>>> :)
>>> 
 After 11 hours I'm at test 1405 of 4621… (running with one job at a time).
>> 
>> The result is:
>> 
>> 79% tests passed, 954 tests failed out of 4621
>> 
>> Label Time Summary:
>> cmplyx   = 214.12 sec
>> export   = 72802.49 sec
>> layout   =  94.51 sec
>> load = 700.28 sec
>> lyx2lyx  =  67.98 sec
>> module   =   9.83 sec
>> reverted = 19013.35 sec
>> roundtrip= 186.85 sec
>> 
>> Total Test time (real) = 74085.89 sec
>> 
>>> For that many export tests, I would use 'ctest -j8' (in case you have 
>>> multiple cores).
>>> I normally run 'ctest -j12 -R "export.*pdf"'. Having 8 cores makes the run 
>>> significantly faster.
>> 
>> Yes, I understood this. But for the first try I didn't want to run it in 
>> parallel.
>> I've observed that e.g. kpsewhich is used heavily while testing and one call 
>> took ~10ms.
>> Perhaps there is some tuning possible? But I'd guess, there is a significant 
>> overhead
>> for the first time to generate all the required pieces like compiled fonts 
>> etc.
>> To check this now I'm starting the test again.
>> 
>>> 
>>> Some tests may fail because of this parallel handling (I did not found why 
>>> yet), so I repeat
>>> the failed tests without the '-j12' param with the command 'ctest 
>>> --rerun-failed'.
>> 
>> 1st question: wouldn't it be useful to split the tests into groups and
>> enable the tests which should pass on a basic system automatically?
>> How could this be done? I don't think it's good to have to enable the
>> export tests explicitly.
> 
> Enabling the export tests increase the configuration time ..., so it felt 
> reasonable to set the
> default to OFF.
> And not 'many' developers tested at that time.
> 
>> 2nd question: which tests should succeed with a "normal" TeX installation?
> 
> 1.) Some export tests do not depend on TeX
>   "ctest -R 'export.*_(xhtml|lyx16)$'"
> 2.) Probably you want to test TeX without system fonts
>   "ctest -R 'export.*_texF$'"

With 

$ (cd /Users/stephan/git/lyx-build/cmake/2.2.0dev;ctest -C Debug -R 
'export/doc/.*_texF$' -j8)

I get this:

77% tests passed, 70 tests failed out of 302

Label Time Summary:
export  = 4782.73 sec
reverted= 538.92 sec

Total Test time (real) = 612.37 sec

One interesting example of a failure is as follows:

==
350/4621 Testing: export/doc/Math_pdf4_texF
350/4621 Test: export/doc/Math_pdf4_texF
Command: "/opt/local/bin/cmake" "-DLYX_ROOT=/Users/stephan/git/lyx/lib/doc" 
"-DLYX_TESTS_USERDIR=/Users/stephan/git/lyx-build/cmake/2.2.0dev/Testing/.lyx" 
"-Dlyx=/Users/stephan/g
it/lyx-build/cmake/2.2.0dev/bin/Debug/LyX" 
"-DWORKDIR=/Users/stephan/git/lyx-build/cmake/2.2.0dev/autotests/out-home" 
"-Dformat=pdf4" "-Dfonttype=texF" "-Dextension=pdf4" "-Dfile
=Math" "-Dreverted=0" "-DTOP_SRC_DIR=/Users/stephan/git/lyx" 
"-DPERL_EXECUTABLE=/opt/local/bin/perl" "-P" 
"/Users/stephan/git/lyx/development/autotests/export.cmake"
Directory: /Users/stephan/git/lyx-build/cmake/2.2.0dev/autotests/out-home
"export/doc/Math_pdf4_texF" start time: Oct 27 13:13 CET
Output:
--
-- using fonttype = texF
-- LYX_TESTS_USERDIR = /Users/stephan/git/lyx-build/cmake/2.2.0dev/Testing/.lyx
-- Converting with perl 
/Users/stephan/git/lyx/development/autotests/useSystemFonts.pl
-- Using source "/Users/stephan/git/lyx/lib/doc/Math.lyx"
-- Using dest 
"/Users/stephan/git/lyx-build/cmake/2.2.0dev/autotests/out-home/AbC_N1_hGi/Math_pdf4_texF.lyx"
Added (1) file "/Users/stephan/git/lyx/lib/doc/../images/math/phantom.png" to 
be copied to 
"/Users/stephan/git/lyx-build/cmake/2.2.0dev/autotests/out-home/AbC_N1_hGi/tmp_pdf4_main_Math/xx_phantom_wncljUBI.png"
Added (1) file "/Users/stephan/git/lyx/lib/doc/../images/math/vphantom.png" to 
be copied to 
"/Users/stephan/git/lyx-build/cmake/2.2.0dev/autotests/out-home/AbC_N1_hGi/tmp_pdf4_main_Math/xx_vphantom_j7I4wBMv.png"
Added (1) file "/Users/stephan/git/lyx/lib/doc/../images/math/hphantom.png" to 
be copied to 
"/Users/stephan

Re: status update on the export tests

[Kornel Benko]

Am Dienstag, 27. Oktober 2015 um 10:46:58, schrieb Stephan Witt 

> Am 27.10.2015 um 10:13 schrieb Kornel Benko :
> 
> > Am Dienstag, 27. Oktober 2015 um 00:48:44, schrieb Stephan Witt 
> > 
> >> Am 26.10.2015 um 12:18 schrieb Kornel Benko :
> >> 
> >>> Am Montag, 26. Oktober 2015 um 11:43:07, schrieb Stephan Witt 
> >>> 
>  Am 26.10.2015 um 09:46 schrieb Kornel Benko :
>  
> > They are started, because otherwise you would not see 'out of 199'.
> > What was your exact test command?
>  
>  This is the command:
>  
>  $ (cd /Users/stephan/git/lyx-build/cmake/2.2.0dev;ctest -C Debug)
> >>> 
> >>> Wow ... never used it this way ...
> >>> But then I wonder, why only 199 tests ...
> >>> Apparently you have not configured cmake for export tests (cmake ... 
> >>> -DLYX_ENABLE_EXPORT_TESTS=ON …)
> >> 
> >> Yes, this make a huge difference - now I've 4621 tests to run :)
> > 
> > :)
> > 
> >> After 11 hours I'm at test 1405 of 4621… (running with one job at a time).
> 
> The result is:
> 
> 79% tests passed, 954 tests failed out of 4621
> 
> Label Time Summary:
> cmplyx   = 214.12 sec
> export   = 72802.49 sec
> layout   =  94.51 sec
> load = 700.28 sec
> lyx2lyx  =  67.98 sec
> module   =   9.83 sec
> reverted = 19013.35 sec
> roundtrip= 186.85 sec
> 
> Total Test time (real) = 74085.89 sec
> 
> > For that many export tests, I would use 'ctest -j8' (in case you have 
> > multiple cores).
> > I normally run 'ctest -j12 -R "export.*pdf"'. Having 8 cores makes the run 
> > significantly faster.
> 
> Yes, I understood this. But for the first try I didn't want to run it in 
> parallel.
> I've observed that e.g. kpsewhich is used heavily while testing and one call 
> took ~10ms.
> Perhaps there is some tuning possible? But I'd guess, there is a significant 
> overhead
> for the first time to generate all the required pieces like compiled fonts 
> etc.
> To check this now I'm starting the test again.
> 
> > 
> > Some tests may fail because of this parallel handling (I did not found why 
> > yet), so I repeat
> > the failed tests without the '-j12' param with the command 'ctest 
> > --rerun-failed'.
> 
> 1st question: wouldn't it be useful to split the tests into groups and
> enable the tests which should pass on a basic system automatically?
> How could this be done? I don't think it's good to have to enable the
> export tests explicitly.

Enabling the export tests increase the configuration time ..., so it felt 
reasonable to set the
default to OFF.
And not 'many' developers tested at that time.

> 2nd question: which tests should succeed with a "normal" TeX installation?

1.) Some export tests do not depend on TeX
"ctest -R 'export.*_(xhtml|lyx16)$'"
2.) Probably you want to test TeX without system fonts
"ctest -R 'export.*_texF$'"

> Stephan

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: status update on the export tests

[Stephan Witt]

Am 27.10.2015 um 10:13 schrieb Kornel Benko :

> Am Dienstag, 27. Oktober 2015 um 00:48:44, schrieb Stephan Witt 
> 
>> Am 26.10.2015 um 12:18 schrieb Kornel Benko :
>> 
>>> Am Montag, 26. Oktober 2015 um 11:43:07, schrieb Stephan Witt 
>>> 
 Am 26.10.2015 um 09:46 schrieb Kornel Benko :
 
> They are started, because otherwise you would not see 'out of 199'.
> What was your exact test command?
 
 This is the command:
 
 $ (cd /Users/stephan/git/lyx-build/cmake/2.2.0dev;ctest -C Debug)
>>> 
>>> Wow ... never used it this way ...
>>> But then I wonder, why only 199 tests ...
>>> Apparently you have not configured cmake for export tests (cmake ... 
>>> -DLYX_ENABLE_EXPORT_TESTS=ON …)
>> 
>> Yes, this make a huge difference - now I've 4621 tests to run :)
> 
> :)
> 
>> After 11 hours I'm at test 1405 of 4621… (running with one job at a time).

The result is:

79% tests passed, 954 tests failed out of 4621

Label Time Summary:
cmplyx   = 214.12 sec
export   = 72802.49 sec
layout   =  94.51 sec
load = 700.28 sec
lyx2lyx  =  67.98 sec
module   =   9.83 sec
reverted = 19013.35 sec
roundtrip= 186.85 sec

Total Test time (real) = 74085.89 sec

> For that many export tests, I would use 'ctest -j8' (in case you have 
> multiple cores).
> I normally run 'ctest -j12 -R "export.*pdf"'. Having 8 cores makes the run 
> significantly faster.

Yes, I understood this. But for the first try I didn't want to run it in 
parallel.
I've observed that e.g. kpsewhich is used heavily while testing and one call 
took ~10ms.
Perhaps there is some tuning possible? But I'd guess, there is a significant 
overhead
for the first time to generate all the required pieces like compiled fonts etc.
To check this now I'm starting the test again.

> 
> Some tests may fail because of this parallel handling (I did not found why 
> yet), so I repeat
> the failed tests without the '-j12' param with the command 'ctest 
> --rerun-failed'.

1st question: wouldn't it be useful to split the tests into groups and
enable the tests which should pass on a basic system automatically?
How could this be done? I don't think it's good to have to enable the
export tests explicitly.

2nd question: which tests should succeed with a "normal" TeX installation?

Stephan

Re: status update on the export tests

[Kornel Benko]

Am Dienstag, 27. Oktober 2015 um 00:48:44, schrieb Stephan Witt 

> Am 26.10.2015 um 12:18 schrieb Kornel Benko :
> 
> > Am Montag, 26. Oktober 2015 um 11:43:07, schrieb Stephan Witt 
> > 
> >> Am 26.10.2015 um 09:46 schrieb Kornel Benko :
> >> 
> >>> They are started, because otherwise you would not see 'out of 199'.
> >>> What was your exact test command?
> >> 
> >> This is the command:
> >> 
> >> $ (cd /Users/stephan/git/lyx-build/cmake/2.2.0dev;ctest -C Debug)
> > 
> > Wow ... never used it this way ...
> > But then I wonder, why only 199 tests ...
> > Apparently you have not configured cmake for export tests (cmake ... 
> > -DLYX_ENABLE_EXPORT_TESTS=ON …)
> 
> Yes, this make a huge difference - now I've 4621 tests to run :)

:)

> After 11 hours I'm at test 1405 of 4621… (running with one job at a time).

For that many export tests, I would use 'ctest -j8' (in case you have multiple 
cores).
I normally run 'ctest -j12 -R "export.*pdf"'. Having 8 cores makes the run 
significantly faster.

Some tests may fail because of this parallel handling (I did not found why 
yet), so I repeat
the failed tests without the '-j12' param with the command 'ctest 
--rerun-failed'.

> Stephan

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: status update on the export tests

[Stephan Witt]

Am 26.10.2015 um 12:18 schrieb Kornel Benko :

> Am Montag, 26. Oktober 2015 um 11:43:07, schrieb Stephan Witt 
> 
>> Am 26.10.2015 um 09:46 schrieb Kornel Benko :
>> 
>>> They are started, because otherwise you would not see 'out of 199'.
>>> What was your exact test command?
>> 
>> This is the command:
>> 
>> $ (cd /Users/stephan/git/lyx-build/cmake/2.2.0dev;ctest -C Debug)
> 
> Wow ... never used it this way ...
> But then I wonder, why only 199 tests ...
> Apparently you have not configured cmake for export tests (cmake ... 
> -DLYX_ENABLE_EXPORT_TESTS=ON …)

Yes, this make a huge difference - now I've 4621 tests to run :)

After 11 hours I'm at test 1405 of 4621… (running with one job at a time).

Stephan

Re: status update on the export tests

[Georg Baum]

Scott Kostyshak wrote:

> OK. Do you agree that this is not a blocker for alpha?

Yes.


Georg

Re: status update on the export tests

[Scott Kostyshak]

On Mon, Oct 26, 2015 at 09:40:06PM +0100, Georg Baum wrote:
> Scott Kostyshak wrote:
> 
> > OK. I just want to make sure we define "there are no regressions". These
> > tests passed before edd37de8. So it is quite possible that documents
> > that users have that compile with LyX 2.1.x suddenly fail to compile
> > with LyX 2.2.0 because of edd37de8. In this sense, there is a
> > regression.
> 
> Sorry, then I mixed this up.

Well it is a confusing situation with the tests. I might make a new
ticket that makes it clear what the current state of this "regression"
is and target it for 2.2.0.

> This should of course be fixed for the release 
> IMO. I'll try to find out what is still missing.

OK. Do you agree that this is not a blocker for alpha?

Scott

Re: status update on the export tests

[Georg Baum]

Scott Kostyshak wrote:

> OK. I just want to make sure we define "there are no regressions". These
> tests passed before edd37de8. So it is quite possible that documents
> that users have that compile with LyX 2.1.x suddenly fail to compile
> with LyX 2.2.0 because of edd37de8. In this sense, there is a
> regression.

Sorry, then I mixed this up. This should of course be fixed for the release 
IMO. I'll try to find out what is still missing.


Georg

Re: status update on the export tests

[Guenter Milde]

On 2015-10-26, Liviu Andronic wrote:
> On Mon, Oct 26, 2015 at 10:24 AM, Guenter Milde  wrote:
>> On 2015-10-26, Scott Kostyshak wrote:

...

>> Could this prevent some of the regressions? (We need to look carefully, not
>> only if the relevant documents compile without error, but also if the
>> exported document is OK.)

> By exported document do you mean .tex or .pdf? If it is .tex, would it
> be a good idea to check whether the latest export is identical to a
> reference .tex generated when creating the test; if not, display a
> diff. Simply relying on the exit code seems like an easy way to miss
> non-error generating regressions...

I think we have to distinguish several test methods:

a) automatic export tests with lots of "real life" documents (manuals,
   templates, examples)

b) functional tests: import/export of a complete test document and comparing
   it to the expected output
   
c) unit tests: test of certain features or sub-units.

Unit tests (c) are currently not implemented, AFAIK.

The tex2lyx tests are "functional tests" (b), where we keep track of the
expected output. They include also export tests (in the "round trip" suite).
Here, we have to manually check and update the "expected" output documents
discriminating between intended changes and regressions/bug-indicators.

For a), it would be a pain to keep track of and update all the output
documents, because this would not only be required for different export
routines but also for changes in the input docs. However, if the exit
status of a test changes (from fail to pass or vice versa), we should
check whether this is due to a new bug, a fix or just exposing previously
hidden problems.

Günter

Re: status update on the export tests

[Kornel Benko]

Am Montag, 26. Oktober 2015 um 11:43:07, schrieb Stephan Witt 
> Am 26.10.2015 um 09:46 schrieb Kornel Benko :
> 
> > Am Montag, 26. Oktober 2015 um 09:13:44, schrieb Stephan Witt 
> > 
> > ...
> >>> These tests fails because of language switching done with polyglossia, 
> >>> but I'm not sure what the root problem is:
> >>> export/doc/fr/UserGuide_pdf5_systemF
> >>> export/examples/fr/Foils_pdf5_systemF
> >>> # kluwer gives the following error with polyglossia:
> >>> # LaTeX Error: Command \ifdef already defined.
> >>> export/templates/kluwer_pdf5_systemF
> >>> 
> >>> Please let me (or Kornel) know if you have any questions.
> >> 
> >> I tried to test these exports on my Mac and cannot see how it should be 
> >> triggered.

...

> >> On my system there is no X11-utility installed and LyX doesn't belong to 
> >> X11.
> >> Is this the reason why these tests are not started?
> >> 
> >> Stephan
> > 
> > They are started, because otherwise you would not see 'out of 199'.
> > What was your exact test command?
> 
> This is the command:
>  
> $ (cd /Users/stephan/git/lyx-build/cmake/2.2.0dev;ctest -C Debug)

Wow ... never used it this way ...
But then I wonder, why only 199 tests ...
Apparently you have not configured cmake for export tests (cmake ... 
-DLYX_ENABLE_EXPORT_TESTS=ON ...)

> > 
> > To see what went wrong, look in file "Testing/Temporary/LastTest.log".
> 
> 
> One example is:
> 
> 6/199 Testing: tex2lyx/roundtrip/test.ltx
> 6/199 Test: tex2lyx/roundtrip/test.ltx

...


> -- Error output of /Users/stephan/git/lyx/src/tex2lyx/test/runtests.py = 0
> 
> Test time =   2.79 sec
> --
> Test Passed.
> "tex2lyx/roundtrip/test.ltx" end time: Oct 26 11:05 CET
> "tex2lyx/roundtrip/test.ltx" time elapsed: 00:00:02
> --

Apparently this test passed ...

> Stephan

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: status update on the export tests

[Stephan Witt]

Am 26.10.2015 um 09:46 schrieb Kornel Benko :

> Am Montag, 26. Oktober 2015 um 09:13:44, schrieb Stephan Witt 
> 
> ...
>>> These tests fails because of language switching done with polyglossia, but 
>>> I'm not sure what the root problem is:
>>> export/doc/fr/UserGuide_pdf5_systemF
>>> export/examples/fr/Foils_pdf5_systemF
>>> # kluwer gives the following error with polyglossia:
>>> # LaTeX Error: Command \ifdef already defined.
>>> export/templates/kluwer_pdf5_systemF
>>> 
>>> Please let me (or Kornel) know if you have any questions.
>> 
>> I tried to test these exports on my Mac and cannot see how it should be 
>> triggered.
>> 
>> Below I'm listing the summary output:
>> ==
>> 100% tests passed, 0 tests failed out of 199
>> 
>> Label Time Summary:
>> cmplyx   = 120.88 sec
>> layout   = 107.01 sec
>> lyx2lyx  =   0.05 sec
>> module   =  12.17 sec
>> roundtrip=  76.86 sec
>> ==
>> 
>> On my system there is no X11-utility installed and LyX doesn't belong to X11.
>> Is this the reason why these tests are not started?
>> 
>> Stephan
> 
> They are started, because otherwise you would not see 'out of 199'.
> What was your exact test command?

This is the command:
 
$ (cd /Users/stephan/git/lyx-build/cmake/2.2.0dev;ctest -C Debug)

> 
> To see what went wrong, look in file "Testing/Temporary/LastTest.log".


One example is:

6/199 Testing: tex2lyx/roundtrip/test.ltx
6/199 Test: tex2lyx/roundtrip/test.ltx
Command: "/opt/local/bin/cmake" 
"-DLYX_TESTS_USERDIR=/Users/stephan/git/lyx-build/cmake/2.2.0dev/Testing/.lyx" 
"-DLYX_USERDIR_VER=LYX_USERDIR_22x" 
"-DLYX_PYTHON_EXECUTABLE=/opt/local/bin/python2.7" 
"-DPY_SCRIPT=/Users/stephan/git/lyx/src/tex2lyx/test/runtests.py" 
"-DTEX2LYX_EXE=/Users/stephan/git/lyx-build/cmake/2.2.0dev/bin/Debug/tex2lyx" 
"-DSCRIPT_DIR=/Users/stephan/git/lyx/lib/scripts" 
"-DWORKDIR=/Users/stephan/git/lyx-build/cmake/2.2.0dev/src/tex2lyx/test" 
"-DTESTFILE=test.ltx" "-P" 
"/Users/stephan/git/lyx/src/tex2lyx/test/runtests.cmake"
Directory: /Users/stephan/git/lyx-build/cmake/2.2.0dev/src/tex2lyx/test
"tex2lyx/roundtrip/test.ltx" start time: Oct 26 11:05 CET
Output:
--
-- SCRIPT_DIR = /Users/stephan/git/lyx/lib/scripts
-- Executing: /Users/stephan/git/lyx-build/cmake/2.2.0dev/bin/Debug/tex2lyx 
-roundtrip -copyfiles -f /Users/stephan/git/lyx/src/tex2lyx/test/test.ltx 
/Users/stephan/git/lyx-build/cmake/2.2.0dev/src/tex2lyx/test/test.lyx

Overwriting existing file 
/Users/stephan/git/lyx-build/cmake/2.2.0dev/src/tex2lyx/test/test.lyx
Error: New binding for 'Kontroll-T 1' is overriding old binding...
Error: New binding for 'Kontroll-T 2' is overriding old binding...
Error: New binding for 'Kontroll-T O' is overriding old binding...
Error: New binding for 'Kontroll-T T' is overriding old binding...
Error: New binding for 'Kontroll-T X' is overriding old binding...
Warning: Die Dokumentklasse ist nicht verfügbar

Die ausgewählte Dokumentklasse
Article (Standardklasse)
benötigt externe Dateien, welche nicht verfügbar sind.
Die Dokumentklasse kann zwar verwendet werden,
das Dokument kann aber nicht ausgegeben werden,
solange die folgenden Dateien nicht installiert sind:
article.cls
In Abschnitt 3.1.2.2 des Benutzerhandbuchs
finden Sie weitere Hilfe.


-- Error output of /Users/stephan/git/lyx/src/tex2lyx/test/runtests.py = 0

Test time =   2.79 sec
--
Test Passed.
"tex2lyx/roundtrip/test.ltx" end time: Oct 26 11:05 CET
"tex2lyx/roundtrip/test.ltx" time elapsed: 00:00:02
--

Stephan

Re: status update on the export tests

[Liviu Andronic]

On Mon, Oct 26, 2015 at 10:24 AM, Guenter Milde  wrote:
> On 2015-10-26, Scott Kostyshak wrote:
>
>> OK. I just want to make sure we define "there are no regressions". These
>> tests passed before edd37de8. So it is quite possible that documents
>> that users have that compile with LyX 2.1.x suddenly fail to compile
>> with LyX 2.2.0 because of edd37de8. In this sense, there is a
>> regression.
>
>> So there is a trade-off: the bad side is that there is the possibility
>> that user's documents suddenly fail to compile. The good side is that we
>> now encourage the use of polyglossia by default which apparently has
>> advantages over babel. Jürgen knowingly made this decision and he knows
>> more than I do to be able to judge this tradeoff so I hesitantly accept
>> it.
>
> Polyglossia has been the better (and sometimes only) choice for use with
> Xe/LuaTeX Unicode fonts for a long time, but Babel is catching up in the
> last years.
>
> It may be that we need a new keyword for lib/languages, to tag
> languages where Babel is superior to Polyglossia also with non-TeX fonts.
>
> Then, the "automatic" setting (which is also the default "default") could
> take into account that some languages should use Babel also with non-TeX
> fonts even if Polyglossia is available for them.
>


> Could this prevent some of the regressions? (We need to look carefully, not
> only if the relevant documents compile without error, but also if the
> exported document is OK.)
>
By exported document do you mean .tex or .pdf? If it is .tex, would it
be a good idea to check whether the latest export is identical to a
reference .tex generated when creating the test; if not, display a
diff. Simply relying on the exit code seems like an easy way to miss
non-error generating regressions...

Liviu


> Günter
>



-- 
Do you think you know what math is?
http://www.ideasroadshow.com/issues/ian-stewart-2013-08-02
Or what it means to be intelligent?
http://www.ideasroadshow.com/issues/john-duncan-2013-08-30
Think again:
http://www.ideasroadshow.com/library

Re: status update on the export tests

[Guenter Milde]

On 2015-10-26, Scott Kostyshak wrote:

> OK. I just want to make sure we define "there are no regressions". These
> tests passed before edd37de8. So it is quite possible that documents
> that users have that compile with LyX 2.1.x suddenly fail to compile
> with LyX 2.2.0 because of edd37de8. In this sense, there is a
> regression.

> So there is a trade-off: the bad side is that there is the possibility
> that user's documents suddenly fail to compile. The good side is that we
> now encourage the use of polyglossia by default which apparently has
> advantages over babel. Jürgen knowingly made this decision and he knows
> more than I do to be able to judge this tradeoff so I hesitantly accept
> it.

Polyglossia has been the better (and sometimes only) choice for use with
Xe/LuaTeX Unicode fonts for a long time, but Babel is catching up in the
last years.

It may be that we need a new keyword for lib/languages, to tag
languages where Babel is superior to Polyglossia also with non-TeX fonts.

Then, the "automatic" setting (which is also the default "default") could
take into account that some languages should use Babel also with non-TeX
fonts even if Polyglossia is available for them.

Could this prevent some of the regressions? (We need to look carefully, not
only if the relevant documents compile without error, but also if the
exported document is OK.)

Günter

Re: status update on the export tests

[Kornel Benko]

Am Montag, 26. Oktober 2015 um 09:13:44, schrieb Stephan Witt 
...
> > These tests fails because of language switching done with polyglossia, but 
> > I'm not sure what the root problem is:
> > export/doc/fr/UserGuide_pdf5_systemF
> > export/examples/fr/Foils_pdf5_systemF
> > # kluwer gives the following error with polyglossia:
> > # LaTeX Error: Command \ifdef already defined.
> > export/templates/kluwer_pdf5_systemF
> > 
> > Please let me (or Kornel) know if you have any questions.
> 
> I tried to test these exports on my Mac and cannot see how it should be 
> triggered.
> 
> Below I'm listing the summary output:
> ==
> 100% tests passed, 0 tests failed out of 199
> 
> Label Time Summary:
> cmplyx   = 120.88 sec
> layout   = 107.01 sec
> lyx2lyx  =   0.05 sec
> module   =  12.17 sec
> roundtrip=  76.86 sec
> ==
> 
> On my system there is no X11-utility installed and LyX doesn't belong to X11.
> Is this the reason why these tests are not started?
> 
> Stephan

They are started, because otherwise you would not see 'out of 199'.
What was your exact test command?

To see what went wrong, look in file "Testing/Temporary/LastTest.log".

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: status update on the export tests

[Stephan Witt]

Am 25.10.2015 um 05:29 schrieb Scott Kostyshak :

> Dear all,
> 
> There seems to be more interest in running the export tests. This is great and
> will encourage me to improve the (shamefully lacking) documentation for these
> tests.
> 
> First, I am focusing on the test results after reverting (locally) two 
> commits,
> as follows:
> 
>  git revert --no-edit 664ef2c4 1523fc60
> 
> If you do not revert these two commits, many tests will fail and it will be
> hard to figure out when there are regressions.
> 
> To run all the tests successfully, you need to have a lot of dependencies 
> (even
> if you have a full TeX Live installation). Thus, you might just want to limit
> your running of the tests to our manuals, and exclude the templates and
> examples. To do that, you can do something like: ctest -R "doc/"
> 
> If you are interested in whether a certain patch breaks anything, feel free to
> let me or Kornel know and we will do our best to run the tests before and 
> after
> your change and see if any test suggests a regression.
> 
> The following tests are currently expected to fail because of issues with
> nested language handling and polyglossia. Georg, if you still have interest 
> and
> time for fixing this type of issue, take a look at these:
> 
> export/doc/nb/Intro_pdf5_systemF
> export/doc/sk/Intro_pdf5_systemF
> export/doc/es/Customization_pdf5_systemF
> export/doc/de/Customization_pdf5_systemF
> export/doc/fr/Customization_pdf5_systemF
> 
> I'm not sure but these might also suffer from a similar problem:
> export/examples/seminar_pdf5_systemF
> export/examples/fr/seminar_pdf5_systemF
> 
> These tests fails because of language switching done with polyglossia, but 
> I'm not sure what the root problem is:
> export/doc/fr/UserGuide_pdf5_systemF
> export/examples/fr/Foils_pdf5_systemF
> # kluwer gives the following error with polyglossia:
> # LaTeX Error: Command \ifdef already defined.
> export/templates/kluwer_pdf5_systemF
> 
> Please let me (or Kornel) know if you have any questions.

I tried to test these exports on my Mac and cannot see how it should be 
triggered.

Below I'm listing the summary output:
==
100% tests passed, 0 tests failed out of 199

Label Time Summary:
cmplyx   = 120.88 sec
layout   = 107.01 sec
lyx2lyx  =   0.05 sec
module   =  12.17 sec
roundtrip=  76.86 sec
==

On my system there is no X11-utility installed and LyX doesn't belong to X11.
Is this the reason why these tests are not started?

Stephan

Re: status update on the export tests

[Scott Kostyshak]

On Sun, Oct 25, 2015 at 11:10:56AM +0100, Georg Baum wrote:
> Scott Kostyshak wrote:
> 
> > The following tests are currently expected to fail because of issues with
> > nested language handling and polyglossia. Georg, if you still have
> > interest and time for fixing this type of issue, take a look at these:
> > 
> > export/doc/nb/Intro_pdf5_systemF
> > export/doc/sk/Intro_pdf5_systemF
> > export/doc/es/Customization_pdf5_systemF
> > export/doc/de/Customization_pdf5_systemF
> > export/doc/fr/Customization_pdf5_systemF
> 
> I am still interested, but don't know when I'll have time to look at these, 
> so don't wait for me. I think the current status is much better than with 
> 2.1, there are no regressions, and that this the most important thing. If 
> we'll find a safe improvement before the 2.2.0 release, then this is fine, 
> but if we don't then it is not a problem.

OK. I just want to make sure we define "there are no regressions". These
tests passed before edd37de8. So it is quite possible that documents
that users have that compile with LyX 2.1.x suddenly fail to compile
with LyX 2.2.0 because of edd37de8. In this sense, there is a
regression.

So there is a trade-off: the bad side is that there is the possibility
that user's documents suddenly fail to compile. The good side is that we
now encourage the use of polyglossia by default which apparently has
advantages over babel. Jürgen knowingly made this decision and he knows
more than I do to be able to judge this tradeoff so I hesitantly accept
it.

Are we all on the same page?

Scott

Re: status update on the export tests

[Georg Baum]

Scott Kostyshak wrote:

> The following tests are currently expected to fail because of issues with
> nested language handling and polyglossia. Georg, if you still have
> interest and time for fixing this type of issue, take a look at these:
> 
> export/doc/nb/Intro_pdf5_systemF
> export/doc/sk/Intro_pdf5_systemF
> export/doc/es/Customization_pdf5_systemF
> export/doc/de/Customization_pdf5_systemF
> export/doc/fr/Customization_pdf5_systemF

I am still interested, but don't know when I'll have time to look at these, 
so don't wait for me. I think the current status is much better than with 
2.1, there are no regressions, and that this the most important thing. If 
we'll find a safe improvement before the 2.2.0 release, then this is fine, 
but if we don't then it is not a problem.


Georg

status update on the export tests

[Scott Kostyshak]

Dear all,

There seems to be more interest in running the export tests. This is great and
will encourage me to improve the (shamefully lacking) documentation for these
tests.

First, I am focusing on the test results after reverting (locally) two commits,
as follows:

  git revert --no-edit 664ef2c4 1523fc60

If you do not revert these two commits, many tests will fail and it will be
hard to figure out when there are regressions.

To run all the tests successfully, you need to have a lot of dependencies (even
if you have a full TeX Live installation). Thus, you might just want to limit
your running of the tests to our manuals, and exclude the templates and
examples. To do that, you can do something like: ctest -R "doc/"

If you are interested in whether a certain patch breaks anything, feel free to
let me or Kornel know and we will do our best to run the tests before and after
your change and see if any test suggests a regression.

The following tests are currently expected to fail because of issues with
nested language handling and polyglossia. Georg, if you still have interest and
time for fixing this type of issue, take a look at these:

export/doc/nb/Intro_pdf5_systemF
export/doc/sk/Intro_pdf5_systemF
export/doc/es/Customization_pdf5_systemF
export/doc/de/Customization_pdf5_systemF
export/doc/fr/Customization_pdf5_systemF

I'm not sure but these might also suffer from a similar problem:
export/examples/seminar_pdf5_systemF
export/examples/fr/seminar_pdf5_systemF

These tests fails because of language switching done with polyglossia, but I'm 
not sure what the root problem is:
export/doc/fr/UserGuide_pdf5_systemF
export/examples/fr/Foils_pdf5_systemF
# kluwer gives the following error with polyglossia:
# LaTeX Error: Command \ifdef already defined.
export/templates/kluwer_pdf5_systemF

Please let me (or Kornel) know if you have any questions.

Scott