Re: What is our goal for 2.2 regarding the export tests?

[Scott Kostyshak]

On Sat, Dec 19, 2015 at 04:33:42AM -0500, Scott Kostyshak wrote:
> On Sat, Dec 19, 2015 at 10:08:29AM +0100, Kornel Benko wrote:
> 
> > This is meant if we are checking for regression. Yes.
> > But it is better to also check for load, to catch also layout errors.
> > ctest -L 'export|load'
> 
> Good point. I will add this command to the checklist for release
> managers. I am eventually going to clean up my notes and make public a
> checklist for future release managers to follow.

For me the following test fails: 
INVERTED.EXAMPLES_export/examples/he/splash_pdf5_systemF

When I do the export manually, the PDF looks fine (although I do not
know Hebrew).

Günter, does the export succeed for you?

I wonder if this is because I am using an outdated TeX Live.

Scott


signature.asc
Description: PGP signature

Re: What is our goal for 2.2 regarding the export tests?

[Scott Kostyshak]

On Thu, Dec 17, 2015 at 07:44:39AM +0100, Kornel Benko wrote:

> Yesterday I had 0 failures with the command
>   #ctest -j12 -L export -E "xhtml|lyx16"
> (about 2800 tests)
> 
> Today, after the many changes, I don't know.

You do not see any of the UNRELIABLE or SUSPENDED ones failing?

On 9267f177 I get the following failures:

467:SUSPENDED.MANUALS.MISSINGPNG_export/doc/attic/eu_UserGuide_pdf3
468:SUSPENDED.MANUALS.MISSINGPNG_export/doc/attic/eu_UserGuide_pdf4_texF
469:SUSPENDED.MANUALS.MISSINGPNG_export/doc/attic/eu_UserGuide_pdf4_systemF
741:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/Additional_pdf4_texF
754:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/Customization_pdf4_texF
756:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/Customization_pdf5_texF
788:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/EmbeddedObjects_dvi3_texF
793:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/EmbeddedObjects_pdf4_texF
827:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/Math_dvi3_texF
866:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/UserGuide_dvi3_texF
871:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/UserGuide_pdf4_texF
1955:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/seminar_dvi
1958:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/seminar_pdf
1959:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/seminar_pdf2
1960:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/seminar_pdf3
1961:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/seminar_pdf4_texF
1962:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/seminar_pdf4_systemF
2869:UNRELIABLE.EXAMPLES.TODO.WRONG.OUTPUT_export/examples/es/linguistics_pdf4_texF
2950:UNRELIABLE.EXAMPLES.NONSTANDARD_export/examples/fa/splash_dvi3_texF
2951:UNRELIABLE.EXAMPLES.NONSTANDARD_export/examples/fa/splash_dvi3_systemF
2955:UNRELIABLE.EXAMPLES.NONSTANDARD_export/examples/fa/splash_pdf4_texF
2956:UNRELIABLE.EXAMPLES.NONSTANDARD_export/examples/fa/splash_pdf4_systemF
2957:UNRELIABLE.EXAMPLES.NONSTANDARD_export/examples/fa/splash_pdf5_texF
2958:UNRELIABLE.EXAMPLES.NONSTANDARD_export/examples/fa/splash_pdf5_systemF
3093:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/fr/seminar_dvi
3096:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/fr/seminar_pdf
3097:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/fr/seminar_pdf2
3098:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/fr/seminar_pdf3
3099:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/fr/seminar_pdf4_texF
3100:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/fr/seminar_pdf4_systemF
3219:INVERTED.EXAMPLES_export/examples/he/splash_pdf5_systemF
3771:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_dvi
3772:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_dvi3_texF
3773:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_dvi3_systemF
3774:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf
3775:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf2
3776:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf3
3777:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf4_texF
3778:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf4_systemF
3779:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf5_texF
3780:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf5_systemF
3908:UNRELIABLE.NONSTANDARD_export/templates/ectaart_dvi3_texF
3909:UNRELIABLE.NONSTANDARD_export/templates/ectaart_dvi3_systemF
3915:UNRELIABLE.NONSTANDARD_export/templates/ectaart_pdf5_texF
3916:UNRELIABLE.NONSTANDARD_export/templates/ectaart_pdf5_systemF
3939:UNRELIABLE.NONSTANDARD_export/templates/es_beamer-conference-ornate-20min_pdf4_texF
4025:UNRELIABLE.NONSTANDARD_export/templates/ja_beamer-conference-ornate-20min_pdf2
4027:UNRELIABLE.NONSTANDARD_export/templates/ja_beamer-conference-ornate-20min_pdf4_texF
4028:UNRELIABLE.NONSTANDARD_export/templates/ja_beamer-conference-ornate-20min_pdf4_systemF
4029:UNRELIABLE.NONSTANDARD_export/templates/ja_beamer-conference-ornate-20min_pdf5_texF
4030:UNRELIABLE.NONSTANDARD_export/templates/ja_beamer-conference-ornate-20min_pdf5_systemF
4041:UNRELIABLE.NONSTANDARD_export/templates/kluwer_pdf4_systemF
4043:UNRELIABLE.NONSTANDARD_export/templates/kluwer_pdf5_systemF
4239:check_accessible_urls
4242:check_translators

Scott


signature.asc
Description: PGP signature

Re: What is our goal for 2.2 regarding the export tests?

[Scott Kostyshak]

On Sat, Dec 19, 2015 at 12:18:02AM -0800, Kornel Benko wrote:
> Am Samstag, 19. Dezember 2015 um 03:05:35, schrieb Scott Kostyshak 
> 
> > On Thu, Dec 17, 2015 at 07:44:39AM +0100, Kornel Benko wrote:
> > 
> > > Yesterday I had 0 failures with the command
> > >   #ctest -j12 -L export -E "xhtml|lyx16"
> > > (about 2800 tests)
> > > 
> > > Today, after the many changes, I don't know.
> > 
> > You do not see any of the UNRELIABLE or SUSPENDED ones failing?
> 
> Because they are _not_ part of '-L export'

Ah I see.

> Unreliable are such, that the outcome may change with local configuration. My 
> tests are not expected to have the same
> output as yours.

OK.

So when I want to do a final check for the 2.2.0 release, the command I
should run is

  ctest -L export

Is that reasonable?

Scott


signature.asc
Description: PGP signature

Re: What is our goal for 2.2 regarding the export tests?

[Kornel Benko]

Am Samstag, 19. Dezember 2015 um 03:46:34, schrieb Scott Kostyshak 

> On Sat, Dec 19, 2015 at 12:18:02AM -0800, Kornel Benko wrote:
> > Am Samstag, 19. Dezember 2015 um 03:05:35, schrieb Scott Kostyshak 
> > 
> > > On Thu, Dec 17, 2015 at 07:44:39AM +0100, Kornel Benko wrote:
> > > 
> > > > Yesterday I had 0 failures with the command
> > > > #ctest -j12 -L export -E "xhtml|lyx16"
> > > > (about 2800 tests)
> > > > 
> > > > Today, after the many changes, I don't know.
> > > 
> > > You do not see any of the UNRELIABLE or SUSPENDED ones failing?
> > 
> > Because they are _not_ part of '-L export'
> 
> Ah I see.
> 
> > Unreliable are such, that the outcome may change with local configuration. 
> > My tests are not expected to have the same
> > output as yours.
> 
> OK.
> 
> So when I want to do a final check for the 2.2.0 release, the command I
> should run is
> 
>   ctest -L export
> 
> Is that reasonable?

This is meant if we are checking for regression. Yes.
But it is better to also check for load, to catch also layout errors.
ctest -L 'export|load'

> Scott

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: What is our goal for 2.2 regarding the export tests?

[Kornel Benko]

Am Samstag, 19. Dezember 2015 um 03:05:35, schrieb Scott Kostyshak 

> On Thu, Dec 17, 2015 at 07:44:39AM +0100, Kornel Benko wrote:
> 
> > Yesterday I had 0 failures with the command
> > #ctest -j12 -L export -E "xhtml|lyx16"
> > (about 2800 tests)
> > 
> > Today, after the many changes, I don't know.
> 
> You do not see any of the UNRELIABLE or SUSPENDED ones failing?

Because they are _not_ part of '-L export'

> On 9267f177 I get the following failures:
> 
> 467:SUSPENDED.MANUALS.MISSINGPNG_export/doc/attic/eu_UserGuide_pdf3
> 468:SUSPENDED.MANUALS.MISSINGPNG_export/doc/attic/eu_UserGuide_pdf4_texF
> 469:SUSPENDED.MANUALS.MISSINGPNG_export/doc/attic/eu_UserGuide_pdf4_systemF
> 741:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/Additional_pdf4_texF
> 754:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/Customization_pdf4_texF
> 756:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/Customization_pdf5_texF
> 788:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/EmbeddedObjects_dvi3_texF
> 793:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/EmbeddedObjects_pdf4_texF
> 827:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/Math_dvi3_texF
> 866:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/UserGuide_dvi3_texF
> 871:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/UserGuide_pdf4_texF
> 1955:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/seminar_dvi
> 1958:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/seminar_pdf
> 1959:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/seminar_pdf2
> 1960:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/seminar_pdf3
> 1961:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/seminar_pdf4_texF
> 1962:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/seminar_pdf4_systemF
> 2869:UNRELIABLE.EXAMPLES.TODO.WRONG.OUTPUT_export/examples/es/linguistics_pdf4_texF
> 2950:UNRELIABLE.EXAMPLES.NONSTANDARD_export/examples/fa/splash_dvi3_texF
> 2951:UNRELIABLE.EXAMPLES.NONSTANDARD_export/examples/fa/splash_dvi3_systemF
> 2955:UNRELIABLE.EXAMPLES.NONSTANDARD_export/examples/fa/splash_pdf4_texF
> 2956:UNRELIABLE.EXAMPLES.NONSTANDARD_export/examples/fa/splash_pdf4_systemF
> 2957:UNRELIABLE.EXAMPLES.NONSTANDARD_export/examples/fa/splash_pdf5_texF
> 2958:UNRELIABLE.EXAMPLES.NONSTANDARD_export/examples/fa/splash_pdf5_systemF
> 3093:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/fr/seminar_dvi
> 3096:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/fr/seminar_pdf
> 3097:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/fr/seminar_pdf2
> 3098:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/fr/seminar_pdf3
> 3099:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/fr/seminar_pdf4_texF
> 3100:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/fr/seminar_pdf4_systemF
> 3219:INVERTED.EXAMPLES_export/examples/he/splash_pdf5_systemF
> 3771:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_dvi
> 3772:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_dvi3_texF
> 3773:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_dvi3_systemF
> 3774:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf
> 3775:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf2
> 3776:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf3
> 3777:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf4_texF
> 3778:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf4_systemF
> 3779:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf5_texF
> 3780:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf5_systemF
> 3908:UNRELIABLE.NONSTANDARD_export/templates/ectaart_dvi3_texF
> 3909:UNRELIABLE.NONSTANDARD_export/templates/ectaart_dvi3_systemF
> 3915:UNRELIABLE.NONSTANDARD_export/templates/ectaart_pdf5_texF
> 3916:UNRELIABLE.NONSTANDARD_export/templates/ectaart_pdf5_systemF
> 3939:UNRELIABLE.NONSTANDARD_export/templates/es_beamer-conference-ornate-20min_pdf4_texF
> 4025:UNRELIABLE.NONSTANDARD_export/templates/ja_beamer-conference-ornate-20min_pdf2
> 4027:UNRELIABLE.NONSTANDARD_export/templates/ja_beamer-conference-ornate-20min_pdf4_texF
> 4028:UNRELIABLE.NONSTANDARD_export/templates/ja_beamer-conference-ornate-20min_pdf4_systemF
> 4029:UNRELIABLE.NONSTANDARD_export/templates/ja_beamer-conference-ornate-20min_pdf5_texF
> 4030:UNRELIABLE.NONSTANDARD_export/templates/ja_beamer-conference-ornate-20min_pdf5_systemF
> 4041:UNRELIABLE.NONSTANDARD_export/templates/kluwer_pdf4_systemF
> 4043:UNRELIABLE.NONSTANDARD_export/templates/kluwer_pdf5_systemF
> 4239:check_accessible_urls
> 4242:check_translators

> Scott

Unreliable are such, that the outcome may change with local configuration. My 
tests are not expected to have the same
output as yours.

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: What is our goal for 2.2 regarding the export tests?

[Scott Kostyshak]

On Sat, Dec 19, 2015 at 10:08:29AM +0100, Kornel Benko wrote:

> This is meant if we are checking for regression. Yes.
> But it is better to also check for load, to catch also layout errors.
>   ctest -L 'export|load'

Good point. I will add this command to the checklist for release
managers. I am eventually going to clean up my notes and make public a
checklist for future release managers to follow.

Scott


signature.asc
Description: PGP signature

Re: What is our goal for 2.2 regarding the export tests?

[Guenter Milde]

On 2015-12-17, Kornel Benko wrote:
> Am Donnerstag, 17. Dezember 2015 um 01:25:35, schrieb Scott Kostyshak 
> 
>> A big thanks to Kornel and Günter for all of their work on the export
>> tests. These will catch regressions and make us more confident in our
>> releases going forward.

...

>> What is a reasonable expectation of the output of 'ctest' with the
>> following?

>>   -DLYX_ENABLE_EXPORT_TESTS=ON

> Yesterday I had 0 failures with the command
>   #ctest -j12 -L export -E "xhtml|lyx16"
> (about 2800 tests)

> Today, after the many changes, I don't know.
These are now reverted.

Generally, as it is quite easy to invert a test or mark it as unreliable,
the goal should be 0 test-failures.

No test-failures means that all export-failures are recognized as one of

* correct fail (e.g. TeX limitations prevent export to a certain format)

* tolerable (e.g. failures for "exotic" documents with non-default output
  format)
  
* unreliable (e.g. failures due to non-standard requirements, failures
  depending on local configuration, "erratic" behaviour (dependong on phase
  of the moon)
  
* TODO item (e.g. LyX bugs that wait for someone to fix them or
  test-failures that still need proper categorization)

It does not mean that there are no bugs or errors.

Günter

What is our goal for 2.2 regarding the export tests?

[Scott Kostyshak]

A big thanks to Kornel and Günter for all of their work on the export
tests. These will catch regressions and make us more confident in our
releases going forward.

I have lost track of some of the newest changes regarding the tests.
When I run the export tests with our final 2.2.0, what will the expected
output be for all of the different configuration options? For example,
with

  -DLYX_ENABLE_URLTESTS=ON

I think it unfortunately will be expected for 'check_accessible_urls'
to fail because of a false positive that we cannot get rid of.

Several keytests are also expected to fail because our machinery for
that is not working.

What is a reasonable expectation of the output of 'ctest' with the
following?

  -DLYX_ENABLE_EXPORT_TESTS=ON

Scott


signature.asc
Description: PGP signature

Re: What is our goal for 2.2 regarding the export tests?

[Kornel Benko]

Am Donnerstag, 17. Dezember 2015 um 01:25:35, schrieb Scott Kostyshak 

> A big thanks to Kornel and Günter for all of their work on the export
> tests. These will catch regressions and make us more confident in our
> releases going forward.
> 
> I have lost track of some of the newest changes regarding the tests.
> When I run the export tests with our final 2.2.0, what will the expected
> output be for all of the different configuration options? For example,
> with
> 
>   -DLYX_ENABLE_URLTESTS=ON
> 
> I think it unfortunately will be expected for 'check_accessible_urls'
> to fail because of a false positive that we cannot get rid of.

Yes, this depends unfortunately on moon.

> Several keytests are also expected to fail because our machinery for
> that is not working.

Should not ..., but changes in shortcuts often need also adaptations
to the tests. This is not always done.

> What is a reasonable expectation of the output of 'ctest' with the
> following?
> 
>   -DLYX_ENABLE_EXPORT_TESTS=ON

Yesterday I had 0 failures with the command
#ctest -j12 -L export -E "xhtml|lyx16"
(about 2800 tests)

Today, after the many changes, I don't know.

> Scott

Kornel

signature.asc
Description: This is a digitally signed message part.