Re: What is our goal for 2.2 regarding the export tests?
On Sat, Dec 19, 2015 at 04:33:42AM -0500, Scott Kostyshak wrote: > On Sat, Dec 19, 2015 at 10:08:29AM +0100, Kornel Benko wrote: > > > This is meant if we are checking for regression. Yes. > > But it is better to also check for load, to catch also layout errors. > > ctest -L 'export|load' > > Good point. I will add this command to the checklist for release > managers. I am eventually going to clean up my notes and make public a > checklist for future release managers to follow. For me the following test fails: INVERTED.EXAMPLES_export/examples/he/splash_pdf5_systemF When I do the export manually, the PDF looks fine (although I do not know Hebrew). Günter, does the export succeed for you? I wonder if this is because I am using an outdated TeX Live. Scott signature.asc Description: PGP signature
Re: What is our goal for 2.2 regarding the export tests?
On Thu, Dec 17, 2015 at 07:44:39AM +0100, Kornel Benko wrote: > Yesterday I had 0 failures with the command > #ctest -j12 -L export -E "xhtml|lyx16" > (about 2800 tests) > > Today, after the many changes, I don't know. You do not see any of the UNRELIABLE or SUSPENDED ones failing? On 9267f177 I get the following failures: 467:SUSPENDED.MANUALS.MISSINGPNG_export/doc/attic/eu_UserGuide_pdf3 468:SUSPENDED.MANUALS.MISSINGPNG_export/doc/attic/eu_UserGuide_pdf4_texF 469:SUSPENDED.MANUALS.MISSINGPNG_export/doc/attic/eu_UserGuide_pdf4_systemF 741:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/Additional_pdf4_texF 754:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/Customization_pdf4_texF 756:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/Customization_pdf5_texF 788:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/EmbeddedObjects_dvi3_texF 793:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/EmbeddedObjects_pdf4_texF 827:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/Math_dvi3_texF 866:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/UserGuide_dvi3_texF 871:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/UserGuide_pdf4_texF 1955:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/seminar_dvi 1958:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/seminar_pdf 1959:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/seminar_pdf2 1960:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/seminar_pdf3 1961:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/seminar_pdf4_texF 1962:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/seminar_pdf4_systemF 2869:UNRELIABLE.EXAMPLES.TODO.WRONG.OUTPUT_export/examples/es/linguistics_pdf4_texF 2950:UNRELIABLE.EXAMPLES.NONSTANDARD_export/examples/fa/splash_dvi3_texF 2951:UNRELIABLE.EXAMPLES.NONSTANDARD_export/examples/fa/splash_dvi3_systemF 2955:UNRELIABLE.EXAMPLES.NONSTANDARD_export/examples/fa/splash_pdf4_texF 2956:UNRELIABLE.EXAMPLES.NONSTANDARD_export/examples/fa/splash_pdf4_systemF 2957:UNRELIABLE.EXAMPLES.NONSTANDARD_export/examples/fa/splash_pdf5_texF 2958:UNRELIABLE.EXAMPLES.NONSTANDARD_export/examples/fa/splash_pdf5_systemF 3093:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/fr/seminar_dvi 3096:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/fr/seminar_pdf 3097:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/fr/seminar_pdf2 3098:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/fr/seminar_pdf3 3099:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/fr/seminar_pdf4_texF 3100:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/fr/seminar_pdf4_systemF 3219:INVERTED.EXAMPLES_export/examples/he/splash_pdf5_systemF 3771:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_dvi 3772:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_dvi3_texF 3773:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_dvi3_systemF 3774:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf 3775:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf2 3776:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf3 3777:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf4_texF 3778:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf4_systemF 3779:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf5_texF 3780:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf5_systemF 3908:UNRELIABLE.NONSTANDARD_export/templates/ectaart_dvi3_texF 3909:UNRELIABLE.NONSTANDARD_export/templates/ectaart_dvi3_systemF 3915:UNRELIABLE.NONSTANDARD_export/templates/ectaart_pdf5_texF 3916:UNRELIABLE.NONSTANDARD_export/templates/ectaart_pdf5_systemF 3939:UNRELIABLE.NONSTANDARD_export/templates/es_beamer-conference-ornate-20min_pdf4_texF 4025:UNRELIABLE.NONSTANDARD_export/templates/ja_beamer-conference-ornate-20min_pdf2 4027:UNRELIABLE.NONSTANDARD_export/templates/ja_beamer-conference-ornate-20min_pdf4_texF 4028:UNRELIABLE.NONSTANDARD_export/templates/ja_beamer-conference-ornate-20min_pdf4_systemF 4029:UNRELIABLE.NONSTANDARD_export/templates/ja_beamer-conference-ornate-20min_pdf5_texF 4030:UNRELIABLE.NONSTANDARD_export/templates/ja_beamer-conference-ornate-20min_pdf5_systemF 4041:UNRELIABLE.NONSTANDARD_export/templates/kluwer_pdf4_systemF 4043:UNRELIABLE.NONSTANDARD_export/templates/kluwer_pdf5_systemF 4239:check_accessible_urls 4242:check_translators Scott signature.asc Description: PGP signature
Re: What is our goal for 2.2 regarding the export tests?
On Sat, Dec 19, 2015 at 12:18:02AM -0800, Kornel Benko wrote: > Am Samstag, 19. Dezember 2015 um 03:05:35, schrieb Scott Kostyshak >> > On Thu, Dec 17, 2015 at 07:44:39AM +0100, Kornel Benko wrote: > > > > > Yesterday I had 0 failures with the command > > > #ctest -j12 -L export -E "xhtml|lyx16" > > > (about 2800 tests) > > > > > > Today, after the many changes, I don't know. > > > > You do not see any of the UNRELIABLE or SUSPENDED ones failing? > > Because they are _not_ part of '-L export' Ah I see. > Unreliable are such, that the outcome may change with local configuration. My > tests are not expected to have the same > output as yours. OK. So when I want to do a final check for the 2.2.0 release, the command I should run is ctest -L export Is that reasonable? Scott signature.asc Description: PGP signature
Re: What is our goal for 2.2 regarding the export tests?
Am Samstag, 19. Dezember 2015 um 03:46:34, schrieb Scott Kostyshak> On Sat, Dec 19, 2015 at 12:18:02AM -0800, Kornel Benko wrote: > > Am Samstag, 19. Dezember 2015 um 03:05:35, schrieb Scott Kostyshak > > > > > On Thu, Dec 17, 2015 at 07:44:39AM +0100, Kornel Benko wrote: > > > > > > > Yesterday I had 0 failures with the command > > > > #ctest -j12 -L export -E "xhtml|lyx16" > > > > (about 2800 tests) > > > > > > > > Today, after the many changes, I don't know. > > > > > > You do not see any of the UNRELIABLE or SUSPENDED ones failing? > > > > Because they are _not_ part of '-L export' > > Ah I see. > > > Unreliable are such, that the outcome may change with local configuration. > > My tests are not expected to have the same > > output as yours. > > OK. > > So when I want to do a final check for the 2.2.0 release, the command I > should run is > > ctest -L export > > Is that reasonable? This is meant if we are checking for regression. Yes. But it is better to also check for load, to catch also layout errors. ctest -L 'export|load' > Scott Kornel signature.asc Description: This is a digitally signed message part.
Re: What is our goal for 2.2 regarding the export tests?
Am Samstag, 19. Dezember 2015 um 03:05:35, schrieb Scott Kostyshak> On Thu, Dec 17, 2015 at 07:44:39AM +0100, Kornel Benko wrote: > > > Yesterday I had 0 failures with the command > > #ctest -j12 -L export -E "xhtml|lyx16" > > (about 2800 tests) > > > > Today, after the many changes, I don't know. > > You do not see any of the UNRELIABLE or SUSPENDED ones failing? Because they are _not_ part of '-L export' > On 9267f177 I get the following failures: > > 467:SUSPENDED.MANUALS.MISSINGPNG_export/doc/attic/eu_UserGuide_pdf3 > 468:SUSPENDED.MANUALS.MISSINGPNG_export/doc/attic/eu_UserGuide_pdf4_texF > 469:SUSPENDED.MANUALS.MISSINGPNG_export/doc/attic/eu_UserGuide_pdf4_systemF > 741:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/Additional_pdf4_texF > 754:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/Customization_pdf4_texF > 756:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/Customization_pdf5_texF > 788:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/EmbeddedObjects_dvi3_texF > 793:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/EmbeddedObjects_pdf4_texF > 827:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/Math_dvi3_texF > 866:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/UserGuide_dvi3_texF > 871:UNRELIABLE.TODO.WRONG.OUTPUT_export/doc/es/UserGuide_pdf4_texF > 1955:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/seminar_dvi > 1958:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/seminar_pdf > 1959:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/seminar_pdf2 > 1960:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/seminar_pdf3 > 1961:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/seminar_pdf4_texF > 1962:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/seminar_pdf4_systemF > 2869:UNRELIABLE.EXAMPLES.TODO.WRONG.OUTPUT_export/examples/es/linguistics_pdf4_texF > 2950:UNRELIABLE.EXAMPLES.NONSTANDARD_export/examples/fa/splash_dvi3_texF > 2951:UNRELIABLE.EXAMPLES.NONSTANDARD_export/examples/fa/splash_dvi3_systemF > 2955:UNRELIABLE.EXAMPLES.NONSTANDARD_export/examples/fa/splash_pdf4_texF > 2956:UNRELIABLE.EXAMPLES.NONSTANDARD_export/examples/fa/splash_pdf4_systemF > 2957:UNRELIABLE.EXAMPLES.NONSTANDARD_export/examples/fa/splash_pdf5_texF > 2958:UNRELIABLE.EXAMPLES.NONSTANDARD_export/examples/fa/splash_pdf5_systemF > 3093:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/fr/seminar_dvi > 3096:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/fr/seminar_pdf > 3097:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/fr/seminar_pdf2 > 3098:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/fr/seminar_pdf3 > 3099:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/fr/seminar_pdf4_texF > 3100:UNRELIABLE.EXAMPLES.ERRATIC_export/examples/fr/seminar_pdf4_systemF > 3219:INVERTED.EXAMPLES_export/examples/he/splash_pdf5_systemF > 3771:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_dvi > 3772:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_dvi3_texF > 3773:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_dvi3_systemF > 3774:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf > 3775:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf2 > 3776:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf3 > 3777:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf4_texF > 3778:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf4_systemF > 3779:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf5_texF > 3780:UNRELIABLE.NONSTANDARD_export/templates/IUCr-article_pdf5_systemF > 3908:UNRELIABLE.NONSTANDARD_export/templates/ectaart_dvi3_texF > 3909:UNRELIABLE.NONSTANDARD_export/templates/ectaart_dvi3_systemF > 3915:UNRELIABLE.NONSTANDARD_export/templates/ectaart_pdf5_texF > 3916:UNRELIABLE.NONSTANDARD_export/templates/ectaart_pdf5_systemF > 3939:UNRELIABLE.NONSTANDARD_export/templates/es_beamer-conference-ornate-20min_pdf4_texF > 4025:UNRELIABLE.NONSTANDARD_export/templates/ja_beamer-conference-ornate-20min_pdf2 > 4027:UNRELIABLE.NONSTANDARD_export/templates/ja_beamer-conference-ornate-20min_pdf4_texF > 4028:UNRELIABLE.NONSTANDARD_export/templates/ja_beamer-conference-ornate-20min_pdf4_systemF > 4029:UNRELIABLE.NONSTANDARD_export/templates/ja_beamer-conference-ornate-20min_pdf5_texF > 4030:UNRELIABLE.NONSTANDARD_export/templates/ja_beamer-conference-ornate-20min_pdf5_systemF > 4041:UNRELIABLE.NONSTANDARD_export/templates/kluwer_pdf4_systemF > 4043:UNRELIABLE.NONSTANDARD_export/templates/kluwer_pdf5_systemF > 4239:check_accessible_urls > 4242:check_translators > Scott Unreliable are such, that the outcome may change with local configuration. My tests are not expected to have the same output as yours. Kornel signature.asc Description: This is a digitally signed message part.
Re: What is our goal for 2.2 regarding the export tests?
On Sat, Dec 19, 2015 at 10:08:29AM +0100, Kornel Benko wrote: > This is meant if we are checking for regression. Yes. > But it is better to also check for load, to catch also layout errors. > ctest -L 'export|load' Good point. I will add this command to the checklist for release managers. I am eventually going to clean up my notes and make public a checklist for future release managers to follow. Scott signature.asc Description: PGP signature
Re: What is our goal for 2.2 regarding the export tests?
On 2015-12-17, Kornel Benko wrote: > Am Donnerstag, 17. Dezember 2015 um 01:25:35, schrieb Scott Kostyshak >>> A big thanks to Kornel and Günter for all of their work on the export >> tests. These will catch regressions and make us more confident in our >> releases going forward. ... >> What is a reasonable expectation of the output of 'ctest' with the >> following? >> -DLYX_ENABLE_EXPORT_TESTS=ON > Yesterday I had 0 failures with the command > #ctest -j12 -L export -E "xhtml|lyx16" > (about 2800 tests) > Today, after the many changes, I don't know. These are now reverted. Generally, as it is quite easy to invert a test or mark it as unreliable, the goal should be 0 test-failures. No test-failures means that all export-failures are recognized as one of * correct fail (e.g. TeX limitations prevent export to a certain format) * tolerable (e.g. failures for "exotic" documents with non-default output format) * unreliable (e.g. failures due to non-standard requirements, failures depending on local configuration, "erratic" behaviour (dependong on phase of the moon) * TODO item (e.g. LyX bugs that wait for someone to fix them or test-failures that still need proper categorization) It does not mean that there are no bugs or errors. Günter
Re: What is our goal for 2.2 regarding the export tests?
Am Donnerstag, 17. Dezember 2015 um 01:25:35, schrieb Scott Kostyshak> A big thanks to Kornel and Günter for all of their work on the export > tests. These will catch regressions and make us more confident in our > releases going forward. > > I have lost track of some of the newest changes regarding the tests. > When I run the export tests with our final 2.2.0, what will the expected > output be for all of the different configuration options? For example, > with > > -DLYX_ENABLE_URLTESTS=ON > > I think it unfortunately will be expected for 'check_accessible_urls' > to fail because of a false positive that we cannot get rid of. Yes, this depends unfortunately on moon. > Several keytests are also expected to fail because our machinery for > that is not working. Should not ..., but changes in shortcuts often need also adaptations to the tests. This is not always done. > What is a reasonable expectation of the output of 'ctest' with the > following? > > -DLYX_ENABLE_EXPORT_TESTS=ON Yesterday I had 0 failures with the command #ctest -j12 -L export -E "xhtml|lyx16" (about 2800 tests) Today, after the many changes, I don't know. > Scott Kornel signature.asc Description: This is a digitally signed message part.