Re: [LyX/master] Cmake export tests: List of inverted tests is expanding

[Scott Kostyshak]

On Sun, Nov 22, 2015 at 04:56:34PM +0100, Uwe Stöhr wrote:
> Am 20.11.2015 um 01:36 schrieb Scott Kostyshak:
> 
> >For both TeX fonts and system fonts? For me they only compile with TeX
> >fonts.
> 
> Here too. But why do you insist to compile with non-TeX fonts?

Well this is a separate conversation. In this particular case though,
you took my quote out of context. The full context is the following:

>> 112   export/examples/H-P-statements_pdf[45]_systemF
>>
>> I cannot reproduce this, I have hpstatement 2013/07/02 v1.01 and both
>> H-P-statements_pdf4 and H-P-statements_pdf5 works here.

>For both TeX fonts and system fonts? For me they only compile with TeX
>fonts.
> 
> Here too. But why do you insist to compile with non-TeX fonts?

You stated that you cannot reproduce the results of the test. So the
conversation was about the test and about whether you could reproduce
the results. The test states "systemF" which means "system fonts" (or
"non-TeX fonts".

Scott


signature.asc
Description: PGP signature

Re: [LyX/master] Cmake export tests: List of inverted tests is expanding

[Uwe Stöhr]

Am 20.11.2015 um 01:36 schrieb Scott Kostyshak:


For both TeX fonts and system fonts? For me they only compile with TeX
fonts.


Here too. But why do you insist to compile with non-TeX fonts? I can 
understand the package author that he predefines a font for the 
statement text since it depends on the font if characters can be 
displayed or not. I mean, imagine you are a Bulgarian and thus write in 
Cyrillic. Since Bulgaria is in the EU you need the EU-wide H-P 
statements. What will happen if you use a system font designed for 
Cyrillic that might e.g. miss the long hyphen used in the statement texts?


My point is that we should not try to test for features that are not 
available with the packages used in the documents. It is in my opinion 
not a mistake that packages like mhchem require certain fonts to assure 
that you will get a proper and correct output. (hp-statement is part of 
mhchem and for chemical stuff you need special characters that are not 
availably in every system font)


regards Uwe

Re: [LyX/master] Cmake export tests: List of inverted tests is expanding

[Kornel Benko]

Am Sonntag, 22. November 2015 um 18:10:55, schrieb Uwe Stöhr 
> Am 20.11.2015 um 11:40 schrieb Guenter Milde:
> >>> +# Non-ASCII in ERT
> >>> +export/doc/fr/Additional_pdf4_texF
> >
> >> This should now work.
> >
> > Unfortunately, it does not: after updating to c818fea3af2866b3d4b4/lyxgit
> > there are still non-ASCII characters in ERT in doc/fr/Additional.lyx, e.g.
> > the "ê" in line 879:
> 
> Thanks for testing. Obviusly something went wrong in
> 
> ddb05f46b4280acc0be94fa5c717cf37713a4746/lyxgit
> 
> Now it should really work.

It does not work here.
Neither for pdf4_texF nor for pdf5_systemF
This is the whole output I get with xelatex:

This is XeTeX, Version 3.14159265-2.6-0.2 (TeX Live 2015) (preloaded 
format=xelatex)
 restricted \write18 enabled.
entering extended mode
(./Additional.tex
LaTeX2e <2015/10/01> patch level 2
Babel <3.9m> and hyphenation patterns for 79 languages loaded.

Exit 1

Lualatex with system fonts gives errors too, but it creates at least pdf.
...
Overfull \hbox (42.52043pt too wide) in paragraph at lines 522--531
[][]\EU2/lmr/m/n/12 dé-cla-rer les com-mandes dé-fi-nies par l’uti-li-sa-te
ur (avec \EU2/lmtt/m/n/12 \newcommand \EU2/lmr/m/n/12 ou \EU2/lmtt/m/n/12 \rene
wcommand\EU2/lmr/m/n/12 ),
 []

[6] [7] [8](load luc: /usr9/local/texlive/2015/texmf-var/luatex-cache/generic/fo
nts/otf/lmmono8-regular.luc) [9] [10
Missing character: There is no ― (U+2015) in font "file:lmroman12-regular:scri
pt=latn;+trep;+tlig;"!
] [11] [12
...

> thanks and regards
> Uwe

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: [LyX/master] Cmake export tests: List of inverted tests is expanding

[Uwe Stöhr]

Am 20.11.2015 um 11:40 schrieb Guenter Milde:

+# Non-ASCII in ERT
+export/doc/fr/Additional_pdf4_texF



This should now work.


Unfortunately, it does not: after updating to c818fea3af2866b3d4b4/lyxgit
there are still non-ASCII characters in ERT in doc/fr/Additional.lyx, e.g.
the "ê" in line 879:


Thanks for testing. Obviusly something went wrong in

ddb05f46b4280acc0be94fa5c717cf37713a4746/lyxgit

Now it should really work.


The reason is, that the preamble has the code:

% if pdflatex is used
\usepackage{ifpdf}
\ifpdf

 % set fonts for nicer pdf view
 \IfFileExists{lmodern.sty}
  {\usepackage{lmodern}}{}

\fi % end if pdflatex is used

but \ifpdf is also true for XeTeX and LuaTeX with PDF-export.


Interesting. This explains the problems.

I propose the following solution:

\usepackage{ifluatex}
\ifluatex
 % LuaLaTeX specific code
\else % if XeTeX or pdflatex
 \usepackage{ifpdf} % part of the hyperref bundle
 \ifpdf % if pdflatex is used
  % set fonts for nicer pdf view
  \IfFileExists{lmodern.sty}{\usepackage{lmodern}}{}
 \fi % end if pdflatex is used
\fi % end ifluatex

instead of what I have currently in the preamble. This works for me. 
Does this also work on your system?



I proposed several fixes for this some posts ago. My suggestion is to invert
the tests for now and discuss the problem later.


Could you point me to this? I cannot find the thread.

thanks and regards
Uwe

Re: [LyX/master] Cmake export tests: List of inverted tests is expanding

[Guenter Milde]

On 2015-11-22, Uwe Stöhr wrote:
> Am 20.11.2015 um 11:40 schrieb Guenter Milde:


>> The reason is, that the preamble has the code:

>> % if pdflatex is used
>> \usepackage{ifpdf}
>> \ifpdf

>>  % set fonts for nicer pdf view
>>  \IfFileExists{lmodern.sty}
>>   {\usepackage{lmodern}}{}

>> \fi % end if pdflatex is used

>> but \ifpdf is also true for XeTeX and LuaTeX with PDF-export.

> Interesting. This explains the problems.

> I propose the following solution:

> \usepackage{ifluatex}
> \ifluatex
>   % LuaLaTeX specific code
> \else % if XeTeX or pdflatex
>   \usepackage{ifpdf} % part of the hyperref bundle
>   \ifpdf % if pdflatex is used
>% set fonts for nicer pdf view
>\IfFileExists{lmodern.sty}{\usepackage{lmodern}}{}
>   \fi % end if pdflatex is used
> \fi % end ifluatex

> instead of what I have currently in the preamble. This works for me. 
> Does this also work on your system?

No. We must test for Unicode fonts vs. 8-bit TeX fonts, this can be done via
testing for use of the "fontspec" package.

>> I proposed several fixes for this some posts ago. My suggestion is to invert
>> the tests for now and discuss the problem later.

> Could you point me to this? I cannot find the thread.

Scott kindly reposted it yesterday under a more appropriate subject line:

 21 Nov  :[Scott Kostys]Re: Font of LyX manuals


Günter

Re: [LyX/master] Cmake export tests: List of inverted tests is expanding

[Guenter Milde]

On 2015-11-22, Uwe Stöhr wrote:
> Am 20.11.2015 um 01:36 schrieb Scott Kostyshak:

>> For both TeX fonts and system fonts? For me they only compile with TeX
>> fonts.

> Here too. But why do you insist to compile with non-TeX fonts?

Because we want to test compilation with non-TeX fonts and documents that
ship with LyX are a good ressource for a test sample.

>From your point of view, this may look like an "abuse" of the
documentation.

I agree that it is a "mixed blessing":

* automated export test give early notice, if the documentation does not
  compile after a change in the development version of LyX.

  This may point to a problem with the commit or to a hithero hidden
  problem in the documentation - in both cases this is a good thing.

* In order to test more than just pdflatex export, the documents are
  exported to formats they were not designed for and (by default) with both,
  TeX fonts and non-TeX fonts.

  Failure of these "secondary" tests may either point to a problem in LyX,
  to a problem in the docs, or be due to an incompatibility like the one
  below.

  Making the documentation "Unicode font secure" is additonal work, but may
  also be an investment into the future where more and more people expect
  this as the "normal" export route for LateX documents.
  
  For now, the idea is to 
  
  * make simple changes that do not stand in the way directly in the
documentation files, and
  * flag documents that don't compile with non-TeX fonts (or other
modifications or non-default export routes). 

  Most of the noise about export tests in the last weeks was about this
  categorization.


> I can understand the package author that he predefines a font for the
> statement text since it depends on the font if characters can be
> displayed or not. I mean, imagine you are a Bulgarian and thus write in
> Cyrillic. Since Bulgaria is in the EU you need the EU-wide H-P
> statements. What will happen if you use a system font designed for
> Cyrillic that might e.g. miss the long hyphen used in the statement
> texts?

Export will fail, as LyX now reports missing characters as an error.



> My point is that we should not try to test for features that are not
> available with the packages used in the documents. It is in my opinion
> not a mistake that packages like mhchem require certain fonts to assure
> that you will get a proper and correct output. (hp-statement is part of
> mhchem and for chemical stuff you need special characters that are not
> availably in every system font)

Agreed. However, many of our manuals do compile fine to several output
formats and provide a good basis to test these export routes. It would be
a big help, if we can cooperate in this "dual use".


Günter

Re: [LyX/master] Cmake export tests: List of inverted tests is expanding

[Guenter Milde]

Dear Uwe,

thank you for joining in the export test discussion.

On 2015-11-20, Uwe Stöhr wrote:
> Am 19.11.2015 um 03:00 schrieb Kornel Benko:

>> commit ee72677fded357755c76e20a6adb256962d50445
> >
>> +# Non-ASCII in ERT
>> +export/doc/fr/Additional_pdf4_texF

> This should now work.

Unfortunately, it does not: after updating to c818fea3af2866b3d4b4/lyxgit
there are still non-ASCII characters in ERT in doc/fr/Additional.lyx, e.g.
the "ê" in line 879:

 Vous pouvez mettre presque tout dedans, même des figures et des tableaux

I also don't find this line in the diff of ddb05f46b4280/lyxgit
"French Additional.lyx: replace non-ASCII char in TEX code"

BTW: Are you sure you export with a fresh compiled LyX?
 
 For XeTeX with 8-bit TeX fonts, LyX now uses ASCII encoding of the
 source file, ignoring the document's "inputenc" setting.
 
 Older versions used utf-8-plain, but this leads to "missing characters"
 for any character above the 256 range and to wrong characters for many
 in the 128-256 range.

 The source-view should show the use of ASCII-replacements for non-ASCII
 characters if you select "View XeTeX" AND "View complete source" or
 "View main document".
 (There is still a bug with "View current paragraph".

LuaTeX with TeX fonts works, because here we can use the set
input-encoding with "luainputenc". 
 

>> +# lmodern.sty? + maybe more
>> +export/doc/fr/(Additional_pdf5|Customization_pdf5|EmbeddedObjects_pdf4|UserGuide_pdf5)_systemF

> Additional_pdf5, Customization_pdf5 and UserGuide_pdf5 work for me. 
> EmbeddedObjects_pdf4 works here too.

Also, if you change the document to use "non-TeX fonts"? (This is what
the "_systemF" suffix indicates: the test machinery changes the
"useNonTeXFonts" setting to True and exports with Xe/LuaTeX.)
 
After changing fr/Additional.lyx to use non-TeX-fonts, I get 10 missing
character errors with XeTeX and 5 with LuaTeX:
 
 Missing character: There is no ― in font
 [lmroman12-regular]:mapping=tex-text!
 
 ...
 
The reason is, that the preamble has the code:

   % if pdflatex is used
   \usepackage{ifpdf}
   \ifpdf
   
% set fonts for nicer pdf view
\IfFileExists{lmodern.sty}
 {\usepackage{lmodern}}{}
   
   \fi % end if pdflatex is used

but \ifpdf is also true for XeTeX and LuaTeX with PDF-export.
Therefore, this preamble loads "lmodern.sty" also, if "use TeX fonts" is
selected. The package lmodern.sty then overwrites the font encoding setting
and leads to loading 8-bit TeX fonts while LyX believes it uses Unicode
fonts and uses the "utf8-plain" input encoding.

I proposed several fixes for this some posts ago. My suggestion is to invert
the tests for now and discuss the problem later.



>> +# chemgreek incompatible with LuaTeX (cf. Math.lyx)
>> +# for now inverted
>> +export/doc/sk/Intro_pdf5_systemF
>> +export/examples/PDF-comment_pdf5_texF
>> +export/examples/aas_sample_(dvi3|pdf5)_(texF|systemF)

> sk/Intro_pdf5 works here.

If I change it to use non-TeX fonts (to reproduce the *_systemF test), I get

  ! LaTeX Error: \begin{otherlanguage} on input line 360 ended by
  \end{itemize}.

which for me is an indication of the known "language nesting" problem with
polyglossia.

Günter

Re: [LyX/master] Cmake export tests: List of inverted tests is expanding

[Scott Kostyshak]

On Fri, Nov 20, 2015 at 10:40:01AM +, Guenter Milde wrote:

> If I change it to use non-TeX fonts (to reproduce the *_systemF test), I get
> 
>   ! LaTeX Error: \begin{otherlanguage} on input line 360 ended by
>   \end{itemize}.
> 
> which for me is an indication of the known "language nesting" problem with
> polyglossia.

I came to the same conclusion for this as well as others here:
https://www.mail-archive.com/search?l=mid=20151025042921.GA30789%40cotopaxi

If you have an updated idea of which tests are suffering from the
language nesting issue, we can post these tests on #9633 so that when
someone has time to take a look, they will know where to start.

Scott


signature.asc
Description: PGP signature

Re: [LyX/master] Cmake export tests: List of inverted tests is expanding

[Georg Baum]

Scott Kostyshak wrote:

> On Fri, Nov 20, 2015 at 10:40:01AM +, Guenter Milde wrote:
> 
>> If I change it to use non-TeX fonts (to reproduce the *_systemF test), I
>> get
>> 
>>   ! LaTeX Error: \begin{otherlanguage} on input line 360 ended by
>>   \end{itemize}.
>> 
>> which for me is an indication of the known "language nesting" problem
>> with polyglossia.

Yep.
 
> I came to the same conclusion for this as well as others here:
> https://www.mail-archive.com/search?l=mid=20151025042921.GA30789%40cotopaxi
> 
> If you have an updated idea of which tests are suffering from the
> language nesting issue, we can post these tests on #9633 so that when
> someone has time to take a look, they will know where to start.

I know where to start (have reduced test documents locally). My plan is to 
add these dedicated tests, put the non-working ones in the inverted list, 
and then remove them from the inverted list in the same commit that fixes 
the code. This will give a very good documentation of the change and makes 
it easy for other to verify that the fix works.


Georg

Re: [LyX/master] Cmake export tests: List of inverted tests is expanding

[Scott Kostyshak]

On Fri, Nov 20, 2015 at 10:19:30PM +0100, Georg Baum wrote:
> Scott Kostyshak wrote:
> 
> > On Fri, Nov 20, 2015 at 10:40:01AM +, Guenter Milde wrote:
> > 
> >> If I change it to use non-TeX fonts (to reproduce the *_systemF test), I
> >> get
> >> 
> >>   ! LaTeX Error: \begin{otherlanguage} on input line 360 ended by
> >>   \end{itemize}.
> >> 
> >> which for me is an indication of the known "language nesting" problem
> >> with polyglossia.
> 
> Yep.
>  
> > I came to the same conclusion for this as well as others here:
> > https://www.mail-archive.com/search?l=mid=20151025042921.GA30789%40cotopaxi
> > 
> > If you have an updated idea of which tests are suffering from the
> > language nesting issue, we can post these tests on #9633 so that when
> > someone has time to take a look, they will know where to start.
> 
> I know where to start (have reduced test documents locally). My plan is to 
> add these dedicated tests, put the non-working ones in the inverted list, 
> and then remove them from the inverted list in the same commit that fixes 
> the code. This will give a very good documentation of the change and makes 
> it easy for other to verify that the fix works.

That will be great. Since you are planning to work on it, I'll change
the milestone of #9633 to 2.2.0.

Scott


signature.asc
Description: PGP signature

Re: [LyX/master] Cmake export tests: List of inverted tests is expanding

[Uwe Stöhr]

Am 19.11.2015 um 03:00 schrieb Kornel Benko:


commit ee72677fded357755c76e20a6adb256962d50445

>

+# Non-ASCII in ERT
+export/doc/fr/Additional_pdf4_texF


This should now work.


+# lmodern.sty? + maybe more
+export/doc/fr/(Additional_pdf5|Customization_pdf5|EmbeddedObjects_pdf4|UserGuide_pdf5)_systemF


Additional_pdf5, Customization_pdf5 and UserGuide_pdf5 work for me. 
EmbeddedObjects_pdf4 works here too.



+# chemgreek incompatible with LuaTeX (cf. Math.lyx)
+# for now inverted
+export/doc/sk/Intro_pdf5_systemF
+export/examples/PDF-comment_pdf5_texF
+export/examples/aas_sample_(dvi3|pdf5)_(texF|systemF)


sk/Intro_pdf5 works here.

I am using
LuaTeX, Version beta-0.80.
and
XeTeX, Version 3.14159265-2.6-0.2

regards Uwe

Re: [LyX/master] Cmake export tests: List of inverted tests is expanding

[Uwe Stöhr]

Another issue in revertedTests:

108 # Gives the following error:
109 # # ! Package hpstatement Error: H or P statement "H251"
110 # # (hpstatement)is not defined.
111 # 1580:export/examples/H-P-statements_pdf5_systemF
112 export/examples/H-P-statements_pdf[45]_systemF

I cannot reproduce this, I have hpstatement 2013/07/02 v1.01 and both
H-P-statements_pdf4 and H-P-statements_pdf5 works here.

regards Uwe

Re: [LyX/master] Cmake export tests: List of inverted tests is expanding

[Scott Kostyshak]

On Fri, Nov 20, 2015 at 01:19:24AM +0100, Uwe Stöhr wrote:
> Another issue in revertedTests:
> 
> 108 # Gives the following error:
> 109   # # ! Package hpstatement Error: H or P statement "H251"
> 110   # # (hpstatement)is not defined.
> 111   # 1580:export/examples/H-P-statements_pdf5_systemF
> 112   export/examples/H-P-statements_pdf[45]_systemF
> 
> I cannot reproduce this, I have hpstatement 2013/07/02 v1.01 and both
> H-P-statements_pdf4 and H-P-statements_pdf5 works here.

For both TeX fonts and system fonts? For me they only compile with TeX
fonts.

If they compile with system fonts, then we should compare logs to figure
out what is different among our systems.

Scott


signature.asc
Description: PGP signature