Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Georg Baum]

Guenter Milde wrote:

> I assume, heb1.tex is "splash_good.tex" and heb2.tex is "splash_bad.tex",
> right?

I guess so. My newsreader did not display any names for the attachments at 
Scotts message.

> 
>> So, the real diferences of this change are:
> 
>> 1) LyX writes cp1255 encoding instead of utf8
> 
> Yes. This is intended -- if document source is set to use 8-bit TeX fonts,
> the input encoding should be the same for pdftex and luatex.
> (Before, it was forced to be utf8-plain with luatex and xetex, which led
> to many errors (or wrong output).)

Thanks for the explanation. I did not think of the \inputencoding setting of 
the LyX file.

> 
>> 2) LyX adds a \cyrtext command
> 
> If I read the diff correctly, the \cyrtext was added before (heb1) but is
> now only added if required.

Right, I mixed this up with the first version of the diff that I did the 
other way round.


Georg

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Scott Kostyshak]

On Sun, Feb 14, 2016 at 08:49:26PM +, Guenter Milde wrote:

> However, I can confirm that the changes are intended.

Thanks for checking.

Scott


signature.asc
Description: PGP signature

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Guenter Milde]

On 2016-02-14, Georg Baum wrote:
> Scott Kostyshak wrote:
>> On Sat, Feb 13, 2016 at 08:54:57PM +, Guenter Milde wrote:
>>> On 2016-02-13, Scott Kostyshak wrote:

>>> > After this commit, when I run the command

>>> >   lyx -e luatex lib/doc/he/Intro.lyx

>>> > it either fails to produce Intro.tex or (on current master) it produces
>>> > a Intro.tex with gibberish where there should be Hebrew (at least this
>>> > is what Vim shows me).

> It is not gibberish. vim fails to detect the encoding cp1255. If you open 
> the file with kate it correctly detects the encoding and displays hebrew 
> text.

Thanks for finding this out.


>> Do you think it is OK that we now produce splash_bad.tex and before we
>> produced splash_good.tex? If you tell me this is fine and that there
>> is no encoding issue, then I trust your judgment since I know nothing
>> about encoding.

> Attached is a diff that was produced after converting the utf8 encoded file 
> to cp1255:

> recode utf8..cp1255 < heb1.tex > heb1-cp1255.tex
> diff -ub heb2.tex heb1-cp1255.tex > x.diff

I assume, heb1.tex is "splash_good.tex" and heb2.tex is "splash_bad.tex",
right?

> So, the real diferences of this change are:

> 1) LyX writes cp1255 encoding instead of utf8

Yes. This is intended -- if document source is set to use 8-bit TeX fonts,
the input encoding should be the same for pdftex and luatex.
(Before, it was forced to be utf8-plain with luatex and xetex, which led to
many errors (or wrong output).)

> 2) LyX adds a \cyrtext command

If I read the diff correctly, the \cyrtext was added before (heb1) but is
now only added if required.

> I don't know if 1) is wanted, but the way it is done is correct: The file is 
> written in cp1255 encoding, and luainputenc is loaded with the same 
> encoding. 

> 2) looks unnneeded to me.

This is fixed sice some months now.

> Obviously, luatex+hebrew is not completely broken, since the utf8 version 
> worked. 

Did the utf8 version actually compile and give the correct result?

> What is the advantage of using cp1255 instead of utf8 now?

Less surprise for the user: if the document is set to use "language
default", is uses "language default" (which is cp1255 for Hebrew).
If the document is set to use utf8, it uses utf8.

The advantage might not be visible in this example, but, e.g. with external
files (bibtex, ...) the encoding of the generated latex source can be
important.


> --- heb2.tex  2016-02-14 10:35:23.083348413 +0100
> +++ heb1-cp1255.tex   2016-02-14 10:51:29.859358650 +0100
> @@ -2,7 +2,7 @@
>  %% Do not edit unless you really know what you are doing.
>  \documentclass[english,hebrew]{article}
>  \usepackage[T1]{fontenc}
> -\usepackage[cp1255]{luainputenc}
> +\usepackage[utf8]{luainputenc}
>  \usepackage{url}
>  \usepackage{amssymb}

> @@ -10,6 +10,10 @@

>  %% LyX specific LaTeX commands.
>  \providecommand{\LyX}{L\kern-.1667em\lower.25em\hbox{Y}\kern-.125emX\@}
> +\DeclareRobustCommand{\cyrtext}{%
> +  \fontencoding{T2A}\selectfont\def\encodingdefault{T2A}}
> +\DeclareRobustCommand{\textcyr}[1]{\leavevmode{\cyrtext #1}}
> +

>  %% Textclass specific LaTeX commands.
>  \usepackage{theorem}


This does not solve the case of not-working LaTeX export. 

However, I can confirm that the changes are intended.

(The disadvantages of the current input encoding default are a different
topic that are handled in a different bug report ...)


Thanks, 

Günter

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Scott Kostyshak]

On Sun, Feb 14, 2016 at 11:01:06AM +0100, Georg Baum wrote:
> Scott Kostyshak wrote:
> 
> > On Sat, Feb 13, 2016 at 08:54:57PM +, Guenter Milde wrote:
> >> On 2016-02-13, Scott Kostyshak wrote:
> >>
> >> > After this commit, when I run the command
> >> 
> >> >   lyx -e luatex lib/doc/he/Intro.lyx
> >> 
> >> > it either fails to produce Intro.tex or (on current master) it produces
> >> > a Intro.tex with gibberish where there should be Hebrew (at least this
> >> > is what Vim shows me).
> 
> It is not gibberish. vim fails to detect the encoding cp1255. If you open 
> the file with kate it correctly detects the encoding and displays hebrew 
> text.

OK I see.

> >> No, as Luatex+Hebrew failing is an upstream bug.
> > 
> > But I am just talking about exporting to .tex and opening in a text
> > editor. LuaTeX does not even have a chance if our export to .tex
> > corrupts the Hebrew. To be clear, attached are splash_good.tex and
> > splash_bad.tex. Do you think it is OK that we now produce splash_bad.tex
> > and before we produced splash_good.tex? If you tell me this is fine and
> > that there is no encoding issue, then I trust your judgment since I
> > know nothing about encoding.
> 
> Attached is a diff that was produced after converting the utf8 encoded file 
> to cp1255:
> 
> recode utf8..cp1255 < heb1.tex > heb1-cp1255.tex
> diff -ub heb2.tex heb1-cp1255.tex > x.diff
> 
> (recode adds windows line ends, therefore the -b argument for diff)

Thanks for walking me through this. I did not know about recode.

> So, the real diferences of this change are:
> 
> 1) LyX writes cp1255 encoding instead of utf8
> 
> 2) LyX adds a \cyrtext command
> 
> I don't know if 1) is wanted, but the way it is done is correct: The file is 
> written in cp1255 encoding, and luainputenc is loaded with the same 
> encoding. 2) looks unnneeded to me.
> 
> Obviously, luatex+hebrew is not completely broken, since the utf8 version 
> worked. What is the advantage of using cp1255 instead of utf8 now?

Thanks for looking into this.

Scott


signature.asc
Description: PGP signature

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Georg Baum]

Scott Kostyshak wrote:

> On Sat, Feb 13, 2016 at 08:54:57PM +, Guenter Milde wrote:
>> On 2016-02-13, Scott Kostyshak wrote:
>>
>> > After this commit, when I run the command
>> 
>> >   lyx -e luatex lib/doc/he/Intro.lyx
>> 
>> > it either fails to produce Intro.tex or (on current master) it produces
>> > a Intro.tex with gibberish where there should be Hebrew (at least this
>> > is what Vim shows me).

It is not gibberish. vim fails to detect the encoding cp1255. If you open 
the file with kate it correctly detects the encoding and displays hebrew 
text.

>> No, as Luatex+Hebrew failing is an upstream bug.
> 
> But I am just talking about exporting to .tex and opening in a text
> editor. LuaTeX does not even have a chance if our export to .tex
> corrupts the Hebrew. To be clear, attached are splash_good.tex and
> splash_bad.tex. Do you think it is OK that we now produce splash_bad.tex
> and before we produced splash_good.tex? If you tell me this is fine and
> that there is no encoding issue, then I trust your judgment since I
> know nothing about encoding.

Attached is a diff that was produced after converting the utf8 encoded file 
to cp1255:

recode utf8..cp1255 < heb1.tex > heb1-cp1255.tex
diff -ub heb2.tex heb1-cp1255.tex > x.diff

(recode adds windows line ends, therefore the -b argument for diff)

So, the real diferences of this change are:

1) LyX writes cp1255 encoding instead of utf8

2) LyX adds a \cyrtext command

I don't know if 1) is wanted, but the way it is done is correct: The file is 
written in cp1255 encoding, and luainputenc is loaded with the same 
encoding. 2) looks unnneeded to me.

Obviously, luatex+hebrew is not completely broken, since the utf8 version 
worked. What is the advantage of using cp1255 instead of utf8 now?


Georg
--- heb2.tex	2016-02-14 10:35:23.083348413 +0100
+++ heb1-cp1255.tex	2016-02-14 10:51:29.859358650 +0100
@@ -2,7 +2,7 @@
 %% Do not edit unless you really know what you are doing.
 \documentclass[english,hebrew]{article}
 \usepackage[T1]{fontenc}
-\usepackage[cp1255]{luainputenc}
+\usepackage[utf8]{luainputenc}
 \usepackage{url}
 \usepackage{amssymb}
 
@@ -10,6 +10,10 @@
 
 %% LyX specific LaTeX commands.
 \providecommand{\LyX}{L\kern-.1667em\lower.25em\hbox{Y}\kern-.125emX\@}
+\DeclareRobustCommand{\cyrtext}{%
+  \fontencoding{T2A}\selectfont\def\encodingdefault{T2A}}
+\DeclareRobustCommand{\textcyr}[1]{\leavevmode{\cyrtext #1}}
+
 
 %% Textclass specific LaTeX commands.
 \usepackage{theorem}

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Scott Kostyshak]

On Tue, Oct 20, 2015 at 07:16:44PM +0200, Günter Milde wrote:
> commit 1523fc6023440f10ca0d82a681ded5c060d8fd33
> Author: Günter Milde 
> Date:   Tue Oct 20 19:14:39 2015 +0200
> 
> Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
> 
> Fixes output for 3 of the 4 test lyx-files.
> 
> Includes "FIXME"s at places where further action is required to get the 
> XeTeX
> export right but I don't know how.

After this commit, when I run the command

  lyx -e luatex lib/doc/he/Intro.lyx

it either fails to produce Intro.tex or (on current master) it produces
a Intro.tex with gibberish where there should be Hebrew (at least this
is what Vim shows me).

Is this expected? I know you put FIXMEs and one of them talks about
ASCII encoding, so I'm guessing it is indeed expected, but I want to
make sure. If it is expected, do we have a bug report?

Scott


signature.asc
Description: PGP signature

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Guenter Milde]

On 2016-02-13, Scott Kostyshak wrote:
> On Tue, Oct 20, 2015 at 07:16:44PM +0200, Günter Milde wrote:
>> commit 1523fc6023440f10ca0d82a681ded5c060d8fd33
>> Author: Günter Milde 
>> Date:   Tue Oct 20 19:14:39 2015 +0200

>> Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

>> Fixes output for 3 of the 4 test lyx-files.

>> Includes "FIXME"s at places where further action is required to get the 
>> XeTeX
>> export right but I don't know how.

> After this commit, when I run the command

>   lyx -e luatex lib/doc/he/Intro.lyx

> it either fails to produce Intro.tex or (on current master) it produces
> a Intro.tex with gibberish where there should be Hebrew (at least this
> is what Vim shows me).

Are you sure this has to do with this (partial) fix?

In suspiciousTests, I see:

# Hebrew docs do not currently work with LuaTeX (will be fixed in TeXLive-16)
# TODO: set working system fonts in the lyx sources
export/.*/he/.*(dvi3|pdf5).*

> Is this expected? 

So, until further notice of a fix, I expect problems with Hebrew and LuaTeX
with both, tex- and unicode-fonts.

> I know you put FIXMEs and one of them talks about ASCII encoding, so
> I'm guessing it is indeed expected, but I want to make sure. 

ASCII is required for XeTeX, LuaTeX can work with encoded source using
luainputenc.

> If it is expected, do we have a bug report?

No, as Luatex+Hebrew failing is an upstream bug.

Günter

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Scott Kostyshak]

On Sat, Feb 13, 2016 at 08:54:57PM +, Guenter Milde wrote:
> On 2016-02-13, Scott Kostyshak wrote:
> > On Tue, Oct 20, 2015 at 07:16:44PM +0200, Günter Milde wrote:
> >> commit 1523fc6023440f10ca0d82a681ded5c060d8fd33
> >> Author: Günter Milde 
> >> Date:   Tue Oct 20 19:14:39 2015 +0200
> 
> >> Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
> 
> >> Fixes output for 3 of the 4 test lyx-files.
> 
> >> Includes "FIXME"s at places where further action is required to get 
> >> the XeTeX
> >> export right but I don't know how.
> 
> > After this commit, when I run the command
> 
> >   lyx -e luatex lib/doc/he/Intro.lyx
> 
> > it either fails to produce Intro.tex or (on current master) it produces
> > a Intro.tex with gibberish where there should be Hebrew (at least this
> > is what Vim shows me).
> 
> Are you sure this has to do with this (partial) fix?

No. I did a git bisect that led me to this commit, but there were
various types of "failing" which makes me less confident that the
problem was linear and thus less likely the bisect works. By multiple
types of failings, I mean that either the Intro.tex file would not even
be created or it would but would not contain Hebrew. So I would not be
too surprised if the git bisect worked. I can do a manual check (compile
just before), but first I want to check if something is indeed wrong.

> In suspiciousTests, I see:
> 
> # Hebrew docs do not currently work with LuaTeX (will be fixed in
> TeXLive-16) # TODO: set working system fonts in the lyx sources
> export/.*/he/.*(dvi3|pdf5).*
> 
> > Is this expected? 
> 
> So, until further notice of a fix, I expect problems with Hebrew and
> LuaTeX with both, tex- and unicode-fonts.
> 
> > I know you put FIXMEs and one of them talks about ASCII encoding, so
> > I'm guessing it is indeed expected, but I want to make sure. 
> 
> ASCII is required for XeTeX, LuaTeX can work with encoded source using
> luainputenc.
> 
> > If it is expected, do we have a bug report?
> 
> No, as Luatex+Hebrew failing is an upstream bug.

But I am just talking about exporting to .tex and opening in a text
editor. LuaTeX does not even have a chance if our export to .tex
corrupts the Hebrew. To be clear, attached are splash_good.tex and
splash_bad.tex. Do you think it is OK that we now produce splash_bad.tex
and before we produced splash_good.tex? If you tell me this is fine and
that there is no encoding issue, then I trust your judgment since I
know nothing about encoding.

Scott


splash_good.tex
Description: TeX document


splash_bad.tex
Description: TeX document


signature.asc
Description: PGP signature

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Kornel Benko]

Am Mittwoch, 4. November 2015 um 10:33:58, schrieb Guenter Milde 

> On 2015-11-03, Scott Kostyshak wrote:
> > On Tue, Nov 03, 2015 at 06:49:01AM -0800, Kornel Benko wrote:
> >> > 
> >> > * to fix the regressions with "XeTeX + TeX fonts", a simple change in
> >> >   PDFOptions.cpp is sufficient.
> 
> >> That is, 29 tests are now better than before.
> >> Checking the output, all 29 are of type pdf4_texF (XeTeX + TeX fonts).
> 
> Good news.
> 
> Committed: f739c98fd7aa9cc6c7607/lyxgit

Thanks.

> ...
> 
> >> > * the updated "comprehensive but incomplete" patch based on your work is 
> >> > now
> >> >   at http://www.lyx.org/trac/ticket/9839, so the work is not lost.
> >> >   
> >> >   With the "show output anyway" option in 2.2, the status quo is actually
> >> >   tolerable. I'll try the patch for #9830, then there is also a 
> >> > workaround for
> >> >   ASCII export (i.e. the user can write G\"unter instead of Günter).
> 
> > Do we have a bug report for the workaround for ASCII export?
> 
> I added a comment http://www.lyx.org/trac/ticket/9839#comment:1
> 
> ...
> 
> > Günter, thanks for your work on these tricky issues. After alpha is
> > released, let me know if there is anything pending that you would like
> > for me to work on. Since I know nothing about this topic, you would need
> > to break it down into simple steps like you did for me before so I can
> > focus on the C++.

+1

> Thanks for your offer. We will see what we can do one alpha is out...
> 
> I'll also have a look at the other FIXMES and may ask for help there.
> 
> 
> Günter

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Guenter Milde]

On 2015-11-03, Scott Kostyshak wrote:
> On Tue, Nov 03, 2015 at 06:49:01AM -0800, Kornel Benko wrote:
>> > 
>> > * to fix the regressions with "XeTeX + TeX fonts", a simple change in
>> >   PDFOptions.cpp is sufficient.

>> That is, 29 tests are now better than before.
>> Checking the output, all 29 are of type pdf4_texF (XeTeX + TeX fonts).

Good news.

Committed: f739c98fd7aa9cc6c7607/lyxgit

...

>> > * the updated "comprehensive but incomplete" patch based on your work is 
>> > now
>> >   at http://www.lyx.org/trac/ticket/9839, so the work is not lost.
>> >   
>> >   With the "show output anyway" option in 2.2, the status quo is actually
>> >   tolerable. I'll try the patch for #9830, then there is also a workaround 
>> > for
>> >   ASCII export (i.e. the user can write G\"unter instead of Günter).

> Do we have a bug report for the workaround for ASCII export?

I added a comment http://www.lyx.org/trac/ticket/9839#comment:1

...

> Günter, thanks for your work on these tricky issues. After alpha is
> released, let me know if there is anything pending that you would like
> for me to work on. Since I know nothing about this topic, you would need
> to break it down into simple steps like you did for me before so I can
> focus on the C++.

Thanks for your offer. We will see what we can do one alpha is out...

I'll also have a look at the other FIXMES and may ask for help there.


Günter

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Guenter Milde]

On 2015-11-02, Scott Kostyshak wrote:
> On Mon, Nov 02, 2015 at 10:40:27PM +, Guenter Milde wrote:

Dear Scott,

>> > thanks for the patch.

>> I modified it to use UTF8 with XeTeX without the 
>> \inputencoding commands. 

...

> I would say go ahead and commit if you are satisfied with it. From what I
> understand, the patch is now a complete patch in the sense that it is a full
> step forward. 

Actually, I am not satisfied, because the "normal" use case (exporting to a
LaTeX file with inputencoding ASCII) is now worse than before. I this sense,
it is a partial patch.

For the use case "XeTeX with TeX fonts", it is overly complicated.

So I split it in two:

* to fix the regressions with "XeTeX + TeX fonts", a simple change in
  PDFOptions.cpp is sufficient. See below.
  
  The nice thing is, that this is simple, localized and only changes
  where change is required - actually a completion for the patch-set
  fixing #9740.

  >> A short test here showed that this
  >> helps with Umlauts or Cyrillic charactes in the PDF Info.

* the updated "comprehensive but incomplete" patch based on your work is now
  at http://www.lyx.org/trac/ticket/9839, so the work is not lost.
  
  With the "show output anyway" option in 2.2, the status quo is actually
  tolerable. I'll try the patch for #9830, then there is also a workaound for
  ASCII export (i.e. the user can write G\"unter instead of Günter).


> Let me know if you want me to run the export tests before/after.

Please check, if this patch cures some regressions introduced by
fixing #9740.

Günter


Exec: git 'diff' 'PDFOptions.cpp' 2>&1
Dir: /usr/local/src/lyx/src/

diff --git a/src/PDFOptions.cpp b/src/PDFOptions.cpp
index 6646888..143d715 100644
--- a/src/PDFOptions.cpp
+++ b/src/PDFOptions.cpp
@@ -174,21 +174,15 @@ void PDFOptions::writeLaTeX(OutputParams & runparams, 
otexstream & os,
opt = "\\hypersetup{" + rtrim(opt + hyperset, ",") + "}\n";
}
 
-   // hyperref expects LICR macros for non-ASCII chars. With Xe/LuaTeX 
utf-8 works, too.
-   // Usually, "(lua)inputenc" converts the input to LICR.
+   // hyperref expects LICR macros for non-ASCII chars.
+   // Usually, "(lua)inputenc" converts the input to LICR, with XeTeX 
utf-8 works, too.
// As hyperref provides good coverage for \inputencoding{utf8}, we can 
try
// this if the current input encoding does not support a character.
-   // FIXME: inputenc (part 1 of 2)
-   //   Replace the "FullUnicode" check with
-   //   check for loading of inputenc or luainputenc package
-   //   (see BufferParams::writeEncodingPreamble and 
runparams.encoding->package()).
-   //   Otherwise \inputencoding is not defined
-   //   (e.g. if "latex-encoding" is set to "ascii").
-   //   Dont forget to keep the check below (part 2) in sync!
-   if (need_unicode && enc && enc->iconvName() != "UTF-8"
-   &&!runparams.isFullUnicode()) {
-   os << "\\inputencoding{utf8}\n"
-  << setEncoding("UTF-8");
+   // FIXME: don't use \inputencoding if "inputenc" is not loaded (#9839).
+   if (need_unicode && enc && enc->iconvName() != "UTF-8") {
+   if (runparams.flavor != OutputParams::XETEX)
+   os << "\\inputencoding{utf8}\n";
+   os << setEncoding("UTF-8");
}
// If hyperref is loaded by the document class, we output
// \hypersetup \AtBeginDocument if hypersetup is not (yet)
@@ -204,11 +198,10 @@ void PDFOptions::writeLaTeX(OutputParams & runparams, 
otexstream & os,
   << "\\fi\n";
} else
os << from_utf8(opt);
-   // FIXME: inputenc (part 2 of 2)
-   if (need_unicode && enc && enc->iconvName() != "UTF-8"
-   &&!runparams.isFullUnicode()) {
-   os << setEncoding(enc->iconvName())
-  << "\\inputencoding{" << from_ascii(enc->latexName()) << 
"}\n";
+   if (need_unicode && enc && enc->iconvName() != "UTF-8") {
+   os << setEncoding(enc->iconvName());
+   if (runparams.flavor != OutputParams::XETEX)
+   os << "\\inputencoding{" << 
from_ascii(enc->latexName()) << "}\n";
}
 }
 

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Scott Kostyshak]

On Tue, Nov 03, 2015 at 06:49:01AM -0800, Kornel Benko wrote:
> Am Dienstag, 3. November 2015 um 12:40:22, schrieb Guenter Milde 
> 
> > On 2015-11-02, Scott Kostyshak wrote:
> > > On Mon, Nov 02, 2015 at 10:40:27PM +, Guenter Milde wrote:
> > 
> > Dear Scott,
> > 
> > >> > thanks for the patch.
> > 
> > >> I modified it to use UTF8 with XeTeX without the 
> > >> \inputencoding commands. 
> > 
> > ...
> > 
> > > I would say go ahead and commit if you are satisfied with it. From what I
> > > understand, the patch is now a complete patch in the sense that it is a 
> > > full
> > > step forward. 
> > 
> > Actually, I am not satisfied, because the "normal" use case (exporting to a
> > LaTeX file with inputencoding ASCII) is now worse than before. I this sense,
> > it is a partial patch.

OK makes sense.

> > For the use case "XeTeX with TeX fonts", it is overly complicated.
> > 
> > So I split it in two:
> > 
> > * to fix the regressions with "XeTeX + TeX fonts", a simple change in
> >   PDFOptions.cpp is sufficient. See below.
> >   
> >   The nice thing is, that this is simple, localized and only changes
> >   where change is required - actually a completion for the patch-set
> >   fixing #9740.
> > 
> >   >> A short test here showed that this
> >   >> helps with Umlauts or Cyrillic charactes in the PDF Info.
> > 
> > * the updated "comprehensive but incomplete" patch based on your work is now
> >   at http://www.lyx.org/trac/ticket/9839, so the work is not lost.
> >   
> >   With the "show output anyway" option in 2.2, the status quo is actually
> >   tolerable. I'll try the patch for #9830, then there is also a workaound 
> > for
> >   ASCII export (i.e. the user can write G\"unter instead of Günter).

Do we have a bug report for the workaround for ASCII export?

> > > Let me know if you want me to run the export tests before/after.
> > 
> > Please check, if this patch cures some regressions introduced by
> > fixing #9740.
> 
> Check with 'ctest -L suspended' (my local version)
> Now the following tests (previously not passing) pass:
>   390 - INVERTED_SEE-README.ctest_export/doc/Math_pdf4_texF (Failed)
>   702 - INVERTED_SEE-README.ctest_export/doc/de/Math_pdf4_texF (Failed)
>   870 - INVERTED_SEE-README.ctest_export/doc/es/Math_pdf4_texF (Failed)
>   1052 - INVERTED_SEE-README.ctest_export/doc/fr/Math_pdf4_texF (Failed)
>   1787 - INVERTED_SEE-README.ctest_export/examples/PDF-comment_pdf4_texF 
> (Failed)
> 
> They can now be removed from revertedTests.
> 
> checking now with 'ctest -L export -E '(lyx16|xhtml)$'
>   -> 94% tests passed, 179 tests failed out of 3041
> 
> That is, 29 tests are now better than before.
> Checking the output, all 29 are of type pdf4_texF (XeTeX + TeX fonts).
> 
> I'd say, please commit.

Thanks for checking Kornel.

Günter, thanks for your work on these tricky issues. After alpha is
released, let me know if there is anything pending that you would like
for me to work on. Since I know nothing about this topic, you would need
to break it down into simple steps like you did for me before so I can
focus on the C++.

Scott

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Kornel Benko]

Am Dienstag, 3. November 2015 um 12:40:22, schrieb Guenter Milde 

> On 2015-11-02, Scott Kostyshak wrote:
> > On Mon, Nov 02, 2015 at 10:40:27PM +, Guenter Milde wrote:
> 
> Dear Scott,
> 
> >> > thanks for the patch.
> 
> >> I modified it to use UTF8 with XeTeX without the 
> >> \inputencoding commands. 
> 
> ...
> 
> > I would say go ahead and commit if you are satisfied with it. From what I
> > understand, the patch is now a complete patch in the sense that it is a full
> > step forward. 
> 
> Actually, I am not satisfied, because the "normal" use case (exporting to a
> LaTeX file with inputencoding ASCII) is now worse than before. I this sense,
> it is a partial patch.
> 
> For the use case "XeTeX with TeX fonts", it is overly complicated.
> 
> So I split it in two:
> 
> * to fix the regressions with "XeTeX + TeX fonts", a simple change in
>   PDFOptions.cpp is sufficient. See below.
>   
>   The nice thing is, that this is simple, localized and only changes
>   where change is required - actually a completion for the patch-set
>   fixing #9740.
> 
>   >> A short test here showed that this
>   >> helps with Umlauts or Cyrillic charactes in the PDF Info.
> 
> * the updated "comprehensive but incomplete" patch based on your work is now
>   at http://www.lyx.org/trac/ticket/9839, so the work is not lost.
>   
>   With the "show output anyway" option in 2.2, the status quo is actually
>   tolerable. I'll try the patch for #9830, then there is also a workaound for
>   ASCII export (i.e. the user can write G\"unter instead of Günter).
> 
> 
> > Let me know if you want me to run the export tests before/after.
> 
> Please check, if this patch cures some regressions introduced by
> fixing #9740.

Check with 'ctest -L suspended' (my local version)
Now the following tests (previously not passing) pass:
390 - INVERTED_SEE-README.ctest_export/doc/Math_pdf4_texF (Failed)
702 - INVERTED_SEE-README.ctest_export/doc/de/Math_pdf4_texF (Failed)
870 - INVERTED_SEE-README.ctest_export/doc/es/Math_pdf4_texF (Failed)
1052 - INVERTED_SEE-README.ctest_export/doc/fr/Math_pdf4_texF (Failed)
1787 - INVERTED_SEE-README.ctest_export/examples/PDF-comment_pdf4_texF 
(Failed)

They can now be removed from revertedTests.

checking now with 'ctest -L export -E '(lyx16|xhtml)$'
-> 94% tests passed, 179 tests failed out of 3041

That is, 29 tests are now better than before.
Checking the output, all 29 are of type pdf4_texF (XeTeX + TeX fonts).

I'd say, please commit.

> Günter

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Scott Kostyshak]

On Mon, Nov 02, 2015 at 10:40:27PM +, Guenter Milde wrote:
> On 2015-11-01, Guenter Milde wrote:
> > Dear Scott,
> 
> > thanks for the patch.
> 
> I modified it to use UTF8 with XeTeX without the 
> \inputencoding commands. A short test here showed that this
> helps with Umlauts or Cyrillic charactes in the PDF Info.
> 
> Give it a try.

Nice. I am not able to test it because I don't understand the underlying
issue, but I did look at the code and it looks good to me. I have a
couple picky comments about using tabs instead of spaces below (by the
way, there are ways of configuring Git to catch these types of
formatting inconsistencies, as well as your editor. Ask on the list if
curious).

> + if (inputenc_loaded)

replace spaces with tab?

> + if (inputenc_loaded)

replace spaces with tabs?


I would say go ahead and commit if you are satisfied with it. From what I
understand, the patch is now a complete patch in the sense that it is a full
step forward. I know there are other things you want to do, but I think this
patch is a self-contained step forward and not a partial patch. If that's
correct, then I suggest you commit.

Let me know if you want me to run the export tests before/after.

Scott

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Guenter Milde]

On 2015-11-01, Guenter Milde wrote:
> Dear Scott,

> thanks for the patch.

I modified it to use UTF8 with XeTeX without the 
\inputencoding commands. A short test here showed that this
helps with Umlauts or Cyrillic charactes in the PDF Info.

Give it a try.

Günter

Exec: git 'diff' 2>&1
Dir: /usr/local/src/lyx

diff --git a/src/BufferParams.cpp b/src/BufferParams.cpp
index 537178e..de39309 100644
--- a/src/BufferParams.cpp
+++ b/src/BufferParams.cpp
@@ -1577,7 +1577,7 @@ bool BufferParams::writeLaTeX(otexstream & os, 
LaTeXFeatures & features,
}
 
// handle inputenc etc.
-   writeEncodingPreamble(os, features);
+   bool const inputenc_loaded = writeEncodingPreamble(os, features);
 
// includeonly
if (!features.runparams().includeall && !included_children_.empty()) {
@@ -1903,7 +1903,7 @@ bool BufferParams::writeLaTeX(otexstream & os, 
LaTeXFeatures & features,
 
OutputParams tmp_params = features.runparams();
pdfoptions().writeLaTeX(tmp_params, os,
-   features.isProvided("hyperref"));
+   features.isProvided("hyperref"), 
inputenc_loaded);
// set back for the rest
lyxpreamble.clear();
// correctly break URLs with hyperref and dvi output
@@ -2924,19 +2924,20 @@ docstring BufferParams::getGraphicsDriver(string const 
& package) const
 }
 
 
-void BufferParams::writeEncodingPreamble(otexstream & os,
+bool BufferParams::writeEncodingPreamble(otexstream & os,
 LaTeXFeatures & features) const
 {
// "inputenc" package not required with non-TeX fonts.
if (useNonTeXFonts)
-   return;
+   return false;
// "inputenc"  fails with XeTeX (even in 8-bit compatiblitly mode) and 
with TeX fonts,
// (this is a bug in the "inputenc" package see #9740).
if (features.runparams().flavor == OutputParams::XETEX)
-   return;
+   return false;
// For LuaTeX with TeX fonts, we can load
// the "luainputenc" package with the specified encoding(s) (see below).
 
+   bool use_inputenc = features.isProvided("inputenc");
if (inputenc == "auto") {
string const doc_encoding =
language->encoding()->latexName();
@@ -2973,6 +2974,7 @@ void BufferParams::writeEncodingPreamble(otexstream & os,
os << "]{luainputenc}\n";
else
os << "]{inputenc}\n";
+   use_inputenc = true;
}
if (package == Encoding::CJK || features.mustProvide("CJK")) {
if (language->encoding()->name() == "utf8-cjk"
@@ -2998,6 +3000,7 @@ void BufferParams::writeEncodingPreamble(otexstream & os,
os << "]{luainputenc}\n";
else
os << "]{inputenc}\n";
+   use_inputenc = true;
break;
case Encoding::CJK:
if (encoding().name() == "utf8-cjk"
@@ -3017,6 +3020,7 @@ void BufferParams::writeEncodingPreamble(otexstream & os,
os << "\\usepackage{CJK}\n";
}
}
+   return use_inputenc;
 }
 
 
diff --git a/src/BufferParams.h b/src/BufferParams.h
index b5ab65e..e3ff4f1 100644
--- a/src/BufferParams.h
+++ b/src/BufferParams.h
@@ -412,7 +412,8 @@ public:
/// return supported drivers for specific packages
docstring getGraphicsDriver(std::string const & package) const;
/// handle inputenc etc.
-   void writeEncodingPreamble(otexstream & os, LaTeXFeatures & features) 
const;
+/// return true if (lua)inputenc is written here or already provided
+   bool writeEncodingPreamble(otexstream & os, LaTeXFeatures & features) 
const;
///
std::string const parseFontName(std::string const & name) const;
/// set up the document fonts
diff --git a/src/PDFOptions.cpp b/src/PDFOptions.cpp
index 6646888..b4e91df 100644
--- a/src/PDFOptions.cpp
+++ b/src/PDFOptions.cpp
@@ -90,7 +90,7 @@ void PDFOptions::writeFile(ostream & os) const
 
 
 void PDFOptions::writeLaTeX(OutputParams & runparams, otexstream & os,
-   bool hyperref_already_provided) const
+   bool hyperref_already_provided, bool 
inputenc_loaded) const
 {
// FIXME Unicode
string opt;
@@ -176,20 +176,21 @@ void PDFOptions::writeLaTeX(OutputParams & runparams, 
otexstream & os,
 
// hyperref expects LICR macros for non-ASCII chars. With Xe/LuaTeX 
utf-8 works, too.
// Usually, "(lua)inputenc" converts the input to LICR.
-   // As hyperref provides good coverage for \inputencoding{utf8}, we can 
try
-   // this if the current input encoding 

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Kornel Benko]

Am Sonntag, 1. November 2015 um 11:52:26, schrieb Kornel Benko 
> Am Samstag, 31. Oktober 2015 um 19:11:10, schrieb Scott Kostyshak 
> 
> > On Sat, Oct 31, 2015 at 05:57:46PM -0400, Scott Kostyshak wrote:
> > > On Sat, Oct 31, 2015 at 09:21:15PM +, Guenter Milde wrote:
> > 
> > > > I'd prefer you to do this.
> > > 
> > > OK I'll work on it.
> > 
> > Does the attached patch look good?
> > 
> > We need to do something similar for the other FIXMEs you put in
> > 1523fc60, right?
> > 
> > Scott
> 
> Applying this patch now _all_ export tests 'export/.*(pdf|dvi)' are failing.
> Probably because the patch is only partial I suppose.
> 

I was too fast. The first 208 (out of 3761) did not pass as I wrote the mail.
Now I see some other passing.
I will report the outcome, have to wait a 'little' more.

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Kornel Benko]

Am Sonntag, 1. November 2015 um 11:55:47, schrieb Kornel Benko 
> Am Sonntag, 1. November 2015 um 11:52:26, schrieb Kornel Benko 
> 
> > Am Samstag, 31. Oktober 2015 um 19:11:10, schrieb Scott Kostyshak 
> > 
> > > On Sat, Oct 31, 2015 at 05:57:46PM -0400, Scott Kostyshak wrote:
> > > > On Sat, Oct 31, 2015 at 09:21:15PM +, Guenter Milde wrote:
> > > 
> > > > > I'd prefer you to do this.
> > > > 
> > > > OK I'll work on it.
> > > 
> > > Does the attached patch look good?
> > > 
> > > We need to do something similar for the other FIXMEs you put in
> > > 1523fc60, right?
> > > 
> > > Scott
> > 
> > Applying this patch now _all_ export tests 'export/.*(pdf|dvi)' are failing.
> > Probably because the patch is only partial I suppose.
> > 
> 
> I was too fast. The first 208 (out of 3761) did not pass as I wrote the mail.
> Now I see some other passing.
> I will report the outcome, have to wait a 'little' more.
> 

It turned out, that all but the first 208 testcases pass.
Comparing the failed with the list of previously failed, there is no difference.

So even if your patch does not improve any test, it does not harm either.

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Scott Kostyshak]

On Sun, Nov 01, 2015 at 01:13:39PM +0100, Kornel Benko wrote:
> Am Sonntag, 1. November 2015 um 11:55:47, schrieb Kornel Benko 
> 
> > Am Sonntag, 1. November 2015 um 11:52:26, schrieb Kornel Benko 
> > 
> > > Am Samstag, 31. Oktober 2015 um 19:11:10, schrieb Scott Kostyshak 
> > > 
> > > > On Sat, Oct 31, 2015 at 05:57:46PM -0400, Scott Kostyshak wrote:
> > > > > On Sat, Oct 31, 2015 at 09:21:15PM +, Guenter Milde wrote:
> > > > 
> > > > > > I'd prefer you to do this.
> > > > > 
> > > > > OK I'll work on it.
> > > > 
> > > > Does the attached patch look good?
> > > > 
> > > > We need to do something similar for the other FIXMEs you put in
> > > > 1523fc60, right?
> > > > 
> > > > Scott
> > > 
> > > Applying this patch now _all_ export tests 'export/.*(pdf|dvi)' are 
> > > failing.
> > > Probably because the patch is only partial I suppose.
> > > 
> > 
> > I was too fast. The first 208 (out of 3761) did not pass as I wrote the 
> > mail.
> > Now I see some other passing.
> > I will report the outcome, have to wait a 'little' more.
> > 

Thanks for running the tests, Kornel.

> It turned out, that all but the first 208 testcases pass.
> Comparing the failed with the list of previously failed, there is no 
> difference.

I think that ctest always runs the failed tests first. This has
surprised me before.

> So even if your patch does not improve any test, it does not harm either.

OK let's wait and see what Günter says then. I'm not even confident my
patch is correct. If he confirms that it is, then I can go on and make
the full patch. I'm not sure if the full patch is supposed to fix any
tests (although it would be nice if it whittles down some of those 208).

Scott

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Guenter Milde]

Dear Scott,

thanks for the patch.

On 2015-10-31, Scott Kostyshak wrote:
> On Sat, Oct 31, 2015 at 05:57:46PM -0400, Scott Kostyshak wrote:
>> On Sat, Oct 31, 2015 at 09:21:15PM +, Guenter Milde wrote:

> Does the attached patch look good?

It looks like a step in the right direction. However, this is just a
prerequisite for solving the encoding problems, not the fix yet.

Therefore, it is to be expected that this patch does not cure any of the
export tests (they just fail for another reason).

> We need to do something similar for the other FIXMEs you put in
> 1523fc60, right?

This depends on what you mean with "similar". AFAIK, the problem there is
not about PDFstrings and hence the solution also different.

> Scott

> [-- Type: text/plain, Encoding: , Filename: 
> 0001-Improve-XeTeX-LuaTeX-with-TeX-fonts-9740.patch --]

> From ef7d21fc35679683f537c186ce9200cba232a8b3 Mon Sep 17 00:00:00 2001
> From: Scott Kostyshak 
> Date: Sat, 31 Oct 2015 18:59:23 -0400
> Subject: [PATCH] Improve XeTeX/LuaTeX with TeX fonts, #9740

This is not about XeTex/LuaTeX with TeX fonts:

The problem fixed is "ensure \inputencoding is defined before using it in
the document preamble".

Conditions where this applies are "inputencoding ascii", "inputencoding
(default without inputenc)" and "cjk without utf8" and a non-encodable
character in the PDF info (most of them are currently not tested).

* As XeTeX with TeX fonts requires "inputencoding ascii", it is
  affected indirectly. 

* LuaTeX works fine (as it loads luainputenc which defines \inputencoding).


> More fixes are needed to solve the FIXMEs in 1523fc60.

More fixes are needed to solve the errors with non-encodable characters in
the PDF info fields.

> --- a/src/BufferParams.cpp
> +++ b/src/BufferParams.cpp
> @@ -1577,7 +1577,8 @@ bool BufferParams::writeLaTeX(otexstream & os, 
> LaTeXFeatures & features,
>   }

>   // handle inputenc etc.
> - writeEncodingPreamble(os, features);
> + bool const ie = writeEncodingPreamble(os, features);
> + bool const inputenc_loaded = ie || features.isProvided("inputenc");

As features.isProvided("inputenc") is also tested in writeEncodingPreamble,
we could move this test there and write here simply:

bool const inputenc_loaded = writeEncodingPreamble(os, features);


> -void BufferParams::writeEncodingPreamble(otexstream & os,
> +bool BufferParams::writeEncodingPreamble(otexstream & os,
>LaTeXFeatures & features) const
>  {
>   // "inputenc" package not required with non-TeX fonts.
>   if (useNonTeXFonts)
> - return;
> + return false;
>   // "inputenc"  fails with XeTeX (even in 8-bit compatiblitly mode) and 
> with TeX fonts,
>   // (this is a bug in the "inputenc" package see #9740).
>   if (features.runparams().flavor == OutputParams::XETEX)
> - return;
> + return false;
>   // For LuaTeX with TeX fonts, we can load
>   // the "luainputenc" package with the specified encoding(s) (see below).

> + bool ret = false;

Why not 

bool inputenc_provided = features.isProvided("inputenc");

>   if (inputenc == "auto") {
>   string const doc_encoding =
>   language->encoding()->latexName();
> @@ -2955,6 +2957,7 @@ void BufferParams::writeEncodingPreamble(otexstream & 
> os,
>   && !features.isRequired("japanese")
>   && !features.isProvided("inputenc")) {
>   os << "\\usepackage[";
> + ret = true;
>   set::const_iterator it = encodings.begin();
>   set::const_iterator const end = encodings.end();
>   if (it != end) {

Place the "ret = ..." line rather below the complete loading command, not
inbetween.

> @@ -2993,6 +2996,7 @@ void BufferParams::writeEncodingPreamble(otexstream & 
> os,
>   || features.isProvided("inputenc"))
>   break;
>   os << "\\usepackage[" << 
> from_ascii(encoding().latexName());
> + ret = true;
>   if (features.runparams().flavor == OutputParams::LUATEX
>   || features.runparams().flavor == 
> OutputParams::DVILUATEX)
>   os << "]{luainputenc}\n";
> @@ -3017,6 +3021,7 @@ void BufferParams::writeEncodingPreamble(otexstream & 
> os,
>   os << "\\usepackage{CJK}\n";
>   }
>   }
> + return ret;
>  }

...


>  void PDFOptions::writeLaTeX(OutputParams & runparams, otexstream & os,
> - bool hyperref_already_provided) const
> + bool hyperref_already_provided, bool 
> inputenc_loaded) const
>  {
>   // FIXME Unicode
>   string opt;
> @@ -178,15 +178,12 @@ void PDFOptions::writeLaTeX(OutputParams & runparams, 
> otexstream 

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Guenter Milde]

On 2015-10-30, Scott Kostyshak wrote:
> On Fri, Oct 30, 2015 at 10:45:38PM +, Guenter Milde wrote:
>> On 2015-10-30, Scott Kostyshak wrote:
>> > On Thu, Oct 29, 2015 at 09:12:04AM +, Guenter Milde wrote:
>> >> On 2015-10-29, Scott Kostyshak wrote:

> OK so it seems we're on the same page. The next question is what
> information exactly do you need from writeEncodingPreamble? Do you just
> need a bool 'use_inputenc' that is true if either
> \usepackage[...]{inputenc} or \usepackage[...]{luainputenc} is output by
> writeEncodingPreamble?

The important thing is to know, whether it is safe to use the
\inputencoding macro. This could be a boolean value that is true if:

* \usepackage[...]{inputenc} or \usepackage[...]{luainputenc} is output by
  writeEncodingPreamble
  
* a package/class that loads inputenc (or luainputenc) is loaded
  (this is tested in writeEncodingPreamble via
  features.isProvided("inputenc")).


> Do you want to do the C++ work or would you prefer for me to? I think it
> should be simple but I'm happy to do the work since I've finally wrapped
> my head around the issue.

I'd prefer you to do this.

Deciding, what to do in case inputenc is not availabel remains an open
issue: with XeTeX, we can just use UTF8, otherwise we have to decide
between conversion with lib/unicodesymbols (or something better suited
for "pdfstrings") or giving a warning or error.


Thanks,

Günter

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Scott Kostyshak]

On Sat, Oct 31, 2015 at 09:21:15PM +, Guenter Milde wrote:
> On 2015-10-30, Scott Kostyshak wrote:
> > On Fri, Oct 30, 2015 at 10:45:38PM +, Guenter Milde wrote:
> >> On 2015-10-30, Scott Kostyshak wrote:
> >> > On Thu, Oct 29, 2015 at 09:12:04AM +, Guenter Milde wrote:
> >> >> On 2015-10-29, Scott Kostyshak wrote:
> 
> > OK so it seems we're on the same page. The next question is what
> > information exactly do you need from writeEncodingPreamble? Do you just
> > need a bool 'use_inputenc' that is true if either
> > \usepackage[...]{inputenc} or \usepackage[...]{luainputenc} is output by
> > writeEncodingPreamble?
> 
> The important thing is to know, whether it is safe to use the
> \inputencoding macro. This could be a boolean value that is true if:
> 
> * \usepackage[...]{inputenc} or \usepackage[...]{luainputenc} is output by
>   writeEncodingPreamble
>   
> * a package/class that loads inputenc (or luainputenc) is loaded
>   (this is tested in writeEncodingPreamble via
>   features.isProvided("inputenc")).

Thanks. I have a clear idea of what is needed now.

> > Do you want to do the C++ work or would you prefer for me to? I think it
> > should be simple but I'm happy to do the work since I've finally wrapped
> > my head around the issue.
> 
> I'd prefer you to do this.

OK I'll work on it.

> Deciding, what to do in case inputenc is not availabel remains an open
> issue: with XeTeX, we can just use UTF8, otherwise we have to decide
> between conversion with lib/unicodesymbols (or something better suited
> for "pdfstrings") or giving a warning or error.

I see. This is in essence the (currently non-existing) "else" after the
"if" conditions that we've been focusing on in PDFOptions.cpp. Right?

Scott

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Scott Kostyshak]

On Sat, Oct 31, 2015 at 05:57:46PM -0400, Scott Kostyshak wrote:
> On Sat, Oct 31, 2015 at 09:21:15PM +, Guenter Milde wrote:

> > I'd prefer you to do this.
> 
> OK I'll work on it.

Does the attached patch look good?

We need to do something similar for the other FIXMEs you put in
1523fc60, right?

Scott
>From ef7d21fc35679683f537c186ce9200cba232a8b3 Mon Sep 17 00:00:00 2001
From: Scott Kostyshak 
Date: Sat, 31 Oct 2015 18:59:23 -0400
Subject: [PATCH] Improve XeTeX/LuaTeX with TeX fonts, #9740

More fixes are needed to solve the FIXMEs in 1523fc60.
---
 src/BufferParams.cpp | 15 ++-
 src/BufferParams.h   |  3 ++-
 src/PDFOptions.cpp   | 18 +++---
 src/PDFOptions.h |  2 +-
 4 files changed, 20 insertions(+), 18 deletions(-)

diff --git a/src/BufferParams.cpp b/src/BufferParams.cpp
index 537178e..0fd747b 100644
--- a/src/BufferParams.cpp
+++ b/src/BufferParams.cpp
@@ -1577,7 +1577,8 @@ bool BufferParams::writeLaTeX(otexstream & os, 
LaTeXFeatures & features,
}
 
// handle inputenc etc.
-   writeEncodingPreamble(os, features);
+   bool const ie = writeEncodingPreamble(os, features);
+   bool const inputenc_loaded = ie || features.isProvided("inputenc");
 
// includeonly
if (!features.runparams().includeall && !included_children_.empty()) {
@@ -1903,7 +1904,7 @@ bool BufferParams::writeLaTeX(otexstream & os, 
LaTeXFeatures & features,
 
OutputParams tmp_params = features.runparams();
pdfoptions().writeLaTeX(tmp_params, os,
-   features.isProvided("hyperref"));
+   features.isProvided("hyperref"), 
inputenc_loaded);
// set back for the rest
lyxpreamble.clear();
// correctly break URLs with hyperref and dvi output
@@ -2924,19 +2925,20 @@ docstring BufferParams::getGraphicsDriver(string const 
& package) const
 }
 
 
-void BufferParams::writeEncodingPreamble(otexstream & os,
+bool BufferParams::writeEncodingPreamble(otexstream & os,
 LaTeXFeatures & features) const
 {
// "inputenc" package not required with non-TeX fonts.
if (useNonTeXFonts)
-   return;
+   return false;
// "inputenc"  fails with XeTeX (even in 8-bit compatiblitly mode) and 
with TeX fonts,
// (this is a bug in the "inputenc" package see #9740).
if (features.runparams().flavor == OutputParams::XETEX)
-   return;
+   return false;
// For LuaTeX with TeX fonts, we can load
// the "luainputenc" package with the specified encoding(s) (see below).
 
+   bool ret = false;
if (inputenc == "auto") {
string const doc_encoding =
language->encoding()->latexName();
@@ -2955,6 +2957,7 @@ void BufferParams::writeEncodingPreamble(otexstream & os,
&& !features.isRequired("japanese")
&& !features.isProvided("inputenc")) {
os << "\\usepackage[";
+   ret = true;
set::const_iterator it = encodings.begin();
set::const_iterator const end = encodings.end();
if (it != end) {
@@ -2993,6 +2996,7 @@ void BufferParams::writeEncodingPreamble(otexstream & os,
|| features.isProvided("inputenc"))
break;
os << "\\usepackage[" << 
from_ascii(encoding().latexName());
+   ret = true;
if (features.runparams().flavor == OutputParams::LUATEX
|| features.runparams().flavor == 
OutputParams::DVILUATEX)
os << "]{luainputenc}\n";
@@ -3017,6 +3021,7 @@ void BufferParams::writeEncodingPreamble(otexstream & os,
os << "\\usepackage{CJK}\n";
}
}
+   return ret;
 }
 
 
diff --git a/src/BufferParams.h b/src/BufferParams.h
index b5ab65e..328ca17 100644
--- a/src/BufferParams.h
+++ b/src/BufferParams.h
@@ -412,7 +412,8 @@ public:
/// return supported drivers for specific packages
docstring getGraphicsDriver(std::string const & package) const;
/// handle inputenc etc.
-   void writeEncodingPreamble(otexstream & os, LaTeXFeatures & features) 
const;
+/// return true if (lua)inputenc is written here
+   bool writeEncodingPreamble(otexstream & os, LaTeXFeatures & features) 
const;
///
std::string const parseFontName(std::string const & name) const;
/// set up the document fonts
diff --git a/src/PDFOptions.cpp b/src/PDFOptions.cpp
index 6646888..2ecc5c6 100644
--- a/src/PDFOptions.cpp
+++ b/src/PDFOptions.cpp
@@ -90,7 +90,7 @@ void PDFOptions::writeFile(ostream & os) const
 
 
 void 

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Scott Kostyshak]

On Fri, Oct 30, 2015 at 10:45:38PM +, Guenter Milde wrote:
> On 2015-10-30, Scott Kostyshak wrote:
> > On Thu, Oct 29, 2015 at 09:12:04AM +, Guenter Milde wrote:
> >> On 2015-10-29, Scott Kostyshak wrote:
> 
> ...
> 
> >   words, the only place that knows whether the inputenc package is
> >   loaded is BufferParams::writeEncodingPreamble()
> 
> This is my understanding, too.
> 
> > Actually, after thinking about this a bit longer my new favorite
> > approach is simply for BufferParams::writeEncodingPreamble to return the
> > necessary information (e.g. it could return a boolean using_inputenc)
> > that is then just passed to pdfoptions().writeLaTeX(). That seems the
> > cleanest and safest approach. I thought that it might not be common for
> > a write method to return such information (I figured if anything it
> > would return success/fail), but actually we already do this:
> > BufferParams::writeLaTeX() returns a boolean use_babel.
> 
> Good to know, that there is a precedence.

OK so it seems we're on the same page. The next question is what
information exactly do you need from writeEncodingPreamble? Do you just
need a bool 'use_inputenc' that is true if either
\usepackage[...]{inputenc} or \usepackage[...]{luainputenc} is output by
writeEncodingPreamble?

Do you want to do the C++ work or would you prefer for me to? I think it
should be simple but I'm happy to do the work since I've finally wrapped
my head around the issue.

Scott

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Guenter Milde]

On 2015-10-30, Scott Kostyshak wrote:
> On Thu, Oct 29, 2015 at 09:12:04AM +, Guenter Milde wrote:
>> On 2015-10-29, Scott Kostyshak wrote:

...

>   words, the only place that knows whether the inputenc package is
>   loaded is BufferParams::writeEncodingPreamble()

This is my understanding, too.

> Actually, after thinking about this a bit longer my new favorite
> approach is simply for BufferParams::writeEncodingPreamble to return the
> necessary information (e.g. it could return a boolean using_inputenc)
> that is then just passed to pdfoptions().writeLaTeX(). That seems the
> cleanest and safest approach. I thought that it might not be common for
> a write method to return such information (I figured if anything it
> would return success/fail), but actually we already do this:
> BufferParams::writeLaTeX() returns a boolean use_babel.

Good to know, that there is a precedence.


Thanks,

Günter

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Scott Kostyshak]

On Thu, Oct 29, 2015 at 09:12:04AM +, Guenter Milde wrote:
> On 2015-10-29, Scott Kostyshak wrote:

> As my FIXME was wrong, so is your patch.

I see.

> > In regards to "Dont forget to keep this check in sync with the check
> > above!" this seems to be a good situation for a helper function in order
> > to share code.
> 
> Maybe define a new setting/helper_function "inputenc package available"
> in BufferParams.cpp?

I'm starting to understand more the situation and what you want. Here is
what I think I understand:

- the BufferParams inputenc member is not what you want because that is
  just the setting in the document. It is a complicated process (as
  shown by BufferParams::writeEncodingPreamble()) to figure out what is
  actually done.

- you cannot use features.isRequired("inputenc") because even if
  that is false, it seems that the inputenc package could still be
  loaded, as is done in BufferParams::writeEncodingPreamble(). In other
  words, the only place that knows whether the inputenc package is
  loaded is BufferParams::writeEncodingPreamble() and it does not store
  the information anywhere, it just writes to the preamble.

(EDIT for reader: skip to the paragraph starting with "Actually" unless
you want to see the clumsy thought process leading up to that)

So the solution might be to factor out the code that determines whether
to load the inputenc package from BufferParams::writeEncodingPreamble()
in a place that would be acccessible both by
BufferParams::writeEncodingPreamble() and PDFOptions::writeLaTeX().

writeEncodingPreamble() and pdfoptions().writeLaTeX() are both called in
BufferParams::writeLaTeX(), so one possibility is that instead of
writeEncodingPreamble() doing the necessary steps to determine whether
to load the inputenc package, that could be done in
BufferParams::writeLaTeX(), either through a helper function (maybe
named something like determineEncodingPreamble() ?) or just directly
(but I think a helper function would be preferred). For example, we
could create a boolean using_inputenc that is local to
BufferParams::writeLaTeX() and then pass that boolean to
writeEncodingPreamble() and pdfoptions().writeLaTeX().

Another possibility is to store the information in the BufferParams
class. If we do this, I imagine we should not write new information from
bufferParams::writeLaTeX(). At least, I would not expect it to write any
new members judging from its name. So we would have to factor out the
code further.

Actually, after thinking about this a bit longer my new favorite
approach is simply for BufferParams::writeEncodingPreamble to return the
necessary information (e.g. it could return a boolean using_inputenc)
that is then just passed to pdfoptions().writeLaTeX(). That seems the
cleanest and safest approach. I thought that it might not be common for
a write method to return such information (I figured if anything it
would return success/fail), but actually we already do this:
BufferParams::writeLaTeX() returns a boolean use_babel.

Scott

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Guenter Milde]

On 2015-10-29, Scott Kostyshak wrote:
> On Tue, Oct 20, 2015 at 07:16:44PM +0200, Günter Milde wrote:
>> commit 1523fc6023440f10ca0d82a681ded5c060d8fd33
>> Author: Günter Milde 
>> Date:   Tue Oct 20 19:14:39 2015 +0200

>> Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

>> Fixes output for 3 of the 4 test lyx-files.

>> Includes "FIXME"s at places where further action is required to get the 
>> XeTeX
>> export right but I don't know how.

> I'm not sure if you know or expect this (e.g. because of the FIXMEs), but this
> commit broke some of the tests for PDF-comment.lyx. It is not actually 
> expected
> that this file should be compiled successfully with XeTeX or LuaTeX but it 
> used
> to work (perhaps by luck?). 

At least, I am not surprised. I know there are cases that fail now but
worked before, especially

  XeTeX with TeX-fonts and non-ASCII characters in the PDFinfo:
  
* before, the inputencoding was utf8, which is 
  + fine for the PDFinfo but
  - wrong with non-ASCII characters in the text (either compilation error or
wrong characters in the output).  
  
* with the partial patch (fix 2), the inputencoding is set to "ascii",
  which is
  + correct for the text
  - fails with non-ASCII characters in the PDFinfo
  
> Do you think it is worth taking a look at why the tests started failing
> or should we forget about it in this case because the file is supposed
> to be compiled with pdfTeX?

My outlined FIXME would correct this (using ascii for the text and utf8 for
the PDFinfo. Until some kind soul with proper understanding is going to
complete it, I propose to 

* open a bug report for non-ASCII characters in the PDFinfo with ASCII
  encoding (also fails with 8-bit TeX) (I could to this.)
  
* invert the tests with a pointer to this ticket.



> It seems that XeTeX with system fonts was broken by this commit but
> then it was fixed later on. The same thing happened for a couple of the
> other tests. So I guess we don't have to look at those. The following,
> however, were broken by this commit and the results from master
> indicate that they're still broken:

> export/examples/PDF-comment_pdf5_texF (Failed)
> export/examples/de/PDF-comment_pdf4_texF (Failed)
> export/examples/de/PDF-comment_pdf5_texF (Failed)
> export/examples/fr/PDF-comment_pdf5_texF (Failed)

> Recall that pdf5 refers to compiling with LuaTeX and pdf4 refers to
> compiling with XeTeX. All of the four above tests are regarding TeX
> fonts. 

With XeTeX, the problem is the non-ASCII ö (o-umlaut) in the Author field.

With LuaTeX, the error is unrelated (at least here):

LaTeX info: "xparse/define-command"
. 
. Defining command \selectchemgreekmapping with sig. 'm' on line 485.
.
! Argument of \__str_case_end:nw has an extra }.
 
\par 
l.573 \chemgreek_drop_symbols:

> I have checked in the commit before this one that the PDFs of
> the above four compilations look OK to me at first look (i.e. not
> garbled from what I see).

Before which commit?


Günter

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Guenter Milde]

On 2015-10-29, Scott Kostyshak wrote:
> On Wed, Oct 28, 2015 at 10:28:00PM +, Guenter Milde wrote:
>> On 2015-10-28, Scott Kostyshak wrote:
>> > On Wed, Oct 28, 2015 at 09:31:50AM +, Guenter Milde wrote:
>> >> On 2015-10-26, Guenter Milde wrote:
>> >> > On 2015-10-26, Scott Kostyshak wrote:
>> >> >> Has anything changed in the last couple of days?

>> >> I'l commit an updated FIXME.

>> > This does indeed sound tricky. Can you figure out the necessary items to
>> > condition on from BufferParams or is not everything stored in there?

>> It became a bit simpler after going through the alternatives:

>> In pseudo-code:

> I looked at the pseudo-code but since I don't know this code well it's
> not clear if you still need my help. If you have access to the buffer
> params (which my patch gave), is that enough? Can you test with the
> patch I posted before? Or is something else missing?

As my FIXME was wrong, so is your patch.
In principle, I may be able to use your patch and combine it with my
comments, but alas, I don't know this code either, nor do I know C++, so it
would take too much time and the result would not be clean and I'd rather
leave this to a more eligible person.

> In regards to "Dont forget to keep this check in sync with the check
> above!" this seems to be a good situation for a helper function in order
> to share code.

Maybe define a new setting/helper_function "inputenc package available"
in BufferParams.cpp?

Thanks,

Günter

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Scott Kostyshak]

On Wed, Oct 28, 2015 at 09:31:50AM +, Guenter Milde wrote:
> On 2015-10-26, Guenter Milde wrote:
> > On 2015-10-26, Scott Kostyshak wrote:
> >> Has anything changed in the last couple of days?
> 
> Today, I found out that the problem is tricky:
> 
> As the PDF-Info does not use TeX fonts, Xe/LuaTeX handling (in principle)
> does not depend on the "use-nonTeXfonts" setting. However, as we use
> "luainputenc" with LuaTeX and 8-bit encodings, we have to differentiate:
> 
> LuaTeX: if "luainputenc" is used (i.e. with 8-bit input encodings) 
> this is called before the hyperref setup and hence the 8-bit
>   encoding is active -- "luainputenc" transformes the bytes into LICR --
>   and we need the same \inputenc wrapper as with 8-bit TeX.
>   
> XeTeX:  does not use "inputenc", the input encoding is utf-8 
> (but we need to export LICR ("ascii") for the document parts that
>   are typeset with TeX fonts).
>   
>   "hyperref" with XeTeX and "raw utf8" and works fine with any Unicode 
> character.
>   
>   However, we need the 
>  << setEncoding("UTF-8");
>   (without the 
>   os << "\\inputencoding{utf8}\n"
> )
> For XeTeX and non-TeX fonts.
> 
> 
> I'l commit an updated FIXME.
> 
> > It may well be, that the FIXME in PDFOptions is not enaugh - maybe we have
> > to push the content through unicodesymbols conversion, too.
> 
> This is true for "latexencoding"="ascii" with 8-bit TeX, see my other post.

This does indeed sound tricky. Can you figure out the necessary items to
condition on from BufferParams or is not everything stored in there?

Scott

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Scott Kostyshak]

On Wed, Oct 28, 2015 at 10:28:00PM +, Guenter Milde wrote:
> On 2015-10-28, Scott Kostyshak wrote:
> > On Wed, Oct 28, 2015 at 09:31:50AM +, Guenter Milde wrote:
> >> On 2015-10-26, Guenter Milde wrote:
> >> > On 2015-10-26, Scott Kostyshak wrote:
> >> >> Has anything changed in the last couple of days?
> 
> >> I'l commit an updated FIXME.
> 
> > This does indeed sound tricky. Can you figure out the necessary items to
> > condition on from BufferParams or is not everything stored in there?
> 
> It became a bit simpler after going through the alternatives:
> 
> In pseudo-code:

I looked at the pseudo-code but since I don't know this code well it's
not clear if you still need my help. If you have access to the buffer
params (which my patch gave), is that enough? Can you test with the
patch I posted before? Or is something else missing?

In regards to "Dont forget to keep this check in sync with the check
above!" this seems to be a good situation for a helper function in order
to share code.

Scott

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Scott Kostyshak]

On Tue, Oct 20, 2015 at 07:16:44PM +0200, Günter Milde wrote:
> commit 1523fc6023440f10ca0d82a681ded5c060d8fd33
> Author: Günter Milde 
> Date:   Tue Oct 20 19:14:39 2015 +0200
> 
> Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
> 
> Fixes output for 3 of the 4 test lyx-files.
> 
> Includes "FIXME"s at places where further action is required to get the 
> XeTeX
> export right but I don't know how.

I'm not sure if you know or expect this (e.g. because of the FIXMEs), but this
commit broke some of the tests for PDF-comment.lyx. It is not actually expected
that this file should be compiled successfully with XeTeX or LuaTeX but it used
to work (perhaps by luck?). Do you think it is worth taking a look at why the
tests started failing or should we forget about it in this case because the
file is supposed to be compiled with pdfTeX?

In case you are interested:

The command
ctest -R "comment"
gives me the following failed tests just *before* this commit (which is commit
9d7c3a48):

INVERTED_SEE-README.ctest_export/examples/PDF-comment_dvi (Failed)
INVERTED_SEE-README.ctest_export/examples/PDF-comment_pdf (Failed)
INVERTED_SEE-README.ctest_export/examples/PDF-comment_pdf3 (Failed)
INVERTED_SEE-README.ctest_export/examples/PDF-comment_pdf4_systemF 
(Failed)
export/examples/de/PDF-comment_dvi3_texF (Failed)
export/examples/de/PDF-comment_dvi3_systemF (Failed)
export/examples/fr/PDF-comment_dvi3_texF (Failed)
export/examples/fr/PDF-comment_dvi3_systemF (Failed)
export/examples/fr/PDF-comment_pdf4_texF (Failed)

With this commit (1523fc60), the same command gives the following:

INVERTED_SEE-README.ctest_export/examples/PDF-comment_dvi (Failed)
INVERTED_SEE-README.ctest_export/examples/PDF-comment_pdf (Failed)
INVERTED_SEE-README.ctest_export/examples/PDF-comment_pdf3 (Failed)
export/examples/PDF-comment_pdf5_texF (Failed)
export/examples/de/PDF-comment_dvi3_texF (Failed)
export/examples/de/PDF-comment_dvi3_systemF (Failed)
export/examples/de/PDF-comment_pdf4_texF (Failed)
export/examples/de/PDF-comment_pdf4_systemF (Failed)
export/examples/de/PDF-comment_pdf5_texF (Failed)
export/examples/fr/PDF-comment_dvi3_texF (Failed)
export/examples/fr/PDF-comment_dvi3_systemF (Failed)
export/examples/fr/PDF-comment_pdf4_texF (Failed)
export/examples/fr/PDF-comment_pdf4_systemF (Failed)
export/examples/fr/PDF-comment_pdf5_texF (Failed)

Finally, compiling with current master (02db1833) gives:

INVERTED_SEE-README.ctest_export/examples/PDF-comment_dvi (Failed)
INVERTED_SEE-README.ctest_export/examples/PDF-comment_pdf (Failed)
INVERTED_SEE-README.ctest_export/examples/PDF-comment_pdf3 (Failed)
INVERTED_SEE-README.ctest_export/examples/PDF-comment_pdf4_systemF 
(Failed)
export/examples/PDF-comment_pdf5_texF (Failed)
export/examples/de/PDF-comment_dvi3_texF (Failed)
export/examples/de/PDF-comment_dvi3_systemF (Failed)
export/examples/de/PDF-comment_pdf4_texF (Failed)
export/examples/de/PDF-comment_pdf5_texF (Failed)
export/examples/fr/PDF-comment_dvi3_texF (Failed)
export/examples/fr/PDF-comment_dvi3_systemF (Failed)
export/examples/fr/PDF-comment_pdf4_texF (Failed)
export/examples/fr/PDF-comment_pdf5_texF (Failed)

It seems that XeTeX with system fonts was broken by this commit but then it was 
fixed later on. The same thing happened for a couple of the other tests. So I 
guess we don't have to look at those. The following, however, were broken by 
this commit and the results from master indicate that they're still broken:

export/examples/PDF-comment_pdf5_texF (Failed)
export/examples/de/PDF-comment_pdf4_texF (Failed)
export/examples/de/PDF-comment_pdf5_texF (Failed)
export/examples/fr/PDF-comment_pdf5_texF (Failed)

Recall that pdf5 refers to compiling with LuaTeX and pdf4 refers to compiling 
with XeTeX. All of the four above tests are regarding TeX fonts. I have checked 
in the commit before this one that the PDFs of the above four compilations look 
OK to me at first look (i.e. not garbled from what I see).

Scott

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Guenter Milde]

On 2015-10-28, Scott Kostyshak wrote:
> On Wed, Oct 28, 2015 at 09:31:50AM +, Guenter Milde wrote:
>> On 2015-10-26, Guenter Milde wrote:
>> > On 2015-10-26, Scott Kostyshak wrote:
>> >> Has anything changed in the last couple of days?

>> I'l commit an updated FIXME.

> This does indeed sound tricky. Can you figure out the necessary items to
> condition on from BufferParams or is not everything stored in there?

It became a bit simpler after going through the alternatives:

In pseudo-code:

if (PDFinfo has unencodable characters):
  
  if (inputenc or luainputenc is loaded):
os << "\\inputencoding{utf8}\n"
   << setEncoding("UTF-8");
  else if (Xe/LuaTeX) // support UTF-8 also without inputenc
os << setEncoding("UTF-8");
  else
convert with lib/unicodesymbols or report an error
 
Checking for availability of the \inputencoding command is nontrivial. 
I tried to get the relevant info from
BufferParams::writeEncodingPreamble
and put them in an expanded FIXME
(but did not test under which name they are available in PDFOptions.cpp).

Günter

Here is a patch:

Exec: git 'diff' 'PDFOptions.cpp' 2>&1
Dir: /usr/local/src/lyx/src/

diff --git a/src/PDFOptions.cpp b/src/PDFOptions.cpp
index 6646888..729ee10 100644
--- a/src/PDFOptions.cpp
+++ b/src/PDFOptions.cpp
@@ -175,20 +175,25 @@ void PDFOptions::writeLaTeX(OutputParams & runparams, 
otexstream & os,
}
 
// hyperref expects LICR macros for non-ASCII chars. With Xe/LuaTeX 
utf-8 works, too.
-   // Usually, "(lua)inputenc" converts the input to LICR.
+   // Usually, (lua)inputenc converts the input to LICR.
// As hyperref provides good coverage for \inputencoding{utf8}, we can 
try
// this if the current input encoding does not support a character.
-   // FIXME: inputenc (part 1 of 2)
-   //   Replace the "FullUnicode" check with
-   //   check for loading of inputenc or luainputenc package
-   //   (see BufferParams::writeEncodingPreamble and 
runparams.encoding->package()).
-   //   Otherwise \inputencoding is not defined
-   //   (e.g. if "latex-encoding" is set to "ascii").
-   //   Dont forget to keep the check below (part 2) in sync!
-   if (need_unicode && enc && enc->iconvName() != "UTF-8"
-   &&!runparams.isFullUnicode()) {
-   os << "\\inputencoding{utf8}\n"
-  << setEncoding("UTF-8");
+   if (need_unicode && enc && enc->iconvName() != "UTF-8") {
+ // FIXME check for loading of inputenc or luainputenc package
+ //   (See BufferParams::writeEncodingPreamble and 
runparams.encoding->package().)
+ //   Dont forget to keep the check for the re-set below in sync!
+ // if (encoding().package() == Encoding::inputenc
+ //   && inputencoding != "default" && inputencoding != "ascii"
+ // && !encodings.empty() && !features.isRequired("japanese")
+ //) // inputenc or luainputenc package provided
+   os << "\\inputencoding{utf8}\n"
+  << setEncoding("UTF-8");
+ // else if (runparams.isFullUnicode()) // Xe/LuaTeX support UTF-8 
also without inputenc
+ //os << setEncoding("UTF-8");
+ // else
+ //   FIXME: convert with lib/unicodesymbols or report an error!
+ //(cf. the check and report for unencodable character
+ //in the user preamble in BufferParams.cpp)
}
// If hyperref is loaded by the document class, we output
// \hypersetup \AtBeginDocument if hypersetup is not (yet)
@@ -204,11 +209,18 @@ void PDFOptions::writeLaTeX(OutputParams & runparams, 
otexstream & os,
   << "\\fi\n";
} else
os << from_utf8(opt);
-   // FIXME: inputenc (part 2 of 2)
-   if (need_unicode && enc && enc->iconvName() != "UTF-8"
-   &&!runparams.isFullUnicode()) {
+   if (need_unicode && enc && enc->iconvName() != "UTF-8") {
+ // FIXME check for loading of inputenc or luainputenc package
+ //   (See BufferParams::writeEncodingPreamble and 
runparams.encoding->package().)
+ //   Dont forget to keep this check in sync with the check above!
+ // if (encoding().package() == Encoding::inputenc
+ //   && inputencoding != "default" && inputencoding != "ascii"
+ // && !encodings.empty() && !features.isRequired("japanese")
+ //) // inputenc or luainputenc package provided
os << setEncoding(enc->iconvName())
   << "\\inputencoding{" << from_ascii(enc->latexName()) << 
"}\n";
+ // else if (runparams.isFullUnicode()) // Xe/LuaTeX, reset encoding
+ //os << setEncoding(enc->iconvName());
}
 }
 

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Guenter Milde]

On 2015-10-26, Guenter Milde wrote:
> On 2015-10-26, Scott Kostyshak wrote:
>> On Fri, Oct 23, 2015 at 07:31:49PM +, Guenter Milde wrote:

> With the latest 2 fixes (36b9645b5de6d463b4909 and 2aa65fdc8d2cde1ba863),
> all 4 test files compile.

>>> >> >> Includes "FIXME"s at places where further action is required
>>> >> >> to get the XeTeX export right but I don't know how.
>>> >> >> ...

>>> >> >> + &&!runparams.isFullUnicode()) { // FIXME: check must be 
>>> >> >> done for useNonTeXFonts!
>>> >> >>   os << "\\inputencoding{utf8}\n"
>>> >> >>  << setEncoding("UTF-8");

>>> >> > So to make sure I understand what you want with this FIXME is you
>>> >> > would like to do something like params().useNonTeXFonts as you do
>>> >> > in Buffer.cpp but in PDFOptions.cpp it's not clear how to access
>>> >> > that?

>>> >> It may also be that I misunderstand what is done there, but
>>> >> basically, yes: I believe that the test at these places must be for
>>> >> "useNonTeXFonts" instead of "isFullUnicode", because XeTeX/LuaTeX
>>> >> with TeX fonts behave more similar to 8-bit TeX regarding the in-
>>> >> and output encodings.

>>> > OK, what about the attached patch? Can you check that it does what you
>>> > want as far as the two FIXMEs in PDFOptions.cpp? 

...

>> Has anything changed in the last couple of days?

Today, I found out that the problem is tricky:

As the PDF-Info does not use TeX fonts, Xe/LuaTeX handling (in principle)
does not depend on the "use-nonTeXfonts" setting. However, as we use
"luainputenc" with LuaTeX and 8-bit encodings, we have to differentiate:

LuaTeX: if "luainputenc" is used (i.e. with 8-bit input encodings) 
this is called before the hyperref setup and hence the 8-bit
encoding is active -- "luainputenc" transformes the bytes into LICR --
and we need the same \inputenc wrapper as with 8-bit TeX.

XeTeX:  does not use "inputenc", the input encoding is utf-8 
(but we need to export LICR ("ascii") for the document parts that
are typeset with TeX fonts).

"hyperref" with XeTeX and "raw utf8" and works fine with any Unicode 
character.

However, we need the 
   << setEncoding("UTF-8");
(without the 
os << "\\inputencoding{utf8}\n"
)
For XeTeX and non-TeX fonts.


I'l commit an updated FIXME.

> It may well be, that the FIXME in PDFOptions is not enaugh - maybe we have
> to push the content through unicodesymbols conversion, too.

This is true for "latexencoding"="ascii" with 8-bit TeX, see my other post.


Günter

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Kornel Benko]

Am Montag, 26. Oktober 2015 um 01:53:24, schrieb Scott Kostyshak 

> On Sun, Oct 25, 2015 at 01:21:01PM +0100, Kornel Benko wrote:
> > Am Sonntag, 25. Oktober 2015 um 11:31:09, schrieb Guenter Milde 
> > 
> > > On 2015-10-25, Georg Baum wrote:
> > > > Guenter Milde wrote:
> > > 
> > > >> I'll commit the patch for the patch and hope this will solve some more 
> > > >> of
> > > >> the automatic XeTeX tests.
> > > 
> > > > Please do. 
> > > 
> > > It is in.
> > > 
> > > Scott, could you please test if this fixes some more export tests?
> 
> Thanks for asking! Kornel already beat me to it but I will also run the
> tests and let you know the results tomorrow (sorry for the delay).
> 
> > I get now with "ctest -j12 -R 'export.*pdf4_systemF'":
> > 
> > 87% tests passed, 41 tests failed out of 316
> > But 3 of the failed are 'INVERTED', which means, they are OK now.
> > 
> > 
> > Previously it was
> > 80% tests passed, 62 tests failed out of 316
> > with 2 'INVERTED'.
> > 
> > So yes, some export test are fixed.
> 
> This is indeed good news. But did you check if there were any
> regressions?

Sorry, I did not.

> Scott

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Guenter Milde]

On 2015-10-26, Scott Kostyshak wrote:
> On Fri, Oct 23, 2015 at 07:31:49PM +, Guenter Milde wrote:
>> On 2015-10-23, Scott Kostyshak wrote:
>> > On Tue, Oct 20, 2015 at 09:10:05PM +, Guenter Milde wrote:
>> >> On 2015-10-20, Scott Kostyshak wrote:
>> >> > On Tue, Oct 20, 2015 at 07:16:44PM +0200, Günter Milde wrote:
>> >> >> commit 1523fc6023440f10ca0d82a681ded5c060d8fd33
>> >> >> Author: Günter Milde 
>> >> >> Date:   Tue Oct 20 19:14:39 2015 +0200

>> >> >> Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

>> >> >> Fixes output for 3 of the 4 test lyx-files.

With the latest 2 fixes (36b9645b5de6d463b4909 and 2aa65fdc8d2cde1ba863),
all 4 test files compile.

>> >> >> Includes "FIXME"s at places where further action is required
>> >> >> to get the XeTeX export right but I don't know how.
>> >> >> ...

>> >> >> +  &&!runparams.isFullUnicode()) { // FIXME: check must be 
>> >> >> done for useNonTeXFonts!
>> >> >>os << "\\inputencoding{utf8}\n"
>> >> >>   << setEncoding("UTF-8");

>> >> > So to make sure I understand what you want with this FIXME is you
>> >> > would like to do something like params().useNonTeXFonts as you do
>> >> > in Buffer.cpp but in PDFOptions.cpp it's not clear how to access
>> >> > that?

>> >> It may also be that I misunderstand what is done there, but
>> >> basically, yes: I believe that the test at these places must be for
>> >> "useNonTeXFonts" instead of "isFullUnicode", because XeTeX/LuaTeX
>> >> with TeX fonts behave more similar to 8-bit TeX regarding the in-
>> >> and output encodings.

>> > OK, what about the attached patch? Can you check that it does what you
>> > want as far as the two FIXMEs in PDFOptions.cpp? 

>> From what I see (and with my limited understanding of C++), I believe the
>> fix to be OK.

> Has anything changed in the last couple of days? Specifically, are the
> commits 2aa65fdc and 36b9645b relevant to this discussion?

Unless you modelled your patch on the wrong implementation (missing
negation) before, no.

> If nothing has changed, then I will proceed to work on extending this
> patch to address the other FIXMEs that you added.

As already said, I'd prefer to know what the code in question actually
does and have an example to test the current and later behaviour.

Probabely, ru/Intro.lyx could be minimalized to an example - it currently
fails with non-TeX fonts and XeTeX due to (amongst other problems) an
\inputencoding

It may well be, that the FIXME in PDFOptions is not enaugh - maybe we have
to push the content through unicodesymbols conversion, too.

Günter

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Scott Kostyshak]

On Mon, Oct 26, 2015 at 10:12:32AM +0100, Kornel Benko wrote:
> Am Montag, 26. Oktober 2015 um 01:53:24, schrieb Scott Kostyshak 
> 
> > On Sun, Oct 25, 2015 at 01:21:01PM +0100, Kornel Benko wrote:
> > > > Scott, could you please test if this fixes some more export tests?
> > 
> > Thanks for asking! Kornel already beat me to it but I will also run the
> > tests and let you know the results tomorrow (sorry for the delay).
> > 
> > > I get now with "ctest -j12 -R 'export.*pdf4_systemF'":
> > > 
> > >   87% tests passed, 41 tests failed out of 316
> > > But 3 of the failed are 'INVERTED', which means, they are OK now.
> > > 
> > > 
> > > Previously it was
> > >   80% tests passed, 62 tests failed out of 316
> > > with 2 'INVERTED'.
> > > 
> > > So yes, some export test are fixed.
> > 
> > This is indeed good news. But did you check if there were any
> > regressions?
> 
> Sorry, I did not.

I just ran the tests over night. There are no regressions and as Kornel
mentioned many tests correctly pass now.

Scott

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Scott Kostyshak]

On Fri, Oct 23, 2015 at 11:05:59PM +0200, Georg Baum wrote:

> The fix is simple: Re-add the deleted lines to set the encoding to utf8-
> plain (see attachment). Then Development.lyx compiles again for me. The 
> encoding must always be utf8 for full unicode backends if non-TeX fonts are 
> used. I guess that you were mislead by the comment, which suggested that 
> setting the encoding was done for TeX fonts, but this was only a part of the 
> truth: It was done for non-TeX fonts as well.
> 
> > Before my partial patch, the "latex encoding" was set to "utf-8" for
> > export to XeTeX and LuaTeX, regardless of the used font encoding. This can
> > lead to both, now reported errors for missing characters and wrong output
> > for some non-ACII characters in the "latin extended" block.
> 
> I think the part of the patch that changes the behaviour for XeTeX + TeX 
> fonts was fine and I am pretty sure that it fixes many more problems for 
> this combination than it creates.

Let me know if it would be useful to compare the results of the export
tests. Just let me know which two states to compare (e.g. master +
Georg's patch vs.  master). Or if you have confidence in the patch, feel
free to put it in and I can just test master against master before the
patch.

> >> The commit also fixed a lot of ctests.
> > 
> > Even here, we cannot be sure that XeTeX export is correct now.

Agreed.

Scott

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Guenter Milde]

On 2015-10-23, Georg Baum wrote:

> [-- Type: text/plain, Encoding: 7bit --]

> Guenter Milde wrote:

>> On 2015-10-21, Scott Kostyshak wrote:

>>> It compiles fine before this commit.

>> Yes, but only "per accident": Development.lyx only uses some non-ASCII
>> characters, and these are also at the same place in Unicode and T1.

> It was not by accident. It did compile before correctly because the .tex 
> file was encoded in utf8, as XeTeX expects it, and this was explicitly 
> programmed. After your change, the file is encoded in latin1, which is 
> interpreted as utf8 by XeTeX, which does of course not work and causes the 
> "Invalid UTF-8" error above.

Just to be sure we don't misunderstand: The question is about tests
compiling the documentation "as-is" (i.e. with *TeX fonts*) using
XeTeX.

In this case, while XeTeX expects utf8, it does not convert non-ASCII
characters to the "LaTeX internal character representation" (LICR), but
forwards it directly to the font character selection. Therefore, 

* any character in the 128 ... 255 range (Latin extended) selects a glyph
  from the same range in the T1 TeX font encoding - some of them
  coincide, others differ.

* any character in the range above 255 fails with TeX fonts. This is the
  reason for the many "missing –" (en-dash) errors.


The proper fix would be to restrict the exported file to ASCII characters
for the combination "XeTeX + TeX fonts". 
(Unfortunately, I did not manage to achieve this and therfore declared
my patch as a "partial fix".)

> The fix is simple: Re-add the deleted lines to set the encoding to utf8-
> plain (see attachment). Then Development.lyx compiles again for me. 

For me, Development.lyx compiles fine with XeTeX an and my patch, if I
set "use non TeX fonts" to TRUE.

> The encoding must always be utf8 for full unicode backends if non-TeX
> fonts are used. 

On this, we agree. Therefore my patch tests for "non-TeX fonts" - if non-TeX
fonts are used, there is no need to test for the engine, as both allowed
engines (XeTeX, LuaTeX) expect "utf8-plain" as input encoding.

> I guess that you were mislead by the comment, which suggested that 
> setting the encoding was done for TeX fonts, but this was only a part
> of the truth: It was done for non-TeX fonts as well.

Please test again with a file that selects "non-TeX fonts". If in this case
my patch exports in an ecoding other than utf8-plain, this is a bug.
However, I did not experience this combination here.




>> So, in this example the patch indeed broke the XeTeX export, while it may
>> be that in other "broken" example files export with XeTeX run without
>> error report but still produced wrong results!

> Sure. A passing test after a change does not tell that the change did
> not break anything. For such a statement we would need to keep
> references of the exported files, and this is only possible for
> selected, small examples, not for out complete documentation that does
> change a lot. Only a test which fails after a change and did not fail
> before tells that something became worse (it does not tell that the
> previous state was ok, only that it was less broken).

Yes. However, for "Development.lyx with TeX-fonts and XeTeX", I could
establish, that the "broken" state does not indicate that the partial fix is
wrong -- the now broken test is due to it beeing incomplete.

...

> But it is quite likely that XeTeX export for non-TeX fonts (the main
> use case for XeTeX) was correct before, since this does simply use utf8
> encoding throughout the whole file, which can represent every symbol
> LyX does understand.

Indeed. (Except for cases where missing characters were not reported before
but lead to errors now, but this is a different topic.)

Unfortunately, the case "XeTeX + fontspec" was not tested. The tests use
XeTeX with unmodified documents and unless bug #... is solved this means
they test the corner case XeTeX+TeX-fonts!


But: your patch showed me where and why my patch failed: the correct version
should be 

diff --git a/src/Buffer.cpp b/src/Buffer.cpp
index 829616a..73d2007 100644
--- a/src/Buffer.cpp
+++ b/src/Buffer.cpp
@@ -1588,7 +1588,7 @@ bool Buffer::makeLaTeXFile(FileName const & fname,
 
// XeTeX with TeX fonts is only safe with ASCII encoding,
// See #9740 and FIXME in BufferParams::encoding()
-   if (params().useNonTeXFonts && (runparams.flavor == 
OutputParams::XETEX))
+   if (!params().useNonTeXFonts && (runparams.flavor == 
OutputParams::XETEX))
runparams.encoding = encodings.fromLyXName("ascii");
 
string const encoding = runparams.encoding->iconvName();

i.e. I missed a negation!

With this correction, also Development.lyx + TeX fonts + XeTeX
works :-)

I'll commit the patch for the patch and hope this will solve some more of the
automatic XeTeX tests.

Thanks,
Günter

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Guenter Milde]

On 2015-10-25, Guenter Milde wrote:
> On 2015-10-25, Georg Baum wrote:
>> Guenter Milde wrote:

>>> I'll commit the patch for the patch and hope this will solve some more of
>>> the automatic XeTeX tests.

>> Please do. 

> It is in.

> Scott, could you please test if this fixes some more export tests?

Actually, the patch does now what Georg expected the old patch to do:

set encoding to ASCII for every use of XETEX, also with non-TeX fonts.

This fails with characters without a LICR defined by "xunicode".

So the current best known version would be to admit the limit:

Günter

diff --git a/src/Buffer.cpp b/src/Buffer.cpp
index 73d2007..0d621b1 100644
--- a/src/Buffer.cpp
+++ b/src/Buffer.cpp
@@ -1588,8 +1588,9 @@ bool Buffer::makeLaTeXFile(FileName const & fname,
 
// XeTeX with TeX fonts is only safe with ASCII encoding,
// See #9740 and FIXME in BufferParams::encoding()
-   if (!params().useNonTeXFonts && (runparams.flavor == 
OutputParams::XETEX))
-   runparams.encoding = encodings.fromLyXName("ascii");
+   // However, the code below also sets "ascii" for XeTeX with non-TeX 
fonts (why?)
+   // if (!params().useNonTeXFonts && (runparams.flavor == 
OutputParams::XETEX))
+   //  runparams.encoding = encodings.fromLyXName("ascii");
 
string const encoding = runparams.encoding->iconvName();
LYXERR(Debug::LATEX, "makeLaTeXFile encoding: " << encoding << ", 
fname=" << fname.realPath());


Günter

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Guenter Milde]

On 2015-10-25, Georg Baum wrote:
> Guenter Milde wrote:

>> I'll commit the patch for the patch and hope this will solve some more of
>> the automatic XeTeX tests.

> Please do. 

It is in.

Scott, could you please test if this fixes some more export tests?

> BTW, the braces around the equality comparison are not needed, so 
> to my eyes

> if (!params().useNonTeXFonts && runparams.flavor == OutputParams::XETEX)

> is more readable.

I don't know the operator precedence in C++, therefore I added the braces.


Günter

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Georg Baum]

Guenter Milde wrote:

> Just to be sure we don't misunderstand: The question is about tests
> compiling the documentation "as-is" (i.e. with *TeX fonts*) using
> XeTeX.

Thanks for the clarification, I think we misunderstood. For XeTeX + TeX 
fonts I agree.

> For me, Development.lyx compiles fine with XeTeX an and my patch, if I
> set "use non TeX fonts" to TRUE.

You are right, I forgot that.

> Please test again with a file that selects "non-TeX fonts". If in this
> case my patch exports in an ecoding other than utf8-plain, this is a bug.
> However, I did not experience this combination here.

I do not have one. What I missed was the initialization of the encoding to 
utf8-plain in BufferParams::encoding(). Therefore I thought that the 
encoding for non-TeX-fonts was now not correctly set anymore, and since 
Development.lyx compiled again after my change I took this as a 
confirmation. In the end, I made two mistakes which cancelled each other.

> Unfortunately, the case "XeTeX + fontspec" was not tested. The tests use
> XeTeX with unmodified documents and unless bug #... is solved this means
> they test the corner case XeTeX+TeX-fonts!

Therefore I think that adding a "default" setting for the fonts (first part 
of #9744) is so important.

> I'll commit the patch for the patch and hope this will solve some more of
> the automatic XeTeX tests.

Please do. BTW, the braces around the equality comparison are not needed, so 
to my eyes

if (!params().useNonTeXFonts && runparams.flavor == OutputParams::XETEX)

is more readable.


Georg

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Kornel Benko]

Am Sonntag, 25. Oktober 2015 um 11:31:09, schrieb Guenter Milde 

> On 2015-10-25, Georg Baum wrote:
> > Guenter Milde wrote:
> 
> >> I'll commit the patch for the patch and hope this will solve some more of
> >> the automatic XeTeX tests.
> 
> > Please do. 
> 
> It is in.
> 
> Scott, could you please test if this fixes some more export tests?

I get now with "ctest -j12 -R 'export.*pdf4_systemF'":

87% tests passed, 41 tests failed out of 316
But 3 of the failed are 'INVERTED', which means, they are OK now.


Previously it was
80% tests passed, 62 tests failed out of 316
with 2 'INVERTED'.

So yes, some export test are fixed.

> > BTW, the braces around the equality comparison are not needed, so 
> > to my eyes
> 
> > if (!params().useNonTeXFonts && runparams.flavor == OutputParams::XETEX)
> 
> > is more readable.
> 
> I don't know the operator precedence in C++, therefore I added the braces.
> 
> 
> Günter

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Guenter Milde]

On 2015-10-25, Georg Baum wrote:
> Guenter Milde wrote:

>> Actually, the patch does now what Georg expected the old patch to do:

>> set encoding to ASCII for every use of XETEX, also with non-TeX fonts.

>> This fails with characters without a LICR defined by "xunicode".

>> So the current best known version would be to admit the limit:

> IMO this points to a serious bug and needs to be investigated further: The 
> condition

> if (!params().useNonTeXFonts && runparams.flavor == OutputParams::XETEX)
>   runparams.encoding = encodings.fromLyXName("ascii");

> describes exactly what should happen. So if this sets the encoding to ascii 
> even if non-TeX fonts are used, then params().useNonTeXFonts must be wrong, 
> and this would be a serious problem. My guess would be that the code is 
> correct, but that the other occurence of this code in writeLaTeXSource() 
> does the wrong thing. If the if-condition in this one is adjusted as well, 
> then it will probably work.


The problem was part what you described and part use of a non-minimal test
file: doc/ru/Intro.lyx failed to compile with XeTeX (both, TeX fonts and
non-TeX fonts) for several reasons:

* wrong preamble code prevents use of non-TeX fonts (overwritten by loading 
  lmodern.sty)

* the file does not compile with "inputenc = ascii", because of a spurious
  \textcyr font encoding change between an accent and the base letter (see
  #9637) with both, pdflatex and xelatex.
  
The missing fix in writeLaTeXSource() suggested this were due to wrong input
encoding in both cases, but finally I sorted this out.

So, besides the FIXME (the re-setting to "ascii" should rather be in
BufferEncoding.cpp) and the doubling of code in Buffer.cpp, the fixed fix is
fine and with the second fix it works reasonable with both XeTeX and LuaTeX.

Thank you for the helpfull hints.

Günter
  

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Scott Kostyshak]

On Fri, Oct 23, 2015 at 07:31:49PM +, Guenter Milde wrote:
> On 2015-10-23, Scott Kostyshak wrote:
> > On Tue, Oct 20, 2015 at 09:10:05PM +, Guenter Milde wrote:
> >> On 2015-10-20, Scott Kostyshak wrote:
> >> > On Tue, Oct 20, 2015 at 07:16:44PM +0200, Günter Milde wrote:
> >> >> commit 1523fc6023440f10ca0d82a681ded5c060d8fd33
> >> >> Author: Günter Milde 
> >> >> Date:   Tue Oct 20 19:14:39 2015 +0200
> 
> >> >> Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
> 
> >> >> Fixes output for 3 of the 4 test lyx-files.
> 
> >> >> Includes "FIXME"s at places where further action is required to get 
> >> >> the XeTeX
> >> >> export right but I don't know how.
> >> >> ...
> 
> >> >> +   &&!runparams.isFullUnicode()) { // FIXME: check must be 
> >> >> done for useNonTeXFonts!
> >> >> os << "\\inputencoding{utf8}\n"
> >> >><< setEncoding("UTF-8");
> 
> >> > So to make sure I understand what you want with this FIXME is you would
> >> > like to do something like params().useNonTeXFonts as you do in
> >> > Buffer.cpp but in PDFOptions.cpp it's not clear how to access that?
> 
> >> It may also be that I misunderstand what is done there, but basically, 
> >> yes: 
> >> I believe that the test at these places must be for "useNonTeXFonts" 
> >> instead
> >> of "isFullUnicode", because XeTeX/LuaTeX with TeX fonts behave more similar
> >> to 8-bit TeX regarding the in- and output encodings.
> 
> > OK, what about the attached patch? Can you check that it does what you
> > want as far as the two FIXMEs in PDFOptions.cpp? 
> 
> From what I see (and with my limited understanding of C++), I believe the
> fix to be OK.

Has anything changed in the last couple of days? Specifically, are the
commits 2aa65fdc and 36b9645b relevant to this discussion?

If nothing has changed, then I will proceed to work on extending this
patch to address the other FIXMEs that you added.

Scott

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Scott Kostyshak]

On Sun, Oct 25, 2015 at 01:21:01PM +0100, Kornel Benko wrote:
> Am Sonntag, 25. Oktober 2015 um 11:31:09, schrieb Guenter Milde 
> 
> > On 2015-10-25, Georg Baum wrote:
> > > Guenter Milde wrote:
> > 
> > >> I'll commit the patch for the patch and hope this will solve some more of
> > >> the automatic XeTeX tests.
> > 
> > > Please do. 
> > 
> > It is in.
> > 
> > Scott, could you please test if this fixes some more export tests?

Thanks for asking! Kornel already beat me to it but I will also run the
tests and let you know the results tomorrow (sorry for the delay).

> I get now with "ctest -j12 -R 'export.*pdf4_systemF'":
> 
>   87% tests passed, 41 tests failed out of 316
> But 3 of the failed are 'INVERTED', which means, they are OK now.
> 
> 
> Previously it was
>   80% tests passed, 62 tests failed out of 316
> with 2 'INVERTED'.
> 
> So yes, some export test are fixed.

This is indeed good news. But did you check if there were any
regressions?

Scott

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Georg Baum]

Guenter Milde wrote:

> Actually, the patch does now what Georg expected the old patch to do:
> 
> set encoding to ASCII for every use of XETEX, also with non-TeX fonts.
> 
> This fails with characters without a LICR defined by "xunicode".
> 
> So the current best known version would be to admit the limit:

IMO this points to a serious bug and needs to be investigated further: The 
condition

if (!params().useNonTeXFonts && runparams.flavor == OutputParams::XETEX)
runparams.encoding = encodings.fromLyXName("ascii");

describes exactly what should happen. So if this sets the encoding to ascii 
even if non-TeX fonts are used, then params().useNonTeXFonts must be wrong, 
and this would be a serious problem. My guess would be that the code is 
correct, but that the other occurence of this code in writeLaTeXSource() 
does the wrong thing. If the if-condition in this one is adjusted as well, 
then it will probably work.


Georg

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Richard Heck]

On 10/25/2015 07:31 AM, Guenter Milde wrote:

On 2015-10-25, Georg Baum wrote:

BTW, the braces around the equality comparison are not needed, so
to my eyes
if (!params().useNonTeXFonts && runparams.flavor == OutputParams::XETEX)
is more readable.

I don't know the operator precedence in C++, therefore I added the braces.


Better to have them than not to have them, if one isn't sure, it always 
seems to me.


Richard

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Georg Baum]

Richard Heck wrote:

> On 10/25/2015 07:31 AM, Guenter Milde wrote:
>> On 2015-10-25, Georg Baum wrote:
>>> BTW, the braces around the equality comparison are not needed, so
>>> to my eyes
>>> if (!params().useNonTeXFonts && runparams.flavor == OutputParams::XETEX)
>>> is more readable.
>> I don't know the operator precedence in C++, therefore I added the
>> braces.

Now you do for this case (until you forget again;-)

> Better to have them than not to have them, if one isn't sure, it always
> seems to me.

Sure. My motivation for mentioning this was to increase the knowledge about 
operator precedence. If you read and write statements like this more often, 
then it eases your work a lot if you know at least the most important 
precedence rules. At least this is my experience.


Georg

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Guenter Milde]

On 2015-10-23, Scott Kostyshak wrote:
> On Tue, Oct 20, 2015 at 09:10:05PM +, Guenter Milde wrote:
>> On 2015-10-20, Scott Kostyshak wrote:
>> > On Tue, Oct 20, 2015 at 07:16:44PM +0200, Günter Milde wrote:
>> >> commit 1523fc6023440f10ca0d82a681ded5c060d8fd33
>> >> Author: Günter Milde 
>> >> Date:   Tue Oct 20 19:14:39 2015 +0200

>> >> Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

>> >> Fixes output for 3 of the 4 test lyx-files.

>> >> Includes "FIXME"s at places where further action is required to get 
>> >> the XeTeX
>> >> export right but I don't know how.
>> >> ...

>> >> + &&!runparams.isFullUnicode()) { // FIXME: check must be done for 
>> >> useNonTeXFonts!
>> >>   os << "\\inputencoding{utf8}\n"
>> >>  << setEncoding("UTF-8");

>> > So to make sure I understand what you want with this FIXME is you would
>> > like to do something like params().useNonTeXFonts as you do in
>> > Buffer.cpp but in PDFOptions.cpp it's not clear how to access that?

>> It may also be that I misunderstand what is done there, but basically, yes: 
>> I believe that the test at these places must be for "useNonTeXFonts" instead
>> of "isFullUnicode", because XeTeX/LuaTeX with TeX fonts behave more similar
>> to 8-bit TeX regarding the in- and output encodings.

> OK, what about the attached patch? Can you check that it does what you
> want as far as the two FIXMEs in PDFOptions.cpp? 

>From what I see (and with my limited understanding of C++), I believe the
fix to be OK.

To be sure, I'd need a use case for this code. (Maybe the commit message
you found with "blame" tells something helpfull about where it is
needed?) The we could compare the exported file before and after patching
and see which works better.

> If so, hopefully someone else can confirm that it is reasonable to pass
> the buffer parameters in this situation. Then after that I could extend
> the fix to address the other FIXMEs that you have regarding
> useNonTeXFonts.

However, the main problem is ensuring the "latex-encoding" is "ASCII" for
the combination of XeTeX and "TeX fonts".

> Günter are you familiar with "git blame"? It is a useful tool for
> purposes such as these where we might want to check with the person who
> introduced a certain line of code. For example, if before your commit
> (otherwise it will say you are the author of that line because you
> edited it to put the comment) I
> do

> git blame PDFOptions.cpp

> and then I search (with '/') for "isFullUnicode" it takes me to the
> first instance and tells me that Jürgen introduced this line in the
> commit baaceb10. So now we ping Jürgen (now CC'ed) and see what his
> opinion is on this.

Thanks

Günter

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Guenter Milde]

On 2015-10-21, Scott Kostyshak wrote:

> [-- Type: text/plain, Encoding: 8bit --]

> On Tue, Oct 20, 2015 at 09:10:05PM +, Guenter Milde wrote:

>> It may also be that I misunderstand what is done there, but basically, yes: 
>> I believe that the test at these places must be for "useNonTeXFonts" instead
>> of "isFullUnicode", because XeTeX/LuaTeX with TeX fonts behave more similar
>> to 8-bit TeX regarding the in- and output encodings.

> This commit breaks a lot of ctests. 

While I was pretty sure that the partial patch will not fix all these
strange use cases, I did not expect it to break working examples.

> For example, the document
> doc/Development.lyx no longer compiles with XeTeX (with TeX fonts).

So let's have a look at the example.

> I get the following error:

>   Missing character: There is no — in font ecrm1200!

This reports a missing en-dash. Strange, because there is no en-dash in
Development.lyx (otherwise it would fail before the patch with the above
error.

Here I get

  Missing character: There is no � in font ecrm1200!

and this is because of:

  Invalid UTF-8 byte or sequence at line 300 replaced by U+FFFD.
  Missing character: There is no � in font ecrm1200!

> It compiles fine before this commit.

Yes, but only "per accident": Development.lyx only uses some non-ASCII
characters, and these are also at the same place in Unicode and T1.

Before my partial patch, the "latex encoding" was set to "utf-8" for export
to XeTeX and LuaTeX, regardless of the used font encoding. This can lead to
both, now reported errors for missing characters and wrong output for some
non-ACII characters in the "latin extended" block.

If you compile the example file "luatex-without-fontspec-latin9" using
XeTeX with an "unpatched" LyX, you get both, the 

  Missing character: There is no — in font ecrm1200!

for the en-dash and wrong high bit characters:

The input:

  £§¥©®°±²³´µ·½¿÷ø ÂÃÄÖÜ äöüÿß
  
looks very different in the PDF (starting with "č" and other
east-european Latin characters and ending with "ßSS" (while drag-and-drop
here results in the almost correct "£§¥©®°±23 ́μ·1⁄2¿÷ø ÂÃÄÖÜ äöüÿß").

However, the German umlauts are kept.

And actually the small o-umlaut "ö" is the only non-ASCII character in the
whole file Development.lyx. 

So, in this example the patch indeed broke the XeTeX export, while it may
be that in other "broken" example files export with XeTeX run without error
report but still produced wrong results!


> The commit also fixed a lot of ctests.

Even here, we cannot be sure that XeTeX export is correct now.

> Attached are the lists of tests that fail before this commit and the
> list that fails after this commit. By comparing them you can see which
> ones this commit changed. Let me know if you have any questions.

I wonder, whether we could limit the number of tests for the corner case
"XeTeX/LuaTeX with TeX fonts".

In my view, we should just admit that this is currently broken and the
proper fix would be "automatic" selection of non-TeX fonts for XeTeX/LuaTeX
(if not explicitely set otherwise by the user).

Otherwise, nothing will spare us a close examination of all these files and
the export result "per hand"...


Günter

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Georg Baum]

Guenter Milde wrote:

> On 2015-10-21, Scott Kostyshak wrote:
> 
>> It compiles fine before this commit.
> 
> Yes, but only "per accident": Development.lyx only uses some non-ASCII
> characters, and these are also at the same place in Unicode and T1.

It was not by accident. It did compile before correctly because the .tex 
file was encoded in utf8, as XeTeX expects it, and this was explicitly 
programmed. After your change, the file is encoded in latin1, which is 
interpreted as utf8 by XeTeX, which does of course not work and causes the 
"Invalid UTF-8" error above.

The fix is simple: Re-add the deleted lines to set the encoding to utf8-
plain (see attachment). Then Development.lyx compiles again for me. The 
encoding must always be utf8 for full unicode backends if non-TeX fonts are 
used. I guess that you were mislead by the comment, which suggested that 
setting the encoding was done for TeX fonts, but this was only a part of the 
truth: It was done for non-TeX fonts as well.

> Before my partial patch, the "latex encoding" was set to "utf-8" for
> export to XeTeX and LuaTeX, regardless of the used font encoding. This can
> lead to both, now reported errors for missing characters and wrong output
> for some non-ACII characters in the "latin extended" block.

I think the part of the patch that changes the behaviour for XeTeX + TeX 
fonts was fine and I am pretty sure that it fixes many more problems for 
this combination than it creates.

> So, in this example the patch indeed broke the XeTeX export, while it may
> be that in other "broken" example files export with XeTeX run without
> error report but still produced wrong results!

Sure. A passing test after a change does not tell that the change did not 
break anything. For such a stement we would need to keep references of the 
exported files, and this is only possible for selected, small examples, not 
for out complete documentation that does change a lot. Only a test which 
fails after a change and did not fail before tells that something became 
worse (it does not tell that the previous state was ok, onlyx that it was 
less broken).

>> The commit also fixed a lot of ctests.
> 
> Even here, we cannot be sure that XeTeX export is correct now.

But it is quite likely that XeTeX export for non-TeX fonts (the main use 
case for XeTeX) was correct before, since this does simply use utf8 encoding 
throughout the whole file, which can represent every symbol LyX does 
understand.


Georg
diff --git a/src/Buffer.cpp b/src/Buffer.cpp
index 829616a..1f5f7c4 100644
--- a/src/Buffer.cpp
+++ b/src/Buffer.cpp
@@ -1590,6 +1590,8 @@ bool Buffer::makeLaTeXFile(FileName const & fname,
 	// See #9740 and FIXME in BufferParams::encoding()
 	if (params().useNonTeXFonts && (runparams.flavor == OutputParams::XETEX))
 		runparams.encoding = encodings.fromLyXName("ascii");
+	else if (runparams.isFullUnicode())
+		runparams.encoding = encodings.fromLyXName("utf8-plain");
 
 	string const encoding = runparams.encoding->iconvName();
 	LYXERR(Debug::LATEX, "makeLaTeXFile encoding: " << encoding << ", fname=" << fname.realPath());

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Scott Kostyshak]

On Tue, Oct 20, 2015 at 09:10:05PM +, Guenter Milde wrote:
> On 2015-10-20, Scott Kostyshak wrote:
> > On Tue, Oct 20, 2015 at 07:16:44PM +0200, Günter Milde wrote:
> >> commit 1523fc6023440f10ca0d82a681ded5c060d8fd33
> >> Author: Günter Milde 
> >> Date:   Tue Oct 20 19:14:39 2015 +0200
> 
> >> Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
> 
> >> Fixes output for 3 of the 4 test lyx-files.
> 
> >> Includes "FIXME"s at places where further action is required to get 
> >> the XeTeX
> >> export right but I don't know how.
> >> ...
> 
> >> +  &&!runparams.isFullUnicode()) { // FIXME: check must be done for 
> >> useNonTeXFonts!
> >>os << "\\inputencoding{utf8}\n"
> >>   << setEncoding("UTF-8");
> 
> > So to make sure I understand what you want with this FIXME is you would
> > like to do something like params().useNonTeXFonts as you do in
> > Buffer.cpp but in PDFOptions.cpp it's not clear how to access that?
> 
> It may also be that I misunderstand what is done there, but basically, yes: 
> I believe that the test at these places must be for "useNonTeXFonts" instead
> of "isFullUnicode", because XeTeX/LuaTeX with TeX fonts behave more similar
> to 8-bit TeX regarding the in- and output encodings.

OK, what about the attached patch? Can you check that it does what you
want as far as the two FIXMEs in PDFOptions.cpp? If so, hopefully
someone else can confirm that it is reasonable to pass the buffer
parameters in this situation. Then after that I could extend the fix to
address the other FIXMEs that you have regarding useNonTeXFonts.

Günter are you familiar with "git blame"? It is a useful tool for
purposes such as these where we might want to check with the person who
introduced a certain line of code. For example, if before your commit
(otherwise it will say you are the author of that line because you
edited it to put the comment) I
do

git blame PDFOptions.cpp

and then I search (with '/') for "isFullUnicode" it takes me to the
first instance and tells me that Jürgen introduced this line in the
commit baaceb10. So now we ping Jürgen (now CC'ed) and see what his
opinion is on this.

Scott
diff --git a/src/BufferParams.cpp b/src/BufferParams.cpp
index 537178e..60f1a24 100644
--- a/src/BufferParams.cpp
+++ b/src/BufferParams.cpp
@@ -1901,8 +1901,9 @@ bool BufferParams::writeLaTeX(otexstream & os, 
LaTeXFeatures & features,
// to access the stream itself in PDFOptions.
os << lyxpreamble;
 
-   OutputParams tmp_params = features.runparams();
-   pdfoptions().writeLaTeX(tmp_params, os,
+   OutputParams output_params = features.runparams();
+   BufferParams buffer_params = *this;
+   pdfoptions().writeLaTeX(output_params, buffer_params, os,
features.isProvided("hyperref"));
// set back for the rest
lyxpreamble.clear();
diff --git a/src/PDFOptions.cpp b/src/PDFOptions.cpp
index afbd5e5..2a82f44 100644
--- a/src/PDFOptions.cpp
+++ b/src/PDFOptions.cpp
@@ -89,8 +89,8 @@ void PDFOptions::writeFile(ostream & os) const
 }
 
 
-void PDFOptions::writeLaTeX(OutputParams & runparams, otexstream & os,
-   bool hyperref_already_provided) const
+void PDFOptions::writeLaTeX(OutputParams & runparams, BufferParams & 
bufferparams,
+   otexstream & os, bool hyperref_already_provided) 
const
 {
// FIXME Unicode
string opt;
@@ -176,7 +176,7 @@ void PDFOptions::writeLaTeX(OutputParams & runparams, 
otexstream & os,
 
// hyperref expects utf8!
if (need_unicode && enc && enc->iconvName() != "UTF-8"
-   &&!runparams.isFullUnicode()) { // FIXME: check must be done for 
useNonTeXFonts!
+   && !bufferparams.useNonTeXFonts) {
os << "\\inputencoding{utf8}\n"
   << setEncoding("UTF-8");
}
@@ -196,7 +196,7 @@ void PDFOptions::writeLaTeX(OutputParams & runparams, 
otexstream & os,
os << from_utf8(opt);
 
if (need_unicode && enc && enc->iconvName() != "UTF-8"
-   &&!runparams.isFullUnicode()) { // FIXME: check for useNonTeXFonts!
+   && !bufferparams.useNonTeXFonts) {
os << setEncoding(enc->iconvName())
   << "\\inputencoding{" << from_ascii(enc->latexName()) << 
"}\n";
}
diff --git a/src/PDFOptions.h b/src/PDFOptions.h
index d74caf4..f4d5250 100644
--- a/src/PDFOptions.h
+++ b/src/PDFOptions.h
@@ -13,6 +13,7 @@
 #define PDFOPTIONS_H
 
 #include "OutputParams.h"
+#include "BufferParams.h"
 
 #include "support/strfwd.h"
 
@@ -36,7 +37,7 @@ public:
/// output to lyx header
void writeFile(std::ostream &) const;
/// output to tex header
-   void writeLaTeX(OutputParams &, otexstream &,
+   void writeLaTeX(OutputParams &, BufferParams &, otexstream &,
 

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Scott Kostyshak]

On Tue, Oct 20, 2015 at 09:10:05PM +, Guenter Milde wrote:

> It may also be that I misunderstand what is done there, but basically, yes: 
> I believe that the test at these places must be for "useNonTeXFonts" instead
> of "isFullUnicode", because XeTeX/LuaTeX with TeX fonts behave more similar
> to 8-bit TeX regarding the in- and output encodings.

This commit breaks a lot of ctests. For example, the document
doc/Development.lyx no longer compiles with XeTeX (with TeX fonts).
I get the following error:

  Missing character: There is no — in font ecrm1200!

It compiles fine before this commit.

The commit also fixed a lot of ctests.

Attached are the lists of tests that fail before this commit and the
list that fails after this commit. By comparing them you can see which
ones this commit changed. Let me know if you have any questions.

Scott
209:export/doc/Additional_pdf4_texF
217:export/doc/Customization_dvi3_texF
222:export/doc/Customization_pdf4_texF
224:export/doc/Customization_pdf5_texF
225:export/doc/Customization_pdf5_systemF
274:export/doc/EmbeddedObjects_pdf4_texF
275:export/doc/EmbeddedObjects_pdf4_systemF
300:export/doc/Intro_pdf4_texF
313:export/doc/LFUNs_pdf4_texF
321:export/doc/LaTeXConfig_dvi3_texF
326:export/doc/LaTeXConfig_pdf4_texF
328:export/doc/LaTeXConfig_pdf5_texF
334:export/doc/Math_dvi3_texF
335:export/doc/Math_dvi3_systemF
339:export/doc/Math_pdf4_texF
341:export/doc/Math_pdf5_texF
342:export/doc/Math_pdf5_systemF
368:export/doc/Tutorial_pdf4_texF
376:export/doc/UserGuide_dvi3_texF
381:export/doc/UserGuide_pdf4_texF
383:export/doc/UserGuide_pdf5_texF
394:export/doc/attic/DocStyle_pdf4_texF
402:export/doc/attic/eu_Additional_dvi3_texF
407:export/doc/attic/eu_Additional_pdf4_texF
408:INVERTED_SEE-README.ctest_export/doc/attic/eu_Additional_pdf4_systemF
409:export/doc/attic/eu_Additional_pdf5_texF
415:export/doc/attic/eu_Customization_dvi3_texF
420:export/doc/attic/eu_Customization_pdf4_texF
422:export/doc/attic/eu_Customization_pdf5_texF
428:export/doc/attic/eu_UserGuide_dvi3_texF
433:export/doc/attic/eu_UserGuide_pdf4_texF
434:INVERTED_SEE-README.ctest_export/doc/attic/eu_UserGuide_pdf4_systemF
435:export/doc/attic/eu_UserGuide_pdf5_texF
441:export/doc/attic/it_Customization_dvi3_texF
446:export/doc/attic/it_Customization_pdf4_texF
448:export/doc/attic/it_Customization_pdf5_texF
449:export/doc/attic/it_Customization_pdf5_systemF
454:export/doc/attic/it_UserGuide_dvi3_texF
455:export/doc/attic/it_UserGuide_dvi3_systemF
459:export/doc/attic/it_UserGuide_pdf4_texF
460:export/doc/attic/it_UserGuide_pdf4_systemF
461:export/doc/attic/it_UserGuide_pdf5_texF
472:export/doc/attic/pl_Additional_pdf4_texF
473:INVERTED_SEE-README.ctest_export/doc/attic/pl_Additional_pdf4_systemF
480:export/doc/attic/sk_UserGuide_dvi3_texF
485:export/doc/attic/sk_UserGuide_pdf4_texF
487:export/doc/attic/sk_UserGuide_pdf5_texF
498:export/doc/ca/Intro_pdf4_texF
511:export/doc/cs/Tutorial_pdf4_texF
524:export/doc/da/Intro_pdf4_texF
537:export/doc/de/Additional_pdf4_texF
545:export/doc/de/Customization_dvi3_texF
550:export/doc/de/Customization_pdf4_texF
552:export/doc/de/Customization_pdf5_texF
553:export/doc/de/Customization_pdf5_systemF
589:export/doc/de/EmbeddedObjects_pdf4_texF
592:export/doc/de/EmbeddedObjects_pdf5_systemF
615:export/doc/de/Intro_pdf4_texF
623:export/doc/de/Math_dvi3_texF
624:export/doc/de/Math_dvi3_systemF
628:export/doc/de/Math_pdf4_texF
630:export/doc/de/Math_pdf5_texF
631:export/doc/de/Math_pdf5_systemF
654:export/doc/de/Tutorial_pdf4_texF
662:export/doc/de/UserGuide_dvi3_texF
667:export/doc/de/UserGuide_pdf4_texF
669:export/doc/de/UserGuide_pdf5_texF
670:export/doc/de/UserGuide_pdf5_systemF
680:export/doc/el/Intro_pdf4_texF
683:export/doc/el/Intro_pdf5_systemF
693:export/doc/es/Additional_pdf4_texF
706:export/doc/es/Customization_pdf4_texF
709:export/doc/es/Customization_pdf5_systemF
745:export/doc/es/EmbeddedObjects_pdf4_texF
748:export/doc/es/EmbeddedObjects_pdf5_systemF
771:export/doc/es/Intro_pdf4_texF
779:export/doc/es/Math_dvi3_texF
784:export/doc/es/Math_pdf4_texF
786:export/doc/es/Math_pdf5_texF
787:export/doc/es/Math_pdf5_systemF
810:export/doc/es/Tutorial_pdf4_texF
813:export/doc/es/Tutorial_pdf5_systemF
818:export/doc/es/UserGuide_dvi3_texF
823:export/doc/es/UserGuide_pdf4_texF
825:export/doc/es/UserGuide_pdf5_texF
826:export/doc/es/UserGuide_pdf5_systemF
836:export/doc/eu/Intro_pdf4_texF
849:export/doc/eu/Tutorial_pdf4_texF
857:export/doc/fr/Additional_dvi3_texF
862:export/doc/fr/Additional_pdf4_texF
864:export/doc/fr/Additional_pdf5_texF
865:export/doc/fr/Additional_pdf5_systemF
870:export/doc/fr/Customization_dvi3_texF
875:export/doc/fr/Customization_pdf4_texF
877:export/doc/fr/Customization_pdf5_texF
878:export/doc/fr/Customization_pdf5_systemF
914:export/doc/fr/EmbeddedObjects_pdf4_texF
940:export/doc/fr/Intro_pdf4_texF
948:export/doc/fr/Math_dvi3_texF
953:export/doc/fr/Math_pdf4_texF
955:export/doc/fr/Math_pdf5_texF
979:export/doc/fr/Tutorial_pdf4_texF

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Scott Kostyshak]

On Tue, Oct 20, 2015 at 07:16:44PM +0200, Günter Milde wrote:
> commit 1523fc6023440f10ca0d82a681ded5c060d8fd33
> Author: Günter Milde 
> Date:   Tue Oct 20 19:14:39 2015 +0200
> 
> Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".
> 
> Fixes output for 3 of the 4 test lyx-files.
> 
> Includes "FIXME"s at places where further action is required to get the 
> XeTeX
> export right but I don't know how.
> ...

> + &&!runparams.isFullUnicode()) { // FIXME: check must be done for 
> useNonTeXFonts!
>   os << "\\inputencoding{utf8}\n"
>  << setEncoding("UTF-8");

So to make sure I understand what you want with this FIXME is you would
like to do something like params().useNonTeXFonts as you do in
Buffer.cpp but in PDFOptions.cpp it's not clear how to access that?

Scott

Re: [LyX/master] Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

[Guenter Milde]

On 2015-10-20, Scott Kostyshak wrote:
> On Tue, Oct 20, 2015 at 07:16:44PM +0200, Günter Milde wrote:
>> commit 1523fc6023440f10ca0d82a681ded5c060d8fd33
>> Author: Günter Milde 
>> Date:   Tue Oct 20 19:14:39 2015 +0200

>> Partial fix for #9740 "XeTeX/LuaTeX with TeX fonts problems".

>> Fixes output for 3 of the 4 test lyx-files.

>> Includes "FIXME"s at places where further action is required to get the 
>> XeTeX
>> export right but I don't know how.
>> ...

>> +&&!runparams.isFullUnicode()) { // FIXME: check must be done for 
>> useNonTeXFonts!
>>  os << "\\inputencoding{utf8}\n"
>> << setEncoding("UTF-8");

> So to make sure I understand what you want with this FIXME is you would
> like to do something like params().useNonTeXFonts as you do in
> Buffer.cpp but in PDFOptions.cpp it's not clear how to access that?

It may also be that I misunderstand what is done there, but basically, yes: 
I believe that the test at these places must be for "useNonTeXFonts" instead
of "isFullUnicode", because XeTeX/LuaTeX with TeX fonts behave more similar
to 8-bit TeX regarding the in- and output encodings.

Günter