Guenter Milde wrote: > On 2015-10-21, Scott Kostyshak wrote: > >> It compiles fine before this commit. > > Yes, but only "per accident": Development.lyx only uses some non-ASCII > characters, and these are also at the same place in Unicode and T1.
It was not by accident. It did compile before correctly because the .tex file was encoded in utf8, as XeTeX expects it, and this was explicitly programmed. After your change, the file is encoded in latin1, which is interpreted as utf8 by XeTeX, which does of course not work and causes the "Invalid UTF-8" error above. The fix is simple: Re-add the deleted lines to set the encoding to utf8- plain (see attachment). Then Development.lyx compiles again for me. The encoding must always be utf8 for full unicode backends if non-TeX fonts are used. I guess that you were mislead by the comment, which suggested that setting the encoding was done for TeX fonts, but this was only a part of the truth: It was done for non-TeX fonts as well. > Before my partial patch, the "latex encoding" was set to "utf-8" for > export to XeTeX and LuaTeX, regardless of the used font encoding. This can > lead to both, now reported errors for missing characters and wrong output > for some non-ACII characters in the "latin extended" block. I think the part of the patch that changes the behaviour for XeTeX + TeX fonts was fine and I am pretty sure that it fixes many more problems for this combination than it creates. > So, in this example the patch indeed broke the XeTeX export, while it may > be that in other "broken" example files export with XeTeX run without > error report but still produced wrong results! Sure. A passing test after a change does not tell that the change did not break anything. For such a stement we would need to keep references of the exported files, and this is only possible for selected, small examples, not for out complete documentation that does change a lot. Only a test which fails after a change and did not fail before tells that something became worse (it does not tell that the previous state was ok, onlyx that it was less broken). >> The commit also fixed a lot of ctests. > > Even here, we cannot be sure that XeTeX export is correct now. But it is quite likely that XeTeX export for non-TeX fonts (the main use case for XeTeX) was correct before, since this does simply use utf8 encoding throughout the whole file, which can represent every symbol LyX does understand. Georg
diff --git a/src/Buffer.cpp b/src/Buffer.cpp index 829616a..1f5f7c4 100644 --- a/src/Buffer.cpp +++ b/src/Buffer.cpp @@ -1590,6 +1590,8 @@ bool Buffer::makeLaTeXFile(FileName const & fname, // See #9740 and FIXME in BufferParams::encoding() if (params().useNonTeXFonts && (runparams.flavor == OutputParams::XETEX)) runparams.encoding = encodings.fromLyXName("ascii"); + else if (runparams.isFullUnicode()) + runparams.encoding = encodings.fromLyXName("utf8-plain"); string const encoding = runparams.encoding->iconvName(); LYXERR(Debug::LATEX, "makeLaTeXFile encoding: " << encoding << ", fname=" << fname.realPath());