Re: How to force tex2lyx to read unicode (from within Lyx)?

[Baris Erkus]

On 11-Feb-19 4:16 AM, Scott Kostyshak wrote:
> On Sat, Nov 03, 2018 at 11:45:14AM -0400, Richard Kimberly Heck wrote:
>> On 11/2/18 5:52 PM, Baris Erkus wrote:
>>> On 11/2/2018 1:33 PM, Baris Erkus wrote:
 Hello,

 I am trying to import the a .tex  file into LyX (using file->import).
 The file has Turkish characters and I believe it is using UTF8 encoding.
 When I import it, it does not show Turkish characters properly. I
 attached the file for your play-around.

 When I import the file on the command line with the UTF8 option tex2lyx
 -e UTF8 ch1.tex, the LyX file looks just fine except one or two characters.

 So is there a way to tell to tex2lyx to use UTF8 encoding from within
 LyX? This issue is occured when I import a tex file that has
 \input{ch1.tex} in it. So, maybe LyX has and option to force tex2lyx to
 read in UTF8 for all imports?

 I found exactly same question on this post, but the answers seem to be
 unrelated:

 https://lyx-users.lyx.narkive.com/Rg1StNIT/how-to-force-tex2lyx-to-read-unicode-from-within-lyx

 Thanks

 Baris

>>> Found the answer:
>> Well, that's *an* answer: You can set up a custom "utf8" import, if you
>> want. As JMarc said, though, tex2lyx should detect the encoding. Can you
>> file a bug report about this so it doesn't get lost?
> For archival purposes, note that a bug report was created here:
>
>https://www.lyx.org/trac/ticket/11364
>
> Scott

Noted with thanks. BE.

-- 
↓↓
Please bottom-post. Start your reply here:

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Scott Kostyshak]

On Sat, Nov 03, 2018 at 11:45:14AM -0400, Richard Kimberly Heck wrote:
> On 11/2/18 5:52 PM, Baris Erkus wrote:
> > On 11/2/2018 1:33 PM, Baris Erkus wrote:
> >> Hello,
> >>
> >> I am trying to import the a .tex  file into LyX (using file->import). 
> >> The file has Turkish characters and I believe it is using UTF8 encoding. 
> >> When I import it, it does not show Turkish characters properly. I 
> >> attached the file for your play-around.
> >>
> >> When I import the file on the command line with the UTF8 option tex2lyx 
> >> -e UTF8 ch1.tex, the LyX file looks just fine except one or two characters.
> >>
> >> So is there a way to tell to tex2lyx to use UTF8 encoding from within 
> >> LyX? This issue is occured when I import a tex file that has 
> >> \input{ch1.tex} in it. So, maybe LyX has and option to force tex2lyx to 
> >> read in UTF8 for all imports?
> >>
> >> I found exactly same question on this post, but the answers seem to be 
> >> unrelated:
> >>
> >> https://lyx-users.lyx.narkive.com/Rg1StNIT/how-to-force-tex2lyx-to-read-unicode-from-within-lyx
> >>
> >> Thanks
> >>
> >> Baris
> >>
> >
> > Found the answer:
> 
> Well, that's *an* answer: You can set up a custom "utf8" import, if you
> want. As JMarc said, though, tex2lyx should detect the encoding. Can you
> file a bug report about this so it doesn't get lost?

For archival purposes, note that a bug report was created here:

  https://www.lyx.org/trac/ticket/11364

Scott


signature.asc
Description: PGP signature

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Richard Kimberly Heck]

On 11/2/18 5:52 PM, Baris Erkus wrote:
> On 11/2/2018 1:33 PM, Baris Erkus wrote:
>> Hello,
>>
>> I am trying to import the a .tex  file into LyX (using file->import). 
>> The file has Turkish characters and I believe it is using UTF8 encoding. 
>> When I import it, it does not show Turkish characters properly. I 
>> attached the file for your play-around.
>>
>> When I import the file on the command line with the UTF8 option tex2lyx 
>> -e UTF8 ch1.tex, the LyX file looks just fine except one or two characters.
>>
>> So is there a way to tell to tex2lyx to use UTF8 encoding from within 
>> LyX? This issue is occured when I import a tex file that has 
>> \input{ch1.tex} in it. So, maybe LyX has and option to force tex2lyx to 
>> read in UTF8 for all imports?
>>
>> I found exactly same question on this post, but the answers seem to be 
>> unrelated:
>>
>> https://lyx-users.lyx.narkive.com/Rg1StNIT/how-to-force-tex2lyx-to-read-unicode-from-within-lyx
>>
>> Thanks
>>
>> Baris
>>
>
> Found the answer:

Well, that's *an* answer: You can set up a custom "utf8" import, if you
want. As JMarc said, though, tex2lyx should detect the encoding. Can you
file a bug report about this so it doesn't get lost?

Riki

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Baris Erkus]

On 11/2/2018 1:33 PM, Baris Erkus wrote:

Hello,

I am trying to import the a .tex  file into LyX (using file->import).
The file has Turkish characters and I believe it is using UTF8 encoding.
When I import it, it does not show Turkish characters properly. I
attached the file for your play-around.

When I import the file on the command line with the UTF8 option tex2lyx
-e UTF8 ch1.tex, the LyX file looks just fine except one or two characters.

So is there a way to tell to tex2lyx to use UTF8 encoding from within
LyX? This issue is occured when I import a tex file that has
\input{ch1.tex} in it. So, maybe LyX has and option to force tex2lyx to
read in UTF8 for all imports?

I found exactly same question on this post, but the answers seem to be
unrelated:

https://lyx-users.lyx.narkive.com/Rg1StNIT/how-to-force-tex2lyx-to-read-unicode-from-within-lyx

Thanks

Baris



Found the answer:

See attached figure

[cid:part1.B290F4D5.E96124FC@hotmail.com]

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Baris Erkus]

Well, seems like file->import simply calls tex2lyx.exe without any 
option such as "-e UTF8". It should not be very difficult to let the 
user enter his choice of options...


On 11/2/2018 10:43 PM, Jean-Marc Lasgouttes wrote:
> Le 02/11/2018 à 10:33, Baris Erkus a écrit :
>> So is there a way to tell to tex2lyx to use UTF8 encoding from within
>> LyX? This issue is occured when I import a tex file that has
>> \input{ch1.tex} in it. So, maybe LyX has and option to force tex2lyx to
>> read in UTF8 for all imports?
>
> THe question is why tex2lyx does not see that your main file contains 
> UTF8. If LaTeX can process it as UTF8, so should tex2lyx.
>
> JMarc

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Jean-Marc Lasgouttes]

Le 02/11/2018 à 10:33, Baris Erkus a écrit :

So is there a way to tell to tex2lyx to use UTF8 encoding from within
LyX? This issue is occured when I import a tex file that has
\input{ch1.tex} in it. So, maybe LyX has and option to force tex2lyx to
read in UTF8 for all imports?


THe question is why tex2lyx does not see that your main file contains 
UTF8. If LaTeX can process it as UTF8, so should tex2lyx.


JMarc

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Scott Kostyshak]

On Fri, Nov 02, 2018 at 10:33:53AM +, Baris Erkus wrote:
> Hello,
> 
> I am trying to import the a .tex  file into LyX (using file->import). 
> The file has Turkish characters and I believe it is using UTF8 encoding. 
> When I import it, it does not show Turkish characters properly. I 
> attached the file for your play-around.
> 
> When I import the file on the command line with the UTF8 option tex2lyx 
> -e UTF8 ch1.tex, the LyX file looks just fine except one or two characters.
> 
> So is there a way to tell to tex2lyx to use UTF8 encoding from within 
> LyX? This issue is occured when I import a tex file that has 
> \input{ch1.tex} in it. So, maybe LyX has and option to force tex2lyx to 
> read in UTF8 for all imports?
> 
> I found exactly same question on this post, but the answers seem to be 
> unrelated:
> 
> https://lyx-users.lyx.narkive.com/Rg1StNIT/how-to-force-tex2lyx-to-read-unicode-from-within-lyx
> 
> Thanks
> 
> Baris
> 

I can reproduce the incorrect LaTeX import, and also Vim says the file
encoding is "utf-8". I can also reproduce with LyX 2.1.0, so it appears
it is not a recent regression. Hopefully someone who knows more will
have an idea.

Scott


signature.asc
Description: PGP signature

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Guenter Milde]

On 2012-02-19, Georg Baum wrote:
 Guenter Milde wrote:
 On 2012-02-15, Georg Baum wrote:
 Jean-Marc Lasgouttes wrote:

 Unfortunately I know next to nothing about use_non_tex_fonts. After
 reading the code and user guide a bit I see that using xetex is only
 possible if use_non_tex_fonts is true, so your patch looks fine.

 It is the other way round: using xetex or luatex is also possible without
 use_non_tex_fonts (which actually means use_fontspec). OTOH, with
 use_non_tex_fonts True, it one of xetex or luatex must be used.

 For luatex I agree, but for xetex I don't believe that you are right. Do you 
 have an example .lyx file, where use_non_tex_fonts is false, but xetex is 
 still used? If you are right, the patch introduces a regression.

Just take a normal file (with use_non_tex_fonts False) and select one
of

FileExportLaTeX (XeTex)
FileExportPDF (XeTeX)
ViewOther FormatsPDF (XeTeX)

The exported LaTeX with my somewhat outdated LyX-svn is

  %% LyX 2.1.0svn created this file.  For more info, see http://www.lyx.org/.
  %% Do not edit unless you really know what you are doing.
  \documentclass[english]{article}
  \usepackage[T1]{fontenc}
  \usepackage{babel}
  \usepackage{xunicode}
  \begin{document}
  Test
  \end{document}

and compiles fine with XeTeX.

Günter

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Guenter Milde]

On 2012-02-19, Georg Baum wrote:
 Guenter Milde wrote:
 On 2012-02-15, Georg Baum wrote:
 Jean-Marc Lasgouttes wrote:

 Unfortunately I know next to nothing about use_non_tex_fonts. After
 reading the code and user guide a bit I see that using xetex is only
 possible if use_non_tex_fonts is true, so your patch looks fine.

 It is the other way round: using xetex or luatex is also possible without
 use_non_tex_fonts (which actually means use_fontspec). OTOH, with
 use_non_tex_fonts True, it one of xetex or luatex must be used.

 For luatex I agree, but for xetex I don't believe that you are right. Do you 
 have an example .lyx file, where use_non_tex_fonts is false, but xetex is 
 still used? If you are right, the patch introduces a regression.

Just take a normal file (with use_non_tex_fonts False) and select one
of

FileExportLaTeX (XeTex)
FileExportPDF (XeTeX)
ViewOther FormatsPDF (XeTeX)

The exported LaTeX with my somewhat outdated LyX-svn is

  %% LyX 2.1.0svn created this file.  For more info, see http://www.lyx.org/.
  %% Do not edit unless you really know what you are doing.
  \documentclass[english]{article}
  \usepackage[T1]{fontenc}
  \usepackage{babel}
  \usepackage{xunicode}
  \begin{document}
  Test
  \end{document}

and compiles fine with XeTeX.

Günter

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Guenter Milde]

On 2012-02-19, Georg Baum wrote:
> Guenter Milde wrote:
>> On 2012-02-15, Georg Baum wrote:
>>> Jean-Marc Lasgouttes wrote:

>>> Unfortunately I know next to nothing about use_non_tex_fonts. After
>>> reading the code and user guide a bit I see that using xetex is only
>>> possible if use_non_tex_fonts is true, so your patch looks fine.

>> It is the other way round: using xetex or luatex is also possible without
>> use_non_tex_fonts (which actually means use_fontspec). OTOH, with
>> use_non_tex_fonts True, it one of xetex or luatex must be used.

> For luatex I agree, but for xetex I don't believe that you are right. Do you 
> have an example .lyx file, where use_non_tex_fonts is false, but xetex is 
> still used? If you are right, the patch introduces a regression.

Just take a "normal" file (with use_non_tex_fonts False) and select one
of

File>Export>LaTeX (XeTex)
File>Export>PDF (XeTeX)
View>Other Formats>PDF (XeTeX)

The exported LaTeX with my somewhat outdated LyX-svn is

  %% LyX 2.1.0svn created this file.  For more info, see http://www.lyx.org/.
  %% Do not edit unless you really know what you are doing.
  \documentclass[english]{article}
  \usepackage[T1]{fontenc}
  \usepackage{babel}
  \usepackage{xunicode}
  \begin{document}
  Test
  \end{document}

and compiles fine with XeTeX.

Günter

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Georg Baum]

Guenter Milde wrote:

 On 2012-02-15, Georg Baum wrote:
 Jean-Marc Lasgouttes wrote:
 
 Georg, Juergen, I'd like some feedback on the soundness of the patch. I
 know next to nil about xetex, and I do not know what is the current
 state of the art wrt tex2lyx.
 
 Unfortunately I know next to nothing about use_non_tex_fonts. After
 reading the code and user guide a bit I see that using xetex is only
 possible if use_non_tex_fonts is true, so your patch looks fine.
 
 It is the other way round: using xetex or luatex is also possible without
 use_non_tex_fonts (which actually means use_fontspec). OTOH, with
 use_non_tex_fonts True, it one of xetex or luatex must be used.

For luatex I agree, but for xetex I don't believe that you are right. Do you 
have an example .lyx file, where use_non_tex_fonts is false, but xetex is 
still used? If you are right, the patch introduces a regression.


Georg

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Georg Baum]

Guenter Milde wrote:

 On 2012-02-15, Georg Baum wrote:
 Jean-Marc Lasgouttes wrote:
 
 Georg, Juergen, I'd like some feedback on the soundness of the patch. I
 know next to nil about xetex, and I do not know what is the current
 state of the art wrt tex2lyx.
 
 Unfortunately I know next to nothing about use_non_tex_fonts. After
 reading the code and user guide a bit I see that using xetex is only
 possible if use_non_tex_fonts is true, so your patch looks fine.
 
 It is the other way round: using xetex or luatex is also possible without
 use_non_tex_fonts (which actually means use_fontspec). OTOH, with
 use_non_tex_fonts True, it one of xetex or luatex must be used.

For luatex I agree, but for xetex I don't believe that you are right. Do you 
have an example .lyx file, where use_non_tex_fonts is false, but xetex is 
still used? If you are right, the patch introduces a regression.


Georg

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Georg Baum]

Guenter Milde wrote:

> On 2012-02-15, Georg Baum wrote:
>> Jean-Marc Lasgouttes wrote:
> 
>>> Georg, Juergen, I'd like some feedback on the soundness of the patch. I
>>> know next to nil about xetex, and I do not know what is the current
>>> state of the art wrt tex2lyx.
> 
>> Unfortunately I know next to nothing about use_non_tex_fonts. After
>> reading the code and user guide a bit I see that using xetex is only
>> possible if use_non_tex_fonts is true, so your patch looks fine.
> 
> It is the other way round: using xetex or luatex is also possible without
> use_non_tex_fonts (which actually means use_fontspec). OTOH, with
> use_non_tex_fonts True, it one of xetex or luatex must be used.

For luatex I agree, but for xetex I don't believe that you are right. Do you 
have an example .lyx file, where use_non_tex_fonts is false, but xetex is 
still used? If you are right, the patch introduces a regression.


Georg

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Guenter Milde]

On 2012-02-15, Georg Baum wrote:
 Jürgen Spitzmüller wrote:

 If other encodings are used, LuaTeX uses the luainputenc package, which
 has the same syntax than inputenc, i.e.

 \usepackage[enc]{luainputenc}

 Maybe we have to care about that.

 luainputenc is already handled.

Fine.

 What is missing for luatex is to set 
 use_non_tex_fonts to true and the encoding to utf8 if luainputenc is not 
 used. Unfortunately this is not so easy as with xetex.

use_non_tex_fonts is actually a toggle to use the fontspec package 
(coupled with the use of polyglossia if the language-package
setting is set to auto, if I remember right).

The fontspec package can be used with the XeTeX and LuaTeX engines but not
with TeX and eTeX.

This means that tex2lyx should look for

  \usepackage{fontspec}

and set use_non_tex_fonts accordingly.

(IMV, the name for this setting should also be changed:
  - use_non_tex_fonts
  + use_fontspec
this would prevent much confusion.)

The default encoding (if neither inputenc nor luainputenc is found) may
be chosen according to use_non_tex_fonts:

True 
  XeTeX or LuaTeX engines must be used to compile
  
  -- assume their default encoding (utf8) as file encoding.

False 
  any tex engine can be used to compile
  
  -- assume either the 8-bit tex default (ASCII) or utf8 as file encoding.
  
Actually, utf8 should be a safe bet in any case when no inpuenc or
luainputenc is found, as ASCII is a subset of utf8 and no tex engine
understands 8-bit encodings (latin-1 ...) without inputenc or a similar
package.

UTF8 decoding errors should lead to a message: unknown encoding, please
specify the file encoding. If tex2lyx is called from the LyX-GUI, this
error should lead to a pop-up dialogue where you can specify an encoding.

Günter



  

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Guenter Milde]

On 2012-02-15, Georg Baum wrote:
 Jean-Marc Lasgouttes wrote:

 Georg, Juergen, I'd like some feedback on the soundness of the patch. I
 know next to nil about xetex, and I do not know what is the current
 state of the art wrt tex2lyx.

 Unfortunately I know next to nothing about use_non_tex_fonts. After reading 
 the code and user guide a bit I see that using xetex is only possible if 
 use_non_tex_fonts is true, so your patch looks fine.

It is the other way round: using xetex or luatex is also possible without
use_non_tex_fonts (which actually means use_fontspec). OTOH, with
use_non_tex_fonts True, it one of xetex or luatex must be used.


Günter

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Eric Weir]

On Feb 16, 2012, at 4:35 AM, Guenter Milde wrote:

 (IMV, the name for this setting should also be changed:
  - use_non_tex_fonts
  + use_fontspec
 this would prevent much confusion.)

While I now know what it refers to, as a new user I would otherwise have found 
the new description mystifying. I would seem that it's possible to be clear 
both to advanced users/developers and new users.

Regards,
--
Eric Weir
Decatur, GA
eew...@bellsouth.net

Style is truth. 

- Ray Bradbury

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Jürgen Spitzmüller]

Eric Weir wrote:
  (IMV, the name for this setting should also be changed:
   - use_non_tex_fonts
   + use_fontspec
 
  this would prevent much confusion.)
 
 While I now know what it refers to, as a new user I would otherwise have
 found the new description mystifying. I would seem that it's possible to be
 clear both to advanced users/developers and new users.

This is an internal value only. But I also think we should keep it as it is, 
i.e. abstract and not bound to some specific package which may change in the 
future.

Jürgen

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Guenter Milde]

On 2012-02-15, Georg Baum wrote:
 Jürgen Spitzmüller wrote:

 If other encodings are used, LuaTeX uses the luainputenc package, which
 has the same syntax than inputenc, i.e.

 \usepackage[enc]{luainputenc}

 Maybe we have to care about that.

 luainputenc is already handled.

Fine.

 What is missing for luatex is to set 
 use_non_tex_fonts to true and the encoding to utf8 if luainputenc is not 
 used. Unfortunately this is not so easy as with xetex.

use_non_tex_fonts is actually a toggle to use the fontspec package 
(coupled with the use of polyglossia if the language-package
setting is set to auto, if I remember right).

The fontspec package can be used with the XeTeX and LuaTeX engines but not
with TeX and eTeX.

This means that tex2lyx should look for

  \usepackage{fontspec}

and set use_non_tex_fonts accordingly.

(IMV, the name for this setting should also be changed:
  - use_non_tex_fonts
  + use_fontspec
this would prevent much confusion.)

The default encoding (if neither inputenc nor luainputenc is found) may
be chosen according to use_non_tex_fonts:

True 
  XeTeX or LuaTeX engines must be used to compile
  
  -- assume their default encoding (utf8) as file encoding.

False 
  any tex engine can be used to compile
  
  -- assume either the 8-bit tex default (ASCII) or utf8 as file encoding.
  
Actually, utf8 should be a safe bet in any case when no inpuenc or
luainputenc is found, as ASCII is a subset of utf8 and no tex engine
understands 8-bit encodings (latin-1 ...) without inputenc or a similar
package.

UTF8 decoding errors should lead to a message: unknown encoding, please
specify the file encoding. If tex2lyx is called from the LyX-GUI, this
error should lead to a pop-up dialogue where you can specify an encoding.

Günter



  

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Guenter Milde]

On 2012-02-15, Georg Baum wrote:
 Jean-Marc Lasgouttes wrote:

 Georg, Juergen, I'd like some feedback on the soundness of the patch. I
 know next to nil about xetex, and I do not know what is the current
 state of the art wrt tex2lyx.

 Unfortunately I know next to nothing about use_non_tex_fonts. After reading 
 the code and user guide a bit I see that using xetex is only possible if 
 use_non_tex_fonts is true, so your patch looks fine.

It is the other way round: using xetex or luatex is also possible without
use_non_tex_fonts (which actually means use_fontspec). OTOH, with
use_non_tex_fonts True, it one of xetex or luatex must be used.


Günter

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Eric Weir]

On Feb 16, 2012, at 4:35 AM, Guenter Milde wrote:

 (IMV, the name for this setting should also be changed:
  - use_non_tex_fonts
  + use_fontspec
 this would prevent much confusion.)

While I now know what it refers to, as a new user I would otherwise have found 
the new description mystifying. I would seem that it's possible to be clear 
both to advanced users/developers and new users.

Regards,
--
Eric Weir
Decatur, GA
eew...@bellsouth.net

Style is truth. 

- Ray Bradbury

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Jürgen Spitzmüller]

Eric Weir wrote:
  (IMV, the name for this setting should also be changed:
   - use_non_tex_fonts
   + use_fontspec
 
  this would prevent much confusion.)
 
 While I now know what it refers to, as a new user I would otherwise have
 found the new description mystifying. I would seem that it's possible to be
 clear both to advanced users/developers and new users.

This is an internal value only. But I also think we should keep it as it is, 
i.e. abstract and not bound to some specific package which may change in the 
future.

Jürgen

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Guenter Milde]

On 2012-02-15, Georg Baum wrote:
> Jürgen Spitzmüller wrote:

>> If other encodings are used, LuaTeX uses the "luainputenc" package, which
>> has the same syntax than inputenc, i.e.

>> \usepackage[]{luainputenc}

>> Maybe we have to care about that.

> luainputenc is already handled.

Fine.

> What is missing for luatex is to set 
> use_non_tex_fonts to true and the encoding to utf8 if luainputenc is not 
> used. Unfortunately this is not so easy as with xetex.

"use_non_tex_fonts" is actually a toggle to use the "fontspec" package 
(coupled with the use of polyglossia if the language-package
setting is set to auto, if I remember right).

The "fontspec" package can be used with the XeTeX and LuaTeX engines but not
with TeX and eTeX.

This means that tex2lyx should look for

  \usepackage{fontspec}

and set "use_non_tex_fonts" accordingly.

(IMV, the name for this setting should also be changed:
  - use_non_tex_fonts
  + use_fontspec
this would prevent much confusion.)

The default encoding (if neither "inputenc" nor "luainputenc" is found) may
be chosen according to use_non_tex_fonts:

True 
  XeTeX or LuaTeX engines must be used to compile
  
  --> assume their default encoding (utf8) as file encoding.

False 
  any tex engine can be used to compile
  
  --> assume either the 8-bit tex default (ASCII) or utf8 as file encoding.
  
Actually, utf8 should be a safe bet in any case when no "inpuenc" or
"luainputenc" is found, as ASCII is a subset of utf8 and no tex engine
understands 8-bit encodings (latin-1 ...) without inputenc or a similar
package.

UTF8 decoding errors should lead to a message: "unknown encoding, please
specify the file encoding". If tex2lyx is called from the LyX-GUI, this
error should lead to a pop-up dialogue where you can specify an encoding.

Günter



  

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Guenter Milde]

On 2012-02-15, Georg Baum wrote:
> Jean-Marc Lasgouttes wrote:

>> Georg, Juergen, I'd like some feedback on the soundness of the patch. I
>> know next to nil about xetex, and I do not know what is the current
>> state of the art wrt tex2lyx.

> Unfortunately I know next to nothing about use_non_tex_fonts. After reading 
> the code and user guide a bit I see that using xetex is only possible if 
> use_non_tex_fonts is true, so your patch looks fine.

It is the other way round: using xetex or luatex is also possible without
use_non_tex_fonts (which actually means use_fontspec). OTOH, with
use_non_tex_fonts True, it one of xetex or luatex must be used.


Günter

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Eric Weir]

On Feb 16, 2012, at 4:35 AM, Guenter Milde wrote:

> (IMV, the name for this setting should also be changed:
>  - use_non_tex_fonts
>  + use_fontspec
> this would prevent much confusion.)

While I now know what it refers to, as a new user I would otherwise have found 
the new description mystifying. I would seem that it's possible to be clear 
both to advanced users/developers and new users.

Regards,
--
Eric Weir
Decatur, GA
eew...@bellsouth.net

"Style is truth." 

- Ray Bradbury

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Jürgen Spitzmüller]

Eric Weir wrote:
> > (IMV, the name for this setting should also be changed:
> >  - use_non_tex_fonts
> >  + use_fontspec
> >
> > this would prevent much confusion.)
> 
> While I now know what it refers to, as a new user I would otherwise have
> found the new description mystifying. I would seem that it's possible to be
> clear both to advanced users/developers and new users.

This is an internal value only. But I also think we should keep it as it is, 
i.e. abstract and not bound to some specific package which may change in the 
future.

Jürgen

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Guenter Milde]

On 2012-02-14, stefano franchi wrote:
 On Tue, Feb 14, 2012 at 8:05 AM, Jean-Marc Lasgouttes

 tex2lyx is not really the culprit. The file you attach is broken : it
 does not use xetex/luatex, but it does use Unicode (XeTeX) (utf8) as
 encoding. As a result, the latex export of the file does not specify
 any encoding, and tex2lyx is not able to guess it.

 Sorry, but I don't understand what you mean by broken here. If I set
 the encoding to pure Unicode (isn't that what Unicode (XeTeX) (utf8)
 means?) shouldn't that be enough to specify that the file is
 Unicode-encoded? 

The setting Unicode (XeTeX) (utf8) specifies that the LyX-generated LaTeX
file should be utf-8 encoded Unicode 

* without forced substitutions (for characters the halfway utf8 support
  of 8-bit LaTeX does not understand or does wrong), and
  
* without calling the inputenc package.

This means that in the exported file, utf-8 encoding is used but there is no
specification of the used encoding inside the *.tex file.

You can consider this an expert setting: it allows to circumvent 
limitations but usually requires additional custom preamble code to
produce valid LaTeX files.

Similar to the use of ERT, LyX does not guarantee proper working.


 That seems obvious to me (which of course may only reflect my ignorance
 of Lyx code). If that's not true then I do not understand what is the
 meaning of the DocumentSettingsLanguageEncoding value.


It seems that tex2lyx relies on the optional argument in the

  \usepackage[encoding]{inputenc}
  
line to determine the *.tex file encoding.

IMV, it should try utf8 first in case this does not give a result.

* if the file is pure ASCII, everything is fine
* if the file is utf8 encoded, fine too
* if another encoding is used, an error occures: try again with the second
  guess.
  
Günter

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Jean-Marc Lasgouttes]

Le 15/02/2012 10:51, Guenter Milde a écrit :

IMV, it should try utf8 first in case this does not give a result.

* if the file is pure ASCII, everything is fine
* if the file is utf8 encoded, fine too
* if another encoding is used, an error occures: try again with the second
   guess.


That is not a bad solution, but probably not so easy to implement with 
our current code.


I have a patch that seems to work with XeTeX, I do not know anything 
about luatex (what shall we do with it?)


JMarc

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Jürgen Spitzmüller]

Jean-Marc Lasgouttes wrote:
 I have a patch that seems to work with XeTeX, I do not know anything 
 about luatex (what shall we do with it?)

I think your patch will work for LuaTeX as well.

Jürgen

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Jean-Marc Lasgouttes]

Le 15/02/2012 11:38, Jürgen Spitzmüller a écrit :

Jean-Marc Lasgouttes wrote:

  I have a patch that seems to work with XeTeX, I do not know anything

  about luatex (what shall we do with it?)

I think your patch will work for LuaTeX as well.


I thought LuaTeX could use encdings other than utf8. Is utf8 the 
default? And does it use the same packages as XeTeX?


I am very confused by these different engines...

JMarc

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Jürgen Spitzmüller]

Jean-Marc Lasgouttes wrote:
 I thought LuaTeX could use encdings other than utf8. Is utf8 the 
 default? And does it use the same packages as XeTeX?

LuaTeX can use other encodings, but not with non-tex fonts. So if 
\use_non_tex_fonts is true (this is what you want to check for, right?), the 
encoding of the file must be utf8.

If other encodings are used, LuaTeX uses the luainputenc package, which has 
the same syntax than inputenc, i.e.

\usepackage[enc]{luainputenc}

Maybe we have to care about that.

The font packages are the same (basically fontspec).

Jürgen

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Guenter Milde]

On 2012-02-15, Jürgen Spitzmüller wrote:
 Jean-Marc Lasgouttes wrote:

 I thought LuaTeX could use encdings other than utf8. Is utf8 the 
 default? And does it use the same packages as XeTeX?

 LuaTeX can use other encodings, but not with non-tex fonts.

Are you sure. I'd think that with luainputenc there would be no limitation
of the input encoding to utf8. Of course, with non-tex fonts the *font
encoding* is Unicode.

 So if 
 \use_non_tex_fonts is true (this is what you want to check for,
 right?), the encoding of the file must be utf8.

This might be a LyX limitation. However this is about tex2lyx, so we would
check for \usepackage{fontenc} or \usepackage{xunicode}.

I propose to also check for

 \usepackage[enc]{luainputenc}

Günter

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Georg Baum]

Jean-Marc Lasgouttes wrote:

 Georg, Juergen, I'd like some feedback on the soundness of the patch. I
 know next to nil about xetex, and I do not know what is the current
 state of the art wrt tex2lyx.

Unfortunately I know next to nothing about use_non_tex_fonts. After reading 
the code and user guide a bit I see that using xetex is only possible if 
use_non_tex_fonts is true, so your patch looks fine.

Georg

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Georg Baum]

Jürgen Spitzmüller wrote:

 LuaTeX can use other encodings, but not with non-tex fonts. So if
 \use_non_tex_fonts is true (this is what you want to check for, right?),
 the encoding of the file must be utf8.

No, it is the other way round: If a xetex package is detected, 
use_non_tex_fonts is set to true, and the encoding is set to utf8.

 If other encodings are used, LuaTeX uses the luainputenc package, which
 has the same syntax than inputenc, i.e.
 
 \usepackage[enc]{luainputenc}
 
 Maybe we have to care about that.

luainputenc is already handled. What is missing for luatex is to set 
use_non_tex_fonts to true and the encoding to utf8 if luainputenc is not 
used. Unfortunately this is not so easy as with xetex.


Georg

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Guenter Milde]

On 2012-02-14, stefano franchi wrote:
 On Tue, Feb 14, 2012 at 8:05 AM, Jean-Marc Lasgouttes

 tex2lyx is not really the culprit. The file you attach is broken : it
 does not use xetex/luatex, but it does use Unicode (XeTeX) (utf8) as
 encoding. As a result, the latex export of the file does not specify
 any encoding, and tex2lyx is not able to guess it.

 Sorry, but I don't understand what you mean by broken here. If I set
 the encoding to pure Unicode (isn't that what Unicode (XeTeX) (utf8)
 means?) shouldn't that be enough to specify that the file is
 Unicode-encoded? 

The setting Unicode (XeTeX) (utf8) specifies that the LyX-generated LaTeX
file should be utf-8 encoded Unicode 

* without forced substitutions (for characters the halfway utf8 support
  of 8-bit LaTeX does not understand or does wrong), and
  
* without calling the inputenc package.

This means that in the exported file, utf-8 encoding is used but there is no
specification of the used encoding inside the *.tex file.

You can consider this an expert setting: it allows to circumvent 
limitations but usually requires additional custom preamble code to
produce valid LaTeX files.

Similar to the use of ERT, LyX does not guarantee proper working.


 That seems obvious to me (which of course may only reflect my ignorance
 of Lyx code). If that's not true then I do not understand what is the
 meaning of the DocumentSettingsLanguageEncoding value.


It seems that tex2lyx relies on the optional argument in the

  \usepackage[encoding]{inputenc}
  
line to determine the *.tex file encoding.

IMV, it should try utf8 first in case this does not give a result.

* if the file is pure ASCII, everything is fine
* if the file is utf8 encoded, fine too
* if another encoding is used, an error occures: try again with the second
  guess.
  
Günter

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Jean-Marc Lasgouttes]

Le 15/02/2012 10:51, Guenter Milde a écrit :

IMV, it should try utf8 first in case this does not give a result.

* if the file is pure ASCII, everything is fine
* if the file is utf8 encoded, fine too
* if another encoding is used, an error occures: try again with the second
   guess.


That is not a bad solution, but probably not so easy to implement with 
our current code.


I have a patch that seems to work with XeTeX, I do not know anything 
about luatex (what shall we do with it?)


JMarc

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Jürgen Spitzmüller]

Jean-Marc Lasgouttes wrote:
 I have a patch that seems to work with XeTeX, I do not know anything 
 about luatex (what shall we do with it?)

I think your patch will work for LuaTeX as well.

Jürgen

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Jean-Marc Lasgouttes]

Le 15/02/2012 11:38, Jürgen Spitzmüller a écrit :

Jean-Marc Lasgouttes wrote:

  I have a patch that seems to work with XeTeX, I do not know anything

  about luatex (what shall we do with it?)

I think your patch will work for LuaTeX as well.


I thought LuaTeX could use encdings other than utf8. Is utf8 the 
default? And does it use the same packages as XeTeX?


I am very confused by these different engines...

JMarc

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Jürgen Spitzmüller]

Jean-Marc Lasgouttes wrote:
 I thought LuaTeX could use encdings other than utf8. Is utf8 the 
 default? And does it use the same packages as XeTeX?

LuaTeX can use other encodings, but not with non-tex fonts. So if 
\use_non_tex_fonts is true (this is what you want to check for, right?), the 
encoding of the file must be utf8.

If other encodings are used, LuaTeX uses the luainputenc package, which has 
the same syntax than inputenc, i.e.

\usepackage[enc]{luainputenc}

Maybe we have to care about that.

The font packages are the same (basically fontspec).

Jürgen

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Guenter Milde]

On 2012-02-15, Jürgen Spitzmüller wrote:
 Jean-Marc Lasgouttes wrote:

 I thought LuaTeX could use encdings other than utf8. Is utf8 the 
 default? And does it use the same packages as XeTeX?

 LuaTeX can use other encodings, but not with non-tex fonts.

Are you sure. I'd think that with luainputenc there would be no limitation
of the input encoding to utf8. Of course, with non-tex fonts the *font
encoding* is Unicode.

 So if 
 \use_non_tex_fonts is true (this is what you want to check for,
 right?), the encoding of the file must be utf8.

This might be a LyX limitation. However this is about tex2lyx, so we would
check for \usepackage{fontenc} or \usepackage{xunicode}.

I propose to also check for

 \usepackage[enc]{luainputenc}

Günter

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Georg Baum]

Jean-Marc Lasgouttes wrote:

 Georg, Juergen, I'd like some feedback on the soundness of the patch. I
 know next to nil about xetex, and I do not know what is the current
 state of the art wrt tex2lyx.

Unfortunately I know next to nothing about use_non_tex_fonts. After reading 
the code and user guide a bit I see that using xetex is only possible if 
use_non_tex_fonts is true, so your patch looks fine.

Georg

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Georg Baum]

Jürgen Spitzmüller wrote:

 LuaTeX can use other encodings, but not with non-tex fonts. So if
 \use_non_tex_fonts is true (this is what you want to check for, right?),
 the encoding of the file must be utf8.

No, it is the other way round: If a xetex package is detected, 
use_non_tex_fonts is set to true, and the encoding is set to utf8.

 If other encodings are used, LuaTeX uses the luainputenc package, which
 has the same syntax than inputenc, i.e.
 
 \usepackage[enc]{luainputenc}
 
 Maybe we have to care about that.

luainputenc is already handled. What is missing for luatex is to set 
use_non_tex_fonts to true and the encoding to utf8 if luainputenc is not 
used. Unfortunately this is not so easy as with xetex.


Georg

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Guenter Milde]

On 2012-02-14, stefano franchi wrote:
> On Tue, Feb 14, 2012 at 8:05 AM, Jean-Marc Lasgouttes

>> tex2lyx is not really the culprit. The file you attach is broken : it
>> does not use xetex/luatex, but it does use "Unicode (XeTeX) (utf8)" as
>> encoding. As a result, the latex export of the file does not specify
>> any encoding, and tex2lyx is not able to guess it.

> Sorry, but I don't understand what you mean by "broken" here. If I set
> the encoding to pure Unicode (isn't that what "Unicode (XeTeX) (utf8)"
> means?) shouldn't that be enough to specify that the file is
> Unicode-encoded? 

The setting "Unicode (XeTeX) (utf8)" specifies that the LyX-generated LaTeX
file should be utf-8 encoded Unicode 

* without "forced" substitutions (for characters the halfway utf8 support
  of 8-bit LaTeX does not understand or does wrong), and
  
* without calling the "inputenc" package.

This means that in the exported file, utf-8 encoding is used but there is no
specification of the used encoding inside the *.tex file.

You can consider this an "expert setting": it allows to circumvent 
limitations but usually requires additional custom preamble code to
produce valid LaTeX files.

Similar to the use of ERT, LyX does not guarantee proper working.


> That seems obvious to me (which of course may only reflect my ignorance
> of Lyx code). If that's not true then I do not understand what is the
> meaning of the Document>>Settings>Language>>Encoding value.


It seems that tex2lyx relies on the optional argument in the

  \usepackage[]{inputenc}
  
line to determine the *.tex file encoding.

IMV, it should try utf8 first in case this does not give a result.

* if the file is pure ASCII, everything is fine
* if the file is utf8 encoded, fine too
* if another encoding is used, an error occures: try again with the second
  guess.
  
Günter

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Jean-Marc Lasgouttes]

Le 15/02/2012 10:51, Guenter Milde a écrit :

IMV, it should try utf8 first in case this does not give a result.

* if the file is pure ASCII, everything is fine
* if the file is utf8 encoded, fine too
* if another encoding is used, an error occures: try again with the second
   guess.


That is not a bad solution, but probably not so easy to implement with 
our current code.


I have a patch that seems to work with XeTeX, I do not know anything 
about luatex (what shall we do with it?)


JMarc

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Jürgen Spitzmüller]

Jean-Marc Lasgouttes wrote:
> I have a patch that seems to work with XeTeX, I do not know anything 
> about luatex (what shall we do with it?)

I think your patch will work for LuaTeX as well.

Jürgen

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Jean-Marc Lasgouttes]

Le 15/02/2012 11:38, Jürgen Spitzmüller a écrit :

Jean-Marc Lasgouttes wrote:

 > I have a patch that seems to work with XeTeX, I do not know anything

 > about luatex (what shall we do with it?)

I think your patch will work for LuaTeX as well.


I thought LuaTeX could use encdings other than utf8. Is utf8 the 
default? And does it use the same packages as XeTeX?


I am very confused by these different engines...

JMarc

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Jürgen Spitzmüller]

Jean-Marc Lasgouttes wrote:
> I thought LuaTeX could use encdings other than utf8. Is utf8 the 
> default? And does it use the same packages as XeTeX?

LuaTeX can use other encodings, but not with non-tex fonts. So if 
\use_non_tex_fonts is true (this is what you want to check for, right?), the 
encoding of the file must be utf8.

If other encodings are used, LuaTeX uses the "luainputenc" package, which has 
the same syntax than inputenc, i.e.

\usepackage[]{luainputenc}

Maybe we have to care about that.

The font packages are the same (basically fontspec).

Jürgen

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Guenter Milde]

On 2012-02-15, Jürgen Spitzmüller wrote:
> Jean-Marc Lasgouttes wrote:

>> I thought LuaTeX could use encdings other than utf8. Is utf8 the 
>> default? And does it use the same packages as XeTeX?

> LuaTeX can use other encodings, but not with non-tex fonts.

Are you sure. I'd think that with "luainputenc" there would be no limitation
of the input encoding to utf8. Of course, with non-tex fonts the *font
encoding* is Unicode.

> So if 
> \use_non_tex_fonts is true (this is what you want to check for,
> right?), the encoding of the file must be utf8.

This might be a LyX limitation. However this is about tex2lyx, so we would
check for \usepackage{fontenc} or \usepackage{xunicode}.

I propose to also check for

> \usepackage[]{luainputenc}

Günter

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Georg Baum]

Jean-Marc Lasgouttes wrote:

> Georg, Juergen, I'd like some feedback on the soundness of the patch. I
> know next to nil about xetex, and I do not know what is the current
> state of the art wrt tex2lyx.

Unfortunately I know next to nothing about use_non_tex_fonts. After reading 
the code and user guide a bit I see that using xetex is only possible if 
use_non_tex_fonts is true, so your patch looks fine.

Georg

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Georg Baum]

Jürgen Spitzmüller wrote:

> LuaTeX can use other encodings, but not with non-tex fonts. So if
> \use_non_tex_fonts is true (this is what you want to check for, right?),
> the encoding of the file must be utf8.

No, it is the other way round: If a xetex package is detected, 
use_non_tex_fonts is set to true, and the encoding is set to utf8.

> If other encodings are used, LuaTeX uses the "luainputenc" package, which
> has the same syntax than inputenc, i.e.
> 
> \usepackage[]{luainputenc}
> 
> Maybe we have to care about that.

luainputenc is already handled. What is missing for luatex is to set 
use_non_tex_fonts to true and the encoding to utf8 if luainputenc is not 
used. Unfortunately this is not so easy as with xetex.


Georg

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Jean-Marc Lasgouttes]

Le 14/02/2012 00:47, Richard Heck a écrit :

Try out the enclosed minimal lyx file.

1. exporting to latex and reimporting into lyx from
FIleImportLatex(plain) produces garbage characters for the dashes

2. However, callingtex2lyx -e UTF8 from the command line produces
the correct file.

Still, it's surprising we fail in this particular case, so I'd guess it
is indeed a bug. I'll cross-post to devel.


tex2lyx is not really the culprit. The file you attach is broken : it 
does not use xetex/luatex, but it does use Unicode (XeTeX) (utf8) as 
encoding. As a result, the latex export of the file does not specify any 
encoding, and tex2lyx is not able to guess it.


Even better, try to visualize the file: the encoding will be bogus, 
without any intervention of tex2lyx.


What happens in this case is that the encoding is set to latin1 (utf8 is 
a better default in some sense, but if you guess it wrong, you can bet 
convertion errors, see bug #7509).


Unfortuantely, it seems that the encodings utf8 and utf8x (handled by 
plain latex), do not understand some of your hyphens, and produce a 
document that does not compile.


So your best bet is probably to produce a file that will be imported as 
XeTeX/LuaTeX. This will happen when some characteristic packages are 
recognized, which will happen after the followinf patch is applied.


Georg, Juergen, I'd like some feedback on the soundness of the patch. I 
know next to nil about xetex, and I do not know what is the current 
state of the art wrt tex2lyx.


JMarc


Index: src/tex2lyx/Preamble.cpp
===
--- src/tex2lyx/Preamble.cpp	(révision 40747)
+++ src/tex2lyx/Preamble.cpp	(copie de travail)
@@ -401,6 +401,7 @@
 	h_font_sans   = default;
 	h_font_typewriter = default;
 	h_font_default_family = default;
+	h_use_non_tex_fonts   = false;
 	h_font_sc = false;
 	h_font_osf= false;
 	h_font_sf_scale   = 100;
@@ -581,8 +582,11 @@
 	add_package(name, options);
 	string scale;
 
-	if (is_known(name, known_xetex_packages))
+	if (is_known(name, known_xetex_packages)) {
 		xetex = true;
+		h_use_non_tex_fonts = true;
+		p.setEncoding(utf8);
+	}
 
 	// roman fonts
 	if (is_known(name, known_roman_fonts)) {
@@ -901,6 +905,7 @@
 	\\font_sans   h_font_sans  \n
 	\\font_typewriter   h_font_typewriter  \n
 	\\font_default_family   h_font_default_family  \n
+	\\use_non_tex_fonts   h_use_non_tex_fonts  \n
 	\\font_sc   h_font_sc  \n
 	\\font_osf   h_font_osf  \n
 	\\font_sf_scale   h_font_sf_scale  \n
Index: src/tex2lyx/Preamble.h
===
--- src/tex2lyx/Preamble.h	(révision 40747)
+++ src/tex2lyx/Preamble.h	(copie de travail)
@@ -90,6 +90,7 @@
 	std::string h_font_sans;
 	std::string h_font_typewriter;
 	std::string h_font_default_family;
+	std::string h_use_non_tex_fonts;
 	std::string h_font_sc;
 	std::string h_font_osf;
 	std::string h_font_sf_scale;

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Jürgen Spitzmüller]

Jean-Marc Lasgouttes wrote:
 Georg, Juergen, I'd like some feedback on the soundness of the patch. I 
 know next to nil about xetex, and I do not know what is the current 
 state of the art wrt tex2lyx.

The latter I don't know either, but the patch looks sane. We set the encoding 
to utf8 internally as well if \use_non_tex_fonts is true.

Jürgen

Re: How to force tex2lyx to read unicode (from within Lyx)?

[stefano franchi]

On Tue, Feb 14, 2012 at 8:05 AM, Jean-Marc Lasgouttes
lasgout...@lyx.org wrote:
 Le 14/02/2012 00:47, Richard Heck a écrit :

 Try out the enclosed minimal lyx file.

 1. exporting to latex and reimporting into lyx from
 FIleImportLatex(plain) produces garbage characters for the dashes

 2. However, callingtex2lyx -e UTF8 from the command line produces
 the correct file.

 Still, it's surprising we fail in this particular case, so I'd guess it
 is indeed a bug. I'll cross-post to devel.


 tex2lyx is not really the culprit. The file you attach is broken : it does
 not use xetex/luatex, but it does use Unicode (XeTeX) (utf8) as encoding.
 As a result, the latex export of the file does not specify any encoding, and
 tex2lyx is not able to guess it.

Sorry, but I don't understand what you mean by broken here. If I set
the encoding to pure Unicode (isn't that what Unicode (XeTeX) (utf8)
means?) shouldn't that be enough to specify that the file is
Unicode-encoded? That seems obvious to me (which of course may only
reflect my ignorance of Lyx code). If that's not true then I do not
understand what is the meaning of the
DocumentSettingsLanguageEncoding value.

Cheers,

Stefano
-- 
__
Stefano Franchi
Associate Research Professor
Department of Hispanic Studies            Ph:   +1 (979) 845-2125
Texas AM University                          Fax:  +1 (979) 845-6421
College Station, Texas, USA

stef...@tamu.edu
http://stefano.cleinias.org

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Jean-Marc Lasgouttes]

Le 14/02/2012 00:47, Richard Heck a écrit :

Try out the enclosed minimal lyx file.

1. exporting to latex and reimporting into lyx from
FIleImportLatex(plain) produces garbage characters for the dashes

2. However, callingtex2lyx -e UTF8 from the command line produces
the correct file.

Still, it's surprising we fail in this particular case, so I'd guess it
is indeed a bug. I'll cross-post to devel.


tex2lyx is not really the culprit. The file you attach is broken : it 
does not use xetex/luatex, but it does use Unicode (XeTeX) (utf8) as 
encoding. As a result, the latex export of the file does not specify any 
encoding, and tex2lyx is not able to guess it.


Even better, try to visualize the file: the encoding will be bogus, 
without any intervention of tex2lyx.


What happens in this case is that the encoding is set to latin1 (utf8 is 
a better default in some sense, but if you guess it wrong, you can bet 
convertion errors, see bug #7509).


Unfortuantely, it seems that the encodings utf8 and utf8x (handled by 
plain latex), do not understand some of your hyphens, and produce a 
document that does not compile.


So your best bet is probably to produce a file that will be imported as 
XeTeX/LuaTeX. This will happen when some characteristic packages are 
recognized, which will happen after the followinf patch is applied.


Georg, Juergen, I'd like some feedback on the soundness of the patch. I 
know next to nil about xetex, and I do not know what is the current 
state of the art wrt tex2lyx.


JMarc


Index: src/tex2lyx/Preamble.cpp
===
--- src/tex2lyx/Preamble.cpp	(révision 40747)
+++ src/tex2lyx/Preamble.cpp	(copie de travail)
@@ -401,6 +401,7 @@
 	h_font_sans   = default;
 	h_font_typewriter = default;
 	h_font_default_family = default;
+	h_use_non_tex_fonts   = false;
 	h_font_sc = false;
 	h_font_osf= false;
 	h_font_sf_scale   = 100;
@@ -581,8 +582,11 @@
 	add_package(name, options);
 	string scale;
 
-	if (is_known(name, known_xetex_packages))
+	if (is_known(name, known_xetex_packages)) {
 		xetex = true;
+		h_use_non_tex_fonts = true;
+		p.setEncoding(utf8);
+	}
 
 	// roman fonts
 	if (is_known(name, known_roman_fonts)) {
@@ -901,6 +905,7 @@
 	\\font_sans   h_font_sans  \n
 	\\font_typewriter   h_font_typewriter  \n
 	\\font_default_family   h_font_default_family  \n
+	\\use_non_tex_fonts   h_use_non_tex_fonts  \n
 	\\font_sc   h_font_sc  \n
 	\\font_osf   h_font_osf  \n
 	\\font_sf_scale   h_font_sf_scale  \n
Index: src/tex2lyx/Preamble.h
===
--- src/tex2lyx/Preamble.h	(révision 40747)
+++ src/tex2lyx/Preamble.h	(copie de travail)
@@ -90,6 +90,7 @@
 	std::string h_font_sans;
 	std::string h_font_typewriter;
 	std::string h_font_default_family;
+	std::string h_use_non_tex_fonts;
 	std::string h_font_sc;
 	std::string h_font_osf;
 	std::string h_font_sf_scale;

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Jürgen Spitzmüller]

Jean-Marc Lasgouttes wrote:
 Georg, Juergen, I'd like some feedback on the soundness of the patch. I 
 know next to nil about xetex, and I do not know what is the current 
 state of the art wrt tex2lyx.

The latter I don't know either, but the patch looks sane. We set the encoding 
to utf8 internally as well if \use_non_tex_fonts is true.

Jürgen

Re: How to force tex2lyx to read unicode (from within Lyx)?

[stefano franchi]

On Tue, Feb 14, 2012 at 8:05 AM, Jean-Marc Lasgouttes
lasgout...@lyx.org wrote:
 Le 14/02/2012 00:47, Richard Heck a écrit :

 Try out the enclosed minimal lyx file.

 1. exporting to latex and reimporting into lyx from
 FIleImportLatex(plain) produces garbage characters for the dashes

 2. However, callingtex2lyx -e UTF8 from the command line produces
 the correct file.

 Still, it's surprising we fail in this particular case, so I'd guess it
 is indeed a bug. I'll cross-post to devel.


 tex2lyx is not really the culprit. The file you attach is broken : it does
 not use xetex/luatex, but it does use Unicode (XeTeX) (utf8) as encoding.
 As a result, the latex export of the file does not specify any encoding, and
 tex2lyx is not able to guess it.

Sorry, but I don't understand what you mean by broken here. If I set
the encoding to pure Unicode (isn't that what Unicode (XeTeX) (utf8)
means?) shouldn't that be enough to specify that the file is
Unicode-encoded? That seems obvious to me (which of course may only
reflect my ignorance of Lyx code). If that's not true then I do not
understand what is the meaning of the
DocumentSettingsLanguageEncoding value.

Cheers,

Stefano
-- 
__
Stefano Franchi
Associate Research Professor
Department of Hispanic Studies            Ph:   +1 (979) 845-2125
Texas AM University                          Fax:  +1 (979) 845-6421
College Station, Texas, USA

stef...@tamu.edu
http://stefano.cleinias.org

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Jean-Marc Lasgouttes]

Le 14/02/2012 00:47, Richard Heck a écrit :

Try out the enclosed minimal lyx file.

1. exporting to latex and reimporting into lyx from
FIle>>Import>>Latex(plain) produces garbage characters for the dashes

2. However, calling>tex2lyx -e UTF8 from the command line produces
the correct file.

Still, it's surprising we fail in this particular case, so I'd guess it
is indeed a bug. I'll cross-post to devel.


tex2lyx is not really the culprit. The file you attach is broken : it 
does not use xetex/luatex, but it does use "Unicode (XeTeX) (utf8)" as 
encoding. As a result, the latex export of the file does not specify any 
encoding, and tex2lyx is not able to guess it.


Even better, try to visualize the file: the encoding will be bogus, 
without any intervention of tex2lyx.


What happens in this case is that the encoding is set to latin1 (utf8 is 
a better default in some sense, but if you guess it wrong, you can bet 
convertion errors, see bug #7509).


Unfortuantely, it seems that the encodings utf8 and utf8x (handled by 
plain latex), do not understand some of your hyphens, and produce a 
document that does not compile.


So your best bet is probably to produce a file that will be imported as 
XeTeX/LuaTeX. This will happen when some characteristic packages are 
recognized, which will happen after the followinf patch is applied.


Georg, Juergen, I'd like some feedback on the soundness of the patch. I 
know next to nil about xetex, and I do not know what is the current 
state of the art wrt tex2lyx.


JMarc


Index: src/tex2lyx/Preamble.cpp
===
--- src/tex2lyx/Preamble.cpp	(révision 40747)
+++ src/tex2lyx/Preamble.cpp	(copie de travail)
@@ -401,6 +401,7 @@
 	h_font_sans   = "default";
 	h_font_typewriter = "default";
 	h_font_default_family = "default";
+	h_use_non_tex_fonts   = "false";
 	h_font_sc = "false";
 	h_font_osf= "false";
 	h_font_sf_scale   = "100";
@@ -581,8 +582,11 @@
 	add_package(name, options);
 	string scale;
 
-	if (is_known(name, known_xetex_packages))
+	if (is_known(name, known_xetex_packages)) {
 		xetex = true;
+		h_use_non_tex_fonts = "true";
+		p.setEncoding("utf8");
+	}
 
 	// roman fonts
 	if (is_known(name, known_roman_fonts)) {
@@ -901,6 +905,7 @@
 	   << "\\font_sans " << h_font_sans << "\n"
 	   << "\\font_typewriter " << h_font_typewriter << "\n"
 	   << "\\font_default_family " << h_font_default_family << "\n"
+	   << "\\use_non_tex_fonts " << h_use_non_tex_fonts << "\n"
 	   << "\\font_sc " << h_font_sc << "\n"
 	   << "\\font_osf " << h_font_osf << "\n"
 	   << "\\font_sf_scale " << h_font_sf_scale << "\n"
Index: src/tex2lyx/Preamble.h
===
--- src/tex2lyx/Preamble.h	(révision 40747)
+++ src/tex2lyx/Preamble.h	(copie de travail)
@@ -90,6 +90,7 @@
 	std::string h_font_sans;
 	std::string h_font_typewriter;
 	std::string h_font_default_family;
+	std::string h_use_non_tex_fonts;
 	std::string h_font_sc;
 	std::string h_font_osf;
 	std::string h_font_sf_scale;

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Jürgen Spitzmüller]

Jean-Marc Lasgouttes wrote:
> Georg, Juergen, I'd like some feedback on the soundness of the patch. I 
> know next to nil about xetex, and I do not know what is the current 
> state of the art wrt tex2lyx.

The latter I don't know either, but the patch looks sane. We set the encoding 
to utf8 internally as well if \use_non_tex_fonts is true.

Jürgen

Re: How to force tex2lyx to read unicode (from within Lyx)?

[stefano franchi]

On Tue, Feb 14, 2012 at 8:05 AM, Jean-Marc Lasgouttes
 wrote:
> Le 14/02/2012 00:47, Richard Heck a écrit :
>
>>> Try out the enclosed minimal lyx file.
>>>
>>> 1. exporting to latex and reimporting into lyx from
>>> FIle>>Import>>Latex(plain) produces garbage characters for the dashes
>>>
>>> 2. However, calling>tex2lyx -e UTF8 from the command line produces
>>> the correct file.
>>
>> Still, it's surprising we fail in this particular case, so I'd guess it
>> is indeed a bug. I'll cross-post to devel.
>
>
> tex2lyx is not really the culprit. The file you attach is broken : it does
> not use xetex/luatex, but it does use "Unicode (XeTeX) (utf8)" as encoding.
> As a result, the latex export of the file does not specify any encoding, and
> tex2lyx is not able to guess it.
>
Sorry, but I don't understand what you mean by "broken" here. If I set
the encoding to pure Unicode (isn't that what "Unicode (XeTeX) (utf8)"
means?) shouldn't that be enough to specify that the file is
Unicode-encoded? That seems obvious to me (which of course may only
reflect my ignorance of Lyx code). If that's not true then I do not
understand what is the meaning of the
Document>>Settings>Language>>Encoding value.

Cheers,

Stefano
-- 
__
Stefano Franchi
Associate Research Professor
Department of Hispanic Studies            Ph:   +1 (979) 845-2125
Texas A University                          Fax:  +1 (979) 845-6421
College Station, Texas, USA

stef...@tamu.edu
http://stefano.cleinias.org

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Richard Heck]

On 02/13/2012 05:41 PM, stefano franchi wrote:

I have been helping Eric Weir with his Scrivener--LaTeX--Lyx import
and we have narrowed down the problem to Lyx not importing a
Unicode-encoded file as Unicode.

I'm glad to hear someone was able to help with this. Thanks for your 
efforts on behalf of the LyX community.



So tex2lyx is the culprit. Wasn't this solved some time ago, however?I
mean: automatic recognition of the imported file encoding?

Well, that's a complicated issue. They may have tried to do a better 
job, but recognizing the file encoding completely reliably is not in 
general possible. There's some famous example of this.



Try out the enclosed minimal lyx file.

1. exporting to latex and reimporting into lyx from
FIleImportLatex(plain) produces garbage characters for the dashes

2. However, callingtex2lyx -e UTF8 from the command line produces
the correct file.
Still, it's surprising we fail in this particular case, so I'd guess it 
is indeed a bug. I'll cross-post to devel.


Richard

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Richard Heck]

On 02/13/2012 05:41 PM, stefano franchi wrote:

I have been helping Eric Weir with his Scrivener--LaTeX--Lyx import
and we have narrowed down the problem to Lyx not importing a
Unicode-encoded file as Unicode.

I'm glad to hear someone was able to help with this. Thanks for your 
efforts on behalf of the LyX community.



So tex2lyx is the culprit. Wasn't this solved some time ago, however?I
mean: automatic recognition of the imported file encoding?

Well, that's a complicated issue. They may have tried to do a better 
job, but recognizing the file encoding completely reliably is not in 
general possible. There's some famous example of this.



Try out the enclosed minimal lyx file.

1. exporting to latex and reimporting into lyx from
FIleImportLatex(plain) produces garbage characters for the dashes

2. However, callingtex2lyx -e UTF8 from the command line produces
the correct file.
Still, it's surprising we fail in this particular case, so I'd guess it 
is indeed a bug. I'll cross-post to devel.


Richard

Re: How to force tex2lyx to read unicode (from within Lyx)?

[Richard Heck]

On 02/13/2012 05:41 PM, stefano franchi wrote:

I have been helping Eric Weir with his Scrivener-->LaTeX-->Lyx import
and we have narrowed down the problem to Lyx not importing a
Unicode-encoded file as Unicode.

I'm glad to hear someone was able to help with this. Thanks for your 
efforts on behalf of the LyX community.



So tex2lyx is the culprit. Wasn't this solved some time ago, however?I
mean: automatic recognition of the imported file encoding?

Well, that's a complicated issue. They may have tried to do a better 
job, but recognizing the file encoding completely reliably is not in 
general possible. There's some famous example of this.



Try out the enclosed minimal lyx file.

1. exporting to latex and reimporting into lyx from
FIle>>Import>>Latex(plain) produces garbage characters for the dashes

2. However, calling>tex2lyx -e UTF8 from the command line produces
the correct file.
Still, it's surprising we fail in this particular case, so I'd guess it 
is indeed a bug. I'll cross-post to devel.


Richard