Re: Re: [Patch] allow unicode in layout style name
Jean-Marc Lasgouttes a écrit : Abdelrazak == Abdelrazak Younes [EMAIL PROTECTED] writes: Why not? It is easier to use has a file with coherent encoding that one that uses a mixed encoding. Abdelrazak I agree with Jose. I cannot see a reason why mixed Abdelrazak encoding would be preferred. I am not sure we are discussing the same thing. Assume one has a file in 1.4 file format with contents in latin1 and some layout names in latin1 too. What do you propose to do? I see two solutions. 1/ first convert the latin1 file to utf8 in one sweep and hope that lyx2lyx will not butcher it when translating to 1.5 file format 2/ first convert to 1.5 file format. Then only the layout names should be converted in a second step. I do not think that iconv would appreciate to see UTF8 characters in a file which is supposed to be latin1. JMarc Hi, I just had time to test Abdel patch layout_name_is_unicode.patch with the latest svn: After a quick test I think it works perfectly! All my layouts (converted in UTF-8) are recognized and lyx accept to open every files created with previous versions (in fact almost: there is still a problem with old files using the tag \language frenchb and perhaps others...?) as they are (i.e. without any conversion). Thus, for me, this is probably the best solution! Thank you very much PhC
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
On Tue, 10 Jul 2007, José Matos wrote: On Tuesday 10 July 2007 15:16:56 [EMAIL PROTECTED] wrote: What's the risk of embarrassing[*] bugs? (As this is just before the release of 1.5.0). There is no RC3 planned, right? Do you want a formula with deltas and epsilons? ;-) No there is not any RC3 planned. Do you think we need one? Not if you don't, it's your call. Just wanted to mention the possibility. Regarding this subject I think that this course of action is sound so I don't expect any major surprises. (Famous last words). If something does happen, your words might well make it into LyX's famous quotes :-) /C -- Christian Ridderström, +46-8-768 39 44 http://www.md.kth.se/~chr
Re: [Patch] allow unicode in layout style name
On Tuesday 10 July 2007 15:13:49 Philippe Charpentier wrote: (in fact almost: there is still a problem with old files using the tag \language frenchb and perhaps others...?) as they are (i.e. without any conversion). lyx2lyx converts those documents coming from 1.3.x or before. Isn't that working? Thus, for me, this is probably the best solution! Thank you very much PhC -- José Abílio
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
On Wednesday 11 July 2007 11:33:32 [EMAIL PROTECTED] wrote: Regarding this subject I think that this course of action is sound so I don't expect any major surprises. (Famous last words). If something does happen, your words might well make it into LyX's famous quotes :-) AFAIU the right solution implies a synchronised change for layout2layout and lyx2lyx. This also implies to distinguish between the style GUI name and the style ID, with the style ID being an ascii identifier. The amount of changes required to have this properly implemented scares me at such a late stage of the development of 1.5.0. The proper solution should then be implemented at early 1.6 development cycle. So in resume, I am aware that this is not the most elegant implementation but for the moment it is the possible due to our stability constraints. /C -- José Abílio
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
José Matos wrote: On Wednesday 11 July 2007 11:33:32 [EMAIL PROTECTED] wrote: Regarding this subject I think that this course of action is sound so I don't expect any major surprises. (Famous last words). If something does happen, your words might well make it into LyX's famous quotes :-) AFAIU the right solution implies a synchronised change for layout2layout and lyx2lyx. This also implies to distinguish between the style GUI name and the style ID, with the style ID being an ascii identifier. I disagree, IMHO supporting unicode layout names is the right solution for the user. The amount of changes required to have this properly implemented scares me at such a late stage of the development of 1.5.0. I agree with this. The proper solution should then be implemented at early 1.6 development cycle. So in resume, I am aware that this is not the most elegant implementation but for the moment it is the possible due to our stability constraints. So, shall I commit? The conversion from latin-1 to UTF-8 encodings can happen afterwards because all our layout files contains only ascii style names. Abdel.
Re: [Patch] allow unicode in layout style name
José Abilio a écrit : On Tuesday 10 July 2007 15:13:49 Philippe Charpentier wrote: (in fact almost: there is still a problem with old files using the tag \language frenchb and perhaps others...?) as they are (i.e. without any conversion). lyx2lyx converts those documents coming from 1.3.x or before. Isn't that working? When I open some old files I obtain the following error: Traceback (most recent call last): File /usr/local/lyx-1.5-UTF-8/share/lyx/./lyx2lyx/lyx2lyx, line 101, in module sys.exit(main(sys.argv)) File /usr/local/lyx-1.5-UTF-8/share/lyx/./lyx2lyx/lyx2lyx, line 92, in main file = LyX.File(end_format, input, output, error, debug, try_hard, cjk_encoding) File /usr/local/lyx-1.5-UTF-8/share/lyx/lyx2lyx/LyX.py, line 556, in __init__ self.read() File /usr/local/lyx-1.5-UTF-8/share/lyx/lyx2lyx/LyX.py, line 242, in read self.encoding = get_encoding(self.language, self.inputencoding, self.format, self.cjk_encoding) File /usr/local/lyx-1.5-UTF-8/share/lyx/lyx2lyx/LyX.py, line 128, in get_encoding return lang[language][3] KeyError: 'frenchb' Error: Échec du script de conversion but, if I manually replace \language frenchb by \language french in the file, lyx2lyx converts it successfully... PhC
Re: [Patch] allow unicode in layout style name
On Wednesday 11 July 2007 13:58:29 Philippe Charpentier wrote: lyx2lyx converts those documents coming from 1.3.x or before. Isn't that working? When I open some old files I obtain the following error: How old are those files? What is the file format number for those files? -- José Abílio
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
On Wednesday 11 July 2007 13:34:18 Abdelrazak Younes wrote: José Matos wrote: On Wednesday 11 July 2007 11:33:32 [EMAIL PROTECTED] wrote: Regarding this subject I think that this course of action is sound so I don't expect any major surprises. (Famous last words). If something does happen, your words might well make it into LyX's famous quotes :-) AFAIU the right solution implies a synchronised change for layout2layout and lyx2lyx. This also implies to distinguish between the style GUI name and the style ID, with the style ID being an ascii identifier. I disagree, IMHO supporting unicode layout names is the right solution for the user. ??? How are we disagreeing? We are discussing an implementation detail, the final result should be the same. Instead of the current: Style Anão ... End I am proposing Style Anao LyXName Anão ... End FWIW anão is dwarf. :-) So, shall I commit? The conversion from latin-1 to UTF-8 encodings can happen afterwards because all our layout files contains only ascii style names. Yes. Abdel. -- José Abílio
Re: [Patch] allow unicode in layout style name
José Abilio à écrit: lyx2lyx converts those documents coming from 1.3.x or before. Isn't that working? When I open some old files I obtain the following error: How old are those files? What is the file format number for those files? The format number of the file I tested is 221 and it was created in 2003 (I think) PhC
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
José Matos wrote: On Wednesday 11 July 2007 13:34:18 Abdelrazak Younes wrote: ??? How are we disagreeing? We are discussing an implementation detail, the final result should be the same. Instead of the current: Style Anão ... End I am proposing Style Anao LyXName Anão ... End That's where we disagree, I don't see a need to force the user to double his work. We'll open the debate again when 1.6.0svn is out if you want :-) FWIW anão is dwarf. :-) So, shall I commit? The conversion from latin-1 to UTF-8 encodings can happen afterwards because all our layout files contains only ascii style names. Yes. Done. Abdel.
Re: [Patch] allow unicode in layout style name
On Wednesday 11 July 2007 14:31:40 Philippe Charpentier wrote: José Abilio à écrit: lyx2lyx converts those documents coming from 1.3.x or before. Isn't that working? When I open some old files I obtain the following error: How old are those files? What is the file format number for those files? The format number of the file I tested is 221 and it was created in 2003 (I think) PhC That makes sense. Would it be possible to send me privately one of those files? It seems that the filter responsible for the convertion is failling and I would like to understand why. In this case it would be enough to study the header section (the part of the file before the real text comes)... -- José Abílio
Re: [Patch] allow unicode in layout style name
On Wednesday 11 July 2007 14:55:09 José Matos wrote: In this case it would be enough to study the header section (the part of the file before the real text comes)... OK. I found the real culprit, we don't have the definition of frenchb in lyx2lyx_lang. Adding that and it works again. I have checked also that frenchb was the only problematic language. So all is well now. :-) -- José Abílio
Re: Re: [Patch] allow unicode in layout style name
Jean-Marc Lasgouttes a écrit : "Abdelrazak" == Abdelrazak Younes <[EMAIL PROTECTED]> writes: Why not? It is easier to use has a file with coherent encoding that one that uses a mixed encoding. Abdelrazak> I agree with Jose. I cannot see a reason why mixed Abdelrazak> encoding would be preferred. I am not sure we are discussing the same thing. Assume one has a file in 1.4 file format with contents in latin1 and some layout names in latin1 too. What do you propose to do? I see two solutions. 1/ first convert the latin1 file to utf8 in one sweep and hope that lyx2lyx will not butcher it when translating to 1.5 file format 2/ first convert to 1.5 file format. Then only the layout names should be converted in a second step. I do not think that iconv would appreciate to see UTF8 characters in a file which is supposed to be latin1. JMarc Hi, I just had time to test Abdel patch "layout_name_is_unicode.patch" with the latest svn: After a quick test I think it works perfectly! All my layouts (converted in UTF-8) are recognized and lyx accept to open every files created with previous versions (in fact almost: there is still a problem with old files using the tag "\language frenchb" and perhaps others...?) as they are (i.e. without any conversion). Thus, for me, this is probably the best solution! Thank you very much PhC
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
On Tue, 10 Jul 2007, José Matos wrote: On Tuesday 10 July 2007 15:16:56 [EMAIL PROTECTED] wrote: What's the risk of embarrassing[*] bugs? (As this is just before the release of 1.5.0). There is no RC3 planned, right? Do you want a formula with deltas and epsilons? ;-) No there is not any RC3 planned. Do you think we need one? Not if you don't, it's your call. Just wanted to mention the possibility. Regarding this subject I think that this course of action is sound so I don't expect any major surprises. (Famous last words). If something does happen, your words might well make it into LyX's famous quotes :-) /C -- Christian Ridderström, +46-8-768 39 44 http://www.md.kth.se/~chr
Re: [Patch] allow unicode in layout style name
On Tuesday 10 July 2007 15:13:49 Philippe Charpentier wrote: > (in fact almost: there is still a problem with old files using the tag > "\language frenchb" and perhaps others...?) > as they are (i.e. without any conversion). lyx2lyx converts those documents coming from 1.3.x or before. Isn't that working? > Thus, for me, this is probably the best solution! > > Thank you very much > > PhC -- José Abílio
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
On Wednesday 11 July 2007 11:33:32 [EMAIL PROTECTED] wrote: > > Regarding this subject I think that this course of action is sound so I > > don't expect any major surprises. (Famous last words). > > If something does happen, your words might well make it into LyX's famous > quotes :-) AFAIU the right solution implies a synchronised change for layout2layout and lyx2lyx. This also implies to distinguish between the style GUI name and the style ID, with the style ID being an ascii identifier. The amount of changes required to have this properly implemented scares me at such a late stage of the development of 1.5.0. The proper solution should then be implemented at early 1.6 development cycle. So in resume, I am aware that this is not the most elegant implementation but for the moment it is the possible due to our stability constraints. > /C -- José Abílio
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
José Matos wrote: On Wednesday 11 July 2007 11:33:32 [EMAIL PROTECTED] wrote: Regarding this subject I think that this course of action is sound so I don't expect any major surprises. (Famous last words). If something does happen, your words might well make it into LyX's famous quotes :-) AFAIU the right solution implies a synchronised change for layout2layout and lyx2lyx. This also implies to distinguish between the style GUI name and the style ID, with the style ID being an ascii identifier. I disagree, IMHO supporting unicode layout names is the right solution for the user. The amount of changes required to have this properly implemented scares me at such a late stage of the development of 1.5.0. I agree with this. The proper solution should then be implemented at early 1.6 development cycle. So in resume, I am aware that this is not the most elegant implementation but for the moment it is the possible due to our stability constraints. So, shall I commit? The conversion from latin-1 to UTF-8 encodings can happen afterwards because all our layout files contains only ascii style names. Abdel.
Re: [Patch] allow unicode in layout style name
José Abilio a écrit : On Tuesday 10 July 2007 15:13:49 Philippe Charpentier wrote: (in fact almost: there is still a problem with old files using the tag "\language frenchb" and perhaps others...?) as they are (i.e. without any conversion). lyx2lyx converts those documents coming from 1.3.x or before. Isn't that working? When I open some old files I obtain the following error: Traceback (most recent call last): File "/usr/local/lyx-1.5-UTF-8/share/lyx/./lyx2lyx/lyx2lyx", line 101, in sys.exit(main(sys.argv)) File "/usr/local/lyx-1.5-UTF-8/share/lyx/./lyx2lyx/lyx2lyx", line 92, in main file = LyX.File(end_format, input, output, error, debug, try_hard, cjk_encoding) File "/usr/local/lyx-1.5-UTF-8/share/lyx/lyx2lyx/LyX.py", line 556, in __init__ self.read() File "/usr/local/lyx-1.5-UTF-8/share/lyx/lyx2lyx/LyX.py", line 242, in read self.encoding = get_encoding(self.language, self.inputencoding, self.format, self.cjk_encoding) File "/usr/local/lyx-1.5-UTF-8/share/lyx/lyx2lyx/LyX.py", line 128, in get_encoding return lang[language][3] KeyError: 'frenchb' Error: Échec du script de conversion but, if I manually replace "\language frenchb" by "\language french" in the file, lyx2lyx converts it successfully... PhC
Re: [Patch] allow unicode in layout style name
On Wednesday 11 July 2007 13:58:29 Philippe Charpentier wrote: > > lyx2lyx converts those documents coming from 1.3.x or before. Isn't > > that working? > > When I open some old files I obtain the following error: How old are those files? What is the file format number for those files? -- José Abílio
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
On Wednesday 11 July 2007 13:34:18 Abdelrazak Younes wrote: > José Matos wrote: > > On Wednesday 11 July 2007 11:33:32 [EMAIL PROTECTED] wrote: > >>> Regarding this subject I think that this course of action is sound so > >>> I don't expect any major surprises. (Famous last words). > >> > >> If something does happen, your words might well make it into LyX's > >> famous quotes :-) > > > > AFAIU the right solution implies a synchronised change for > > layout2layout and lyx2lyx. This also implies to distinguish between the > > style GUI name and the style ID, with the style ID being an ascii > > identifier. > > I disagree, IMHO supporting unicode layout names is the right solution > for the user. ??? How are we disagreeing? We are discussing an implementation detail, the final result should be the same. Instead of the current: Style Anão ... End I am proposing Style Anao LyXName "Anão" ... End FWIW anão is dwarf. :-) > So, shall I commit? The conversion from latin-1 to UTF-8 encodings can > happen afterwards because all our layout files contains only ascii style > names. Yes. > Abdel. -- José Abílio
Re: [Patch] allow unicode in layout style name
José Abilio à écrit: > lyx2lyx converts those documents coming from 1.3.x or before. Isn't > that working? When I open some old files I obtain the following error: How old are those files? What is the file format number for those files? The format number of the file I tested is 221 and it was created in 2003 (I think) PhC
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
José Matos wrote: On Wednesday 11 July 2007 13:34:18 Abdelrazak Younes wrote: ??? How are we disagreeing? We are discussing an implementation detail, the final result should be the same. Instead of the current: Style Anão ... End I am proposing Style Anao LyXName "Anão" ... End That's where we disagree, I don't see a need to force the user to double his work. We'll open the debate again when 1.6.0svn is out if you want :-) FWIW anão is dwarf. :-) So, shall I commit? The conversion from latin-1 to UTF-8 encodings can happen afterwards because all our layout files contains only ascii style names. Yes. Done. Abdel.
Re: [Patch] allow unicode in layout style name
On Wednesday 11 July 2007 14:31:40 Philippe Charpentier wrote: > José Abilio à écrit: > >> > lyx2lyx converts those documents coming from 1.3.x or before. Isn't > >> > that working? > >> > >> When I open some old files I obtain the following error: > > > >How old are those files? What is the file format number for those files? > > The format number of the file I tested is 221 and it was created in 2003 (I > think) > > PhC That makes sense. Would it be possible to send me privately one of those files? It seems that the filter responsible for the convertion is failling and I would like to understand why. In this case it would be enough to study the header section (the part of the file before the real text comes)... -- José Abílio
Re: [Patch] allow unicode in layout style name
On Wednesday 11 July 2007 14:55:09 José Matos wrote: > In this case it would be enough to study the header section (the part of > the file before the real text comes)... OK. I found the real culprit, we don't have the definition of frenchb in lyx2lyx_lang. Adding that and it works again. I have checked also that frenchb was the only problematic language. So all is well now. :-) -- José Abílio
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
Abdelrazak Younes wrote: Jean-Pierre Chrétien wrote: Abdelrazak Younes [EMAIL PROTECTED] writes: 1) provide a python script that converts the layout as good as possible: Théorème - Theoreme Liste_à_puce - Liste_a_puce The problem is that it could be done easily for French but maybe not for other languages. Second problem is that I am not sure we have a lyx2lyx equivalent for layoouts, do we? 2) Accept the layout field as is and do not try to translate them. Given the burden of gettext management, this would clearly be better, I've implemented that already, see attached patch. I agree this is better to the end-user who doesn't care if his styles are not translated. Especially if these styles are not meant to be distributed outside his classroom for example. So? This patch fixes a crash and a regression against 1.4. Unless you want to implement the layout2layout support and do the GUIName addition before 1.5.0, this patch should go in now. It's just a string - docstring conversion and it fixes a number of FIXME UNICODE. It also allows unicode style names for non-english speaking people which I think is very important. Abdel.
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
On Tuesday 10 July 2007 10:23:02 Abdelrazak Younes wrote: So? This patch fixes a crash and a regression against 1.4. Unless you want to implement the layout2layout support and do the GUIName addition before 1.5.0, this patch should go in now. It's just a string - docstring conversion and it fixes a number of FIXME UNICODE. It also allows unicode style names for non-english speaking people which I think is very important. This should also be clearly stated in the RELEASE NOTES, all layout files should be converted to utf8. That is the idea, right? Abdel. -- José Abílio
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
José Matos wrote: On Tuesday 10 July 2007 10:23:02 Abdelrazak Younes wrote: So? This patch fixes a crash and a regression against 1.4. Unless you want to implement the layout2layout support and do the GUIName addition before 1.5.0, this patch should go in now. It's just a string - docstring conversion and it fixes a number of FIXME UNICODE. It also allows unicode style names for non-english speaking people which I think is very important. This should also be clearly stated in the RELEASE NOTES, all layout files should be converted to utf8. That is the idea, right? Right. Abdel.
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
On Tuesday 10 July 2007 10:50:28 Abdelrazak Younes wrote: Right. Does anyone has any objection to this change? The purpose of this release is the support of unicode, we are using utf8 for the saved files so it makes all sense to follow the same route for layout files. In the special cases where layout files are in the ascii subset the no change is need because that is already valid utf-8. Abdel. The plan for this change is then: * to convert our layout files to utf8; * apply your patch; * add a comment to the release notes. I propose to do this tomorrow to give some time for feedback. If by tomorrow there are no voices against we will implement this plan. OK? -- José Abílio
Re: [Patch] allow unicode in layout style name
José == José Matos [EMAIL PROTECTED] writes: José The plan for this change is then: José * to convert our layout files to utf8; * apply your patch; * add José a comment to the release notes. It would be even better to provide explicit commands or some kind of script to update .lyx file. I suspect it is not trivial. JMarc
Re: [Patch] allow unicode in layout style name
On Tuesday 10 July 2007 11:47:36 Jean-Marc Lasgouttes wrote: It would be even better to provide explicit commands or some kind of script to update .lyx file. I suspect it is not trivial. JMarc Using iconv? Something like this for bash cd /path/to/layouts for l in * do cp $l tmp.txt iconv -f latin1 -t utf8 tmp.txt -o $l done -- José Abílio
Re: [Patch] allow unicode in layout style name
José == José Matos [EMAIL PROTECTED] writes: José On Tuesday 10 July 2007 11:47:36 Jean-Marc Lasgouttes wrote: It would be even better to provide explicit commands or some kind of script to update .lyx file. I suspect it is not trivial. JMarc José Using iconv? José Something like this for bash José cd /path/to/layouts for l in * do cp $l tmp.txt iconv -f latin1 José -t utf8 tmp.txt -o $l done You do not want to translate the whole file, only the \begin_layout NameWithAccent lines. JMarc
Re: [Patch] allow unicode in layout style name
On Tuesday 10 July 2007 12:53:56 Jean-Marc Lasgouttes wrote: You do not want to translate the whole file, only the \begin_layout NameWithAccent lines. JMarc Why not? It is easier to use has a file with coherent encoding that one that uses a mixed encoding. -- José Abílio
Re: [Patch] allow unicode in layout style name
José == José Matos [EMAIL PROTECTED] writes: José On Tuesday 10 July 2007 12:53:56 Jean-Marc Lasgouttes wrote: You do not want to translate the whole file, only the \begin_layout NameWithAccent lines. JMarc José Why not? It is easier to use has a file with coherent encoding José that one that uses a mixed encoding. But the rest of the file is converted separately by lyx2lyx. I thought that this translation was more subtle than just running iconv. JMarc
Re: [Patch] allow unicode in layout style name
José Matos wrote: On Tuesday 10 July 2007 12:53:56 Jean-Marc Lasgouttes wrote: You do not want to translate the whole file, only the \begin_layout NameWithAccent lines. JMarc Why not? It is easier to use has a file with coherent encoding that one that uses a mixed encoding. I agree with Jose. I cannot see a reason why mixed encoding would be preferred. Abdel.
Re: [Patch] allow unicode in layout style name
Abdelrazak == Abdelrazak Younes [EMAIL PROTECTED] writes: Why not? It is easier to use has a file with coherent encoding that one that uses a mixed encoding. Abdelrazak I agree with Jose. I cannot see a reason why mixed Abdelrazak encoding would be preferred. I am not sure we are discussing the same thing. Assume one has a file in 1.4 file format with contents in latin1 and some layout names in latin1 too. What do you propose to do? I see two solutions. 1/ first convert the latin1 file to utf8 in one sweep and hope that lyx2lyx will not butcher it when translating to 1.5 file format 2/ first convert to 1.5 file format. Then only the layout names should be converted in a second step. I do not think that iconv would appreciate to see UTF8 characters in a file which is supposed to be latin1. JMarc
Re: [Patch] allow unicode in layout style name
Jean-Marc Lasgouttes wrote: José == José Matos [EMAIL PROTECTED] writes: José On Tuesday 10 July 2007 12:53:56 Jean-Marc Lasgouttes wrote: You do not want to translate the whole file, only the \begin_layout NameWithAccent lines. JMarc José Why not? It is easier to use has a file with coherent encoding José that one that uses a mixed encoding. But the rest of the file is converted separately by lyx2lyx. I thought that this translation was more subtle than just running iconv. If we are to believe Jean-Pierre Chrétien, there's nothing to be done for LyX files: So from the user point of view, nothing else to do with lyx files than loading in 1.5, manual format/unicode conversion of user layouts required. Abdel.
Re: [Patch] allow unicode in layout style name
On Tuesday 10 July 2007 13:15:58 Jean-Marc Lasgouttes wrote: But the rest of the file is converted separately by lyx2lyx. I thought that this translation was more subtle than just running iconv. JMarc OK, now I see what you mean. :-) My problem was with the .layout file and you refered the .lyx file. For .layout files the convertion should be as described. For .lyx files I would expect that the only problem would appear if the paragraph style name uses a different encoding from the lyx file. Are there any such cases? I am not sure if this would have worked before... That is as long as the .lyx and the .layout use the same enconding everything should just work. Or am I missing any case? -- José Abílio
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
José Matos wrote: On Tuesday 10 July 2007 10:50:28 Abdelrazak Younes wrote: Right. Does anyone has any objection to this change? The purpose of this release is the support of unicode, we are using utf8 for the saved files so it makes all sense to follow the same route for layout files. In the special cases where layout files are in the ascii subset the no change is need because that is already valid utf-8. Abdel. The plan for this change is then: * to convert our layout files to utf8; What about the documentation? It is probably time to convert them, isn't it? * apply your patch; * add a comment to the release notes. I propose to do this tomorrow to give some time for feedback. If by tomorrow there are no voices against we will implement this plan. OK? Fine with me. Abel.
Re: [Patch] allow unicode in layout style name
José == José Matos [EMAIL PROTECTED] writes: José That is as long as the .lyx and the .layout use the same José enconding everything should just work. Or am I missing any case? Just do it and you will see if people complain :) JMarc
Re: [Patch] allow unicode in layout style name
Jean-Marc Lasgouttes [EMAIL PROTECTED] writes: José == José Matos [EMAIL PROTECTED] writes: José On Tuesday 10 July 2007 11:47:36 Jean-Marc Lasgouttes wrote: It would be even better to provide explicit commands or some kind of script to update .lyx file. I suspect it is not trivial. JMarc José Using iconv? José Something like this for bash José cd /path/to/layouts for l in * do cp $l tmp.txt iconv -f latin1 José -t utf8 tmp.txt -o $l done You do not want to translate the whole file, only the \begin_layout NameWithAccent lines. But lyx2lyx *does* convert, e.g. \begin_layout Liste_à_puces as \begin_layout Liste à puces (sorry, my Solaris does not display Unicode). lyx-1.5.0rc2 has an assertion if I open the file in the LyX window, but exports perfectly this as itemize if I run the export as batch with lyx-1.5.0rc2 -e latex unicode_test_1.5.lyx So nothing has to be done on the lyx docs side. About layout conversion: layout2layout.py updates the format, iconv turns them in utf8 (needed for non-ascii characters in the style bodies). There are two ways to solve the assertion problem: - forbid non ascii characters in styles names and use gettext to translate: needs manual renaming of style names with non-ascii characters (bugfree with lyx up to 1.4) to ascii names and update of gmo files); - allow non ascii characters: Abdel's patch should solve the assertion problem. Do you need an example ? -- Jean-Pierre
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
On Tuesday 10 July 2007 13:25:04 Abdelrazak Younes wrote: What about the documentation? It is probably time to convert them, isn't it? As far as I remember there are no patches holding that change the file format. If so then yes it is time to start updating the documentation to the latest format. Abel. -- José Abílio
Re: [Patch] allow unicode in layout style name
José == José Matos [EMAIL PROTECTED] writes: José On Tuesday 10 July 2007 13:25:04 Abdelrazak Younes wrote: What about the documentation? It is probably time to convert them, isn't it? José As far as I remember there are no patches holding that change José the file format. If so then yes it is time to start updating the José documentation to the latest format. So you can also update lyx2lyx in 1.4.5svn, right? I will be away starting on next Friday, so I'd like to release on thursday (or tomorrow?). JMarc
Re: [Patch] allow unicode in layout style name
Jean-Marc Lasgouttes wrote: So you can also update lyx2lyx in 1.4.5svn, right? I will be away starting on next Friday, so I'd like to release on thursday (or tomorrow?). Shouldn't the remaining major issues with lyx2lyx be addressed first (bug 3313, 3404, 3958, 3976, 3985)? Jürgen
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
On Tue, 10 Jul 2007, José Matos wrote: The plan for this change is then: * to convert our layout files to utf8; * apply your patch; * add a comment to the release notes. I propose to do this tomorrow to give some time for feedback. If by tomorrow there are no voices against we will implement this plan. OK? What's the risk of embarrassing[*] bugs? (As this is just before the release of 1.5.0). There is no RC3 planned, right? /Christian [*] I think the bugs will be easy to fix, it may just be somewhat embarrasing to do a 1.5.1 release very quickly after 1.5.0. -- Christian Ridderström, +46-8-768 39 44 http://www.md.kth.se/~chr
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
On Tuesday 10 July 2007 15:16:56 [EMAIL PROTECTED] wrote: What's the risk of embarrassing[*] bugs? (As this is just before the release of 1.5.0). There is no RC3 planned, right? Do you want a formula with deltas and epsilons? ;-) No there is not any RC3 planned. Do you think we need one? Regarding this subject I think that this course of action is sound so I don't expect any major surprises. (Famous last words). /Christian [*] I think the bugs will be easy to fix, it may just be somewhat embarrasing to do a 1.5.1 release very quickly after 1.5.0. That is always a possibility. :-) -- José Abílio
Re: [Patch] allow unicode in layout style name
José Matos wrote: I will look into them. Thanks! Seems you are the only one available now with the necessary skills. Jürgen
Re: [Patch] allow unicode in layout style name
José Matos wrote: On Tuesday 10 July 2007 13:15:58 Jean-Marc Lasgouttes wrote: But the rest of the file is converted separately by lyx2lyx. I thought that this translation was more subtle than just running iconv. JMarc OK, now I see what you mean. :-) My problem was with the .layout file and you refered the .lyx file. For .layout files the convertion should be as described. For .lyx files I would expect that the only problem would appear if the paragraph style name uses a different encoding from the lyx file. Are there any such cases? I am not sure if this would have worked before... That is as long as the .lyx and the .layout use the same enconding everything should just work. Or am I missing any case? FYI Jean-Pierre provided me with a layout file converted to utf8 and a 1.4 LyX file using this layout. With my patch, I had no problem opening the LyX file and the conversion to UTF-8 of the style names was done automatically. Abdel.
Re: [Patch] allow unicode in layout style name
Abdelrazak == Abdelrazak Younes [EMAIL PROTECTED] writes: Abdelrazak FYI Jean-Pierre provided me with a layout file converted Abdelrazak to utf8 and a 1.4 LyX file using this layout. With my Abdelrazak patch, I had no problem opening the LyX file and the Abdelrazak conversion to UTF-8 of the style names was done Abdelrazak automatically. Fine. JMarc
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
Abdelrazak Younes wrote: Jean-Pierre Chrétien wrote: Abdelrazak Younes <[EMAIL PROTECTED]> writes: 1) provide a python script that converts the layout as good as possible: Théorème -> Theoreme Liste_à_puce -> Liste_a_puce The problem is that it could be done easily for French but maybe not for other languages. Second problem is that I am not sure we have a lyx2lyx equivalent for layoouts, do we? 2) Accept the layout field as is and do not try to translate them. Given the burden of gettext management, this would clearly be better, I've implemented that already, see attached patch. I agree this is better to the end-user who doesn't care if his styles are not translated. Especially if these styles are not meant to be distributed outside his classroom for example. So? This patch fixes a crash and a regression against 1.4. Unless you want to implement the layout2layout support and do the GUIName addition before 1.5.0, this patch should go in now. It's just a string -> docstring conversion and it fixes a number of "FIXME UNICODE". It also allows unicode style names for non-english speaking people which I think is very important. Abdel.
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
On Tuesday 10 July 2007 10:23:02 Abdelrazak Younes wrote: > So? > > This patch fixes a crash and a regression against 1.4. Unless you want > to implement the layout2layout support and do the GUIName addition > before 1.5.0, this patch should go in now. > > It's just a string -> docstring conversion and it fixes a number of > "FIXME UNICODE". It also allows unicode style names for non-english > speaking people which I think is very important. This should also be clearly stated in the RELEASE NOTES, all layout files should be converted to utf8. That is the idea, right? > Abdel. -- José Abílio
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
José Matos wrote: On Tuesday 10 July 2007 10:23:02 Abdelrazak Younes wrote: So? This patch fixes a crash and a regression against 1.4. Unless you want to implement the layout2layout support and do the GUIName addition before 1.5.0, this patch should go in now. It's just a string -> docstring conversion and it fixes a number of "FIXME UNICODE". It also allows unicode style names for non-english speaking people which I think is very important. This should also be clearly stated in the RELEASE NOTES, all layout files should be converted to utf8. That is the idea, right? Right. Abdel.
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
On Tuesday 10 July 2007 10:50:28 Abdelrazak Younes wrote: > Right. Does anyone has any objection to this change? The purpose of this release is the support of unicode, we are using utf8 for the saved files so it makes all sense to follow the same route for layout files. In the special cases where layout files are in the ascii subset the no change is need because that is already valid utf-8. > Abdel. The plan for this change is then: * to convert our layout files to utf8; * apply your patch; * add a comment to the release notes. I propose to do this tomorrow to give some time for feedback. If by tomorrow there are no voices against we will implement this plan. OK? -- José Abílio
Re: [Patch] allow unicode in layout style name
> "José" == José Matos <[EMAIL PROTECTED]> writes: José> The plan for this change is then: José> * to convert our layout files to utf8; * apply your patch; * add José> a comment to the release notes. It would be even better to provide explicit commands or some kind of script to update .lyx file. I suspect it is not trivial. JMarc
Re: [Patch] allow unicode in layout style name
On Tuesday 10 July 2007 11:47:36 Jean-Marc Lasgouttes wrote: > It would be even better to provide explicit commands or some kind of > script to update .lyx file. I suspect it is not trivial. > > JMarc Using iconv? Something like this for bash cd /path/to/layouts for l in * do cp $l tmp.txt iconv -f latin1 -t utf8 tmp.txt -o $l done -- José Abílio
Re: [Patch] allow unicode in layout style name
> "José" == José Matos <[EMAIL PROTECTED]> writes: José> On Tuesday 10 July 2007 11:47:36 Jean-Marc Lasgouttes wrote: >> It would be even better to provide explicit commands or some kind >> of script to update .lyx file. I suspect it is not trivial. >> >> JMarc José> Using iconv? José> Something like this for bash José> cd /path/to/layouts for l in * do cp $l tmp.txt iconv -f latin1 José> -t utf8 tmp.txt -o $l done You do not want to translate the whole file, only the \begin_layout NameWithAccent lines. JMarc
Re: [Patch] allow unicode in layout style name
On Tuesday 10 July 2007 12:53:56 Jean-Marc Lasgouttes wrote: > You do not want to translate the whole file, only the > \begin_layout NameWithAccent > lines. > > JMarc Why not? It is easier to use has a file with coherent encoding that one that uses a mixed encoding. -- José Abílio
Re: [Patch] allow unicode in layout style name
> "José" == José Matos <[EMAIL PROTECTED]> writes: José> On Tuesday 10 July 2007 12:53:56 Jean-Marc Lasgouttes wrote: >> You do not want to translate the whole file, only the \begin_layout >> NameWithAccent lines. >> >> JMarc José> Why not? It is easier to use has a file with coherent encoding José> that one that uses a mixed encoding. But the rest of the file is converted separately by lyx2lyx. I thought that this translation was more subtle than just running iconv. JMarc
Re: [Patch] allow unicode in layout style name
José Matos wrote: On Tuesday 10 July 2007 12:53:56 Jean-Marc Lasgouttes wrote: You do not want to translate the whole file, only the \begin_layout NameWithAccent lines. JMarc Why not? It is easier to use has a file with coherent encoding that one that uses a mixed encoding. I agree with Jose. I cannot see a reason why mixed encoding would be preferred. Abdel.
Re: [Patch] allow unicode in layout style name
> "Abdelrazak" == Abdelrazak Younes <[EMAIL PROTECTED]> writes: >> Why not? It is easier to use has a file with coherent encoding that >> one that uses a mixed encoding. Abdelrazak> I agree with Jose. I cannot see a reason why mixed Abdelrazak> encoding would be preferred. I am not sure we are discussing the same thing. Assume one has a file in 1.4 file format with contents in latin1 and some layout names in latin1 too. What do you propose to do? I see two solutions. 1/ first convert the latin1 file to utf8 in one sweep and hope that lyx2lyx will not butcher it when translating to 1.5 file format 2/ first convert to 1.5 file format. Then only the layout names should be converted in a second step. I do not think that iconv would appreciate to see UTF8 characters in a file which is supposed to be latin1. JMarc
Re: [Patch] allow unicode in layout style name
Jean-Marc Lasgouttes wrote: "José" == José Matos <[EMAIL PROTECTED]> writes: José> On Tuesday 10 July 2007 12:53:56 Jean-Marc Lasgouttes wrote: You do not want to translate the whole file, only the \begin_layout NameWithAccent lines. JMarc José> Why not? It is easier to use has a file with coherent encoding José> that one that uses a mixed encoding. But the rest of the file is converted separately by lyx2lyx. I thought that this translation was more subtle than just running iconv. If we are to believe Jean-Pierre Chrétien, there's nothing to be done for LyX files: "So from the user point of view, nothing else to do with lyx files than loading in 1.5, manual format/unicode conversion of user layouts required." Abdel.
Re: [Patch] allow unicode in layout style name
On Tuesday 10 July 2007 13:15:58 Jean-Marc Lasgouttes wrote: > But the rest of the file is converted separately by lyx2lyx. I thought > that this translation was more subtle than just running iconv. > > JMarc OK, now I see what you mean. :-) My problem was with the .layout file and you refered the .lyx file. For .layout files the convertion should be as described. For .lyx files I would expect that the only problem would appear if the paragraph style name uses a different encoding from the lyx file. Are there any such cases? I am not sure if this would have worked before... That is as long as the .lyx and the .layout use the same enconding everything should just work. Or am I missing any case? -- José Abílio
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
José Matos wrote: On Tuesday 10 July 2007 10:50:28 Abdelrazak Younes wrote: Right. Does anyone has any objection to this change? The purpose of this release is the support of unicode, we are using utf8 for the saved files so it makes all sense to follow the same route for layout files. In the special cases where layout files are in the ascii subset the no change is need because that is already valid utf-8. Abdel. The plan for this change is then: * to convert our layout files to utf8; What about the documentation? It is probably time to convert them, isn't it? * apply your patch; * add a comment to the release notes. I propose to do this tomorrow to give some time for feedback. If by tomorrow there are no voices against we will implement this plan. OK? Fine with me. Abel.
Re: [Patch] allow unicode in layout style name
> "José" == José Matos <[EMAIL PROTECTED]> writes: José> That is as long as the .lyx and the .layout use the same José> enconding everything should just work. Or am I missing any case? Just do it and you will see if people complain :) JMarc
Re: [Patch] allow unicode in layout style name
Jean-Marc Lasgouttes <[EMAIL PROTECTED]> writes: > > > "José" == José Matos <[EMAIL PROTECTED]> writes: > > José> On Tuesday 10 July 2007 11:47:36 Jean-Marc Lasgouttes wrote: > >> It would be even better to provide explicit commands or some kind > >> of script to update .lyx file. I suspect it is not trivial. > >> > >> JMarc > > José> Using iconv? > > José> Something like this for bash > > José> cd /path/to/layouts for l in * do cp $l tmp.txt iconv -f latin1 > José> -t utf8 tmp.txt -o $l done > > You do not want to translate the whole file, only the > \begin_layout NameWithAccent > lines. But lyx2lyx *does* convert, e.g. \begin_layout Liste_à_puces as \begin_layout Liste à puces (sorry, my Solaris does not display Unicode). lyx-1.5.0rc2 has an assertion if I open the file in the LyX window, but exports perfectly this as itemize if I run the export as batch with lyx-1.5.0rc2 -e latex unicode_test_1.5.lyx So nothing has to be done on the lyx docs side. About layout conversion: layout2layout.py updates the format, iconv turns them in utf8 (needed for non-ascii characters in the style bodies). There are two ways to solve the assertion problem: - forbid non ascii characters in styles names and use gettext to translate: needs manual renaming of style names with non-ascii characters (bugfree with lyx up to 1.4) to ascii names and update of gmo files); - allow non ascii characters: Abdel's patch should solve the assertion problem. Do you need an example ? -- Jean-Pierre
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
On Tuesday 10 July 2007 13:25:04 Abdelrazak Younes wrote: > What about the documentation? It is probably time to convert them, isn't > it? As far as I remember there are no patches holding that change the file format. If so then yes it is time to start updating the documentation to the latest format. > > Abel. -- José Abílio
Re: [Patch] allow unicode in layout style name
> "José" == José Matos <[EMAIL PROTECTED]> writes: José> On Tuesday 10 July 2007 13:25:04 Abdelrazak Younes wrote: >> What about the documentation? It is probably time to convert them, >> isn't it? José> As far as I remember there are no patches holding that change José> the file format. If so then yes it is time to start updating the José> documentation to the latest format. So you can also update lyx2lyx in 1.4.5svn, right? I will be away starting on next Friday, so I'd like to release on thursday (or tomorrow?). JMarc
Re: [Patch] allow unicode in layout style name
Jean-Marc Lasgouttes wrote: > So you can also update lyx2lyx in 1.4.5svn, right? I will be away > starting on next Friday, so I'd like to release on thursday (or > tomorrow?). Shouldn't the remaining major issues with lyx2lyx be addressed first (bug 3313, 3404, 3958, 3976, 3985)? Jürgen
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
On Tue, 10 Jul 2007, José Matos wrote: The plan for this change is then: * to convert our layout files to utf8; * apply your patch; * add a comment to the release notes. I propose to do this tomorrow to give some time for feedback. If by tomorrow there are no voices against we will implement this plan. OK? What's the risk of embarrassing[*] bugs? (As this is just before the release of 1.5.0). There is no RC3 planned, right? /Christian [*] I think the bugs will be easy to fix, it may just be somewhat embarrasing to do a 1.5.1 release very quickly after 1.5.0. -- Christian Ridderström, +46-8-768 39 44 http://www.md.kth.se/~chr
Re: [Patch] allow unicode in layout style name (was Re: Upgrade from 1.4 to 1.5)
On Tuesday 10 July 2007 15:16:56 [EMAIL PROTECTED] wrote: > > What's the risk of embarrassing[*] bugs? (As this is just before the > release of 1.5.0). There is no RC3 planned, right? Do you want a formula with deltas and epsilons? ;-) No there is not any RC3 planned. Do you think we need one? Regarding this subject I think that this course of action is sound so I don't expect any major surprises. (Famous last words). > /Christian > > [*] I think the bugs will be easy to fix, it may just be somewhat > embarrasing to do a 1.5.1 release very quickly after 1.5.0. That is always a possibility. :-) -- José Abílio
Re: [Patch] allow unicode in layout style name
José Matos wrote: > I will look into them. Thanks! Seems you are the only one available now with the necessary skills. Jürgen
Re: [Patch] allow unicode in layout style name
José Matos wrote: On Tuesday 10 July 2007 13:15:58 Jean-Marc Lasgouttes wrote: But the rest of the file is converted separately by lyx2lyx. I thought that this translation was more subtle than just running iconv. JMarc OK, now I see what you mean. :-) My problem was with the .layout file and you refered the .lyx file. For .layout files the convertion should be as described. For .lyx files I would expect that the only problem would appear if the paragraph style name uses a different encoding from the lyx file. Are there any such cases? I am not sure if this would have worked before... That is as long as the .lyx and the .layout use the same enconding everything should just work. Or am I missing any case? FYI Jean-Pierre provided me with a layout file converted to utf8 and a 1.4 LyX file using this layout. With my patch, I had no problem opening the LyX file and the conversion to UTF-8 of the style names was done automatically. Abdel.
Re: [Patch] allow unicode in layout style name
> "Abdelrazak" == Abdelrazak Younes <[EMAIL PROTECTED]> writes: Abdelrazak> FYI Jean-Pierre provided me with a layout file converted Abdelrazak> to utf8 and a 1.4 LyX file using this layout. With my Abdelrazak> patch, I had no problem opening the LyX file and the Abdelrazak> conversion to UTF-8 of the style names was done Abdelrazak> automatically. Fine. JMarc
