Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
Attached is a better patch where everything is implemented: - correct encoding for the fontenc package. Mostafa tested this and gave his OK for this issue. - special quotation mark definitions for Farsi. Mostafa tested this and gave his OK for this issue. - fix for bug 2929 - use \textAR and not \R. Improved version of Dov's patch. Mostafa also tested this successfully. - use the document language also for the TOC. This is a special arabi-package thing. Please test. - To make the TOC settings work and also for other issues, arabic must be loaded to babel too when farsi is the document language. Please test. Mostafa, when you give your OK I'll commit it. When there are possible problems with the two new things, I'll only commit the uncritical and tested issues from above. (With this patch now also your example_lyxified.lyx you sent two days ago to be added to LyX's examples compiles for me.) Dov, OK by you? I think we have now a clean setup for boths languages based on the arabi package. José, can this go in? I think it's worth because without this patch Arabic is not usable and for Farsi only with preamble hacks. regards Uwe Index: lib/languages === --- lib/languages (revision 18849) +++ lib/languages (working copy) @@ -25,7 +25,8 @@ # Esperanto has no country code esperanto esperanto "Esperanto" false iso8859-3 eo "" estonianestonian "Estonian" false iso8859-15 et_EE "" -farsi farsi "Farsi" true utf8 fa_IR "" +# the definitions are only used due to bugs in the arabi-package - remove them if they become unnecessary! +farsi farsi "Farsi" true utf8 fa_IR "[EMAIL PROTECTED]@R{#1}}}\DeclareTextSymbol{\guillemotright}{LFE}{62}\DeclareTextSymbol{\guillemotleft}{LFE}{60}" finnish finnish "Finnish" false iso8859-15 fi_FI "" # We redefine \og and \fg (guillemets) for older french language definitions french french "French" false iso8859-15 fr_FR "\addto\extrasfrench{\providecommand{\og}{\leavevmode\flqq~}\providecommand{\fg}{\ifdim\lastskip>[EMAIL PROTECTED]" Index: src/BufferParams.cpp === --- src/BufferParams.cpp (revision 18849) +++ src/BufferParams.cpp (working copy) @@ -855,6 +855,13 @@ bool const use_babel = features.useBabel(); if (use_babel) { language_options << features.getLanguages(); + // arabic must also be loaded when farsi is the document language + // this is hopefully not needed with future versions of the arabi-package + if (language->lang() == "farsi") { + if (!features.getLanguages().empty()) +language_options << ','; + language_options << "arabic"; + } if (!language->babel().empty()) { if (!language_options.str().empty()) language_options << ','; @@ -892,9 +899,17 @@ if (fontsDefaultFamily != "default") os << "\\renewcommand{\\familydefault}{\\" << from_ascii(fontsDefaultFamily) << "}\n"; + + // set font encoding // this one is not per buffer - if (lyxrc.fontenc != "default") { + // for Arabic and Farsi we also need to load the LAE and LFE encoding + if ((lyxrc.fontenc != "default" && language->lang() == "arabic") || + (lyxrc.fontenc != "default" && language->lang() == "farsi")) { os << "\\usepackage[" << from_ascii(lyxrc.fontenc) + << ",LFE,LAE]{fontenc}\n"; + texrow.newline(); + } else { + os << "\\usepackage[" << from_ascii(lyxrc.fontenc) << "]{fontenc}\n"; texrow.newline(); } @@ -1144,8 +1159,16 @@ lyxpreamble += from_utf8(features.getBabelOptions()); } - lyxpreamble += "\\makeatother\n"; + // these definitions are needed by the arabi-package + // when the document language is Arabic or Farsi, the language is set + // as the language for the TOC + if (language->babel() == "arabic") + lyxpreamble += "\\TOCLanguage{arabic}\n"; + if (language->babel() == "farsi") + lyxpreamble += "\\TOCLanguage{farsi}\n"; + lyxpreamble += "\\makeatother\n\n"; + int const nlines = int(lyx::count(lyxpreamble.begin(), lyxpreamble.end(), '\n')); for (int j = 0; j != nlines; ++j) { Index: src/Font.cpp === --- src/Font.cpp (revision 18849) +++ src/Font.cpp (working copy) @@ -756,6 +756,13 @@ base.language()->lang() == "farsi") { os << "\\textLR{"; count += 8; + } else if (language()->lang() == "arabic") { + os << "\\textAR{"; + count += 8; + } else if (!isRightToLeft() && + base.language()->lang() == "arabic") { + os << "\\textLR{"; + count += 8; } else if (isRightToLeft() != prev.isRightToLeft()) { if (isRightToLeft()) { os << "\\R{"; Index: src/insets/InsetTabular.cpp === --- src/insets/InsetTabular.cpp (revision 18849) +++ src/insets/InsetTabular.cpp (working copy) @@ -2286,6 +2286,8 @@ if (par.getParLanguage(buf.params())->lang() == "farsi")
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
I will test this now. But even if it all works, this is *not* OK as it is --- it breaks ArabTeX Arabic. Please see the email I sent earlier tonight for a patch which separates between arabi and arabtex. This patch should be implemented over the arabic_arabi language. I'll get back to you regarding how this works. Dov Uwe Stöhr wrote: Attached is a better patch where everything is implemented: - correct encoding for the fontenc package. Mostafa tested this and gave his OK for this issue. - special quotation mark definitions for Farsi. Mostafa tested this and gave his OK for this issue. - fix for bug 2929 - use \textAR and not \R. Improved version of Dov's patch. Mostafa also tested this successfully. - use the document language also for the TOC. This is a special arabi-package thing. Please test. - To make the TOC settings work and also for other issues, arabic must be loaded to babel too when farsi is the document language. Please test. Mostafa, when you give your OK I'll commit it. When there are possible problems with the two new things, I'll only commit the uncritical and tested issues from above. (With this patch now also your example_lyxified.lyx you sent two days ago to be added to LyX's examples compiles for me.) Dov, OK by you? I think we have now a clean setup for boths languages based on the arabi package. José, can this go in? I think it's worth because without this patch Arabic is not usable and for Farsi only with preamble hacks. regards Uwe
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
Dov Feldstern schrieb: I will test this now. But even if it all works, this is *not* OK as it is --- it breaks ArabTeX Arabic. I don't understand. Currently ArabTeX is never loaded by LyX for Arabic documents so it is also not used. My patch requires arabi for both languages. To repeat myself: ArabTeX without arabi can't be used due to the missing babel interface and, more important, due to the missing input encodings. The latter was bug 2927 which is now fixed for a while but since then the arabi-package is required for Arabic. But once arabi is installed on your system, it will be used automatically, so ArabTeX is already not an option. Even if you load it in the preamble it will only be used for Arabic-only documents. As soon you have another language in your document arabi will be used again due to its babel interface. So why not using my solution? Note that I created it by the requests of Mostafa. Please see the email I sent earlier tonight for a patch which separates between arabi and arabtex. This patch should be implemented over the arabic_arabi language. I haven't received an email from you yet. regards Uwe
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
Uwe Stöhr wrote: Dov Feldstern schrieb: I will test this now. But even if it all works, this is *not* OK as it is --- it breaks ArabTeX Arabic. I don't understand. Currently ArabTeX is never loaded by LyX for Arabic documents so it is also not used. My patch requires arabi for both languages. In the email I sent a few days ago, I explained exactly how to get LyX working in Arabic, with either ArabTeX or Arabi. As explained there, it is possible to perform a small one-time setup of LyX, and then use ArabTeX for Arabic in LyX. So LyX has been supporting Arabic to a certain extent for quite a while, using ArabTeX. The changes you are proposing now break that support. To repeat myself: ArabTeX without arabi can't be used due to the missing babel interface and ArabTeX is not babel-based, and therefore doesn't require a babel interface. , more important, due to the missing input encodings. ArabTeX can use many different encodings, including a transliteration-type encoding which uses plain ASCII, I think. It does not include cp1256 by default, AFAICT. However, (a) it does include other encodings (iso-8859-8, for example), which LyX did support until recently, I'm not sure if it still does; (b) as you have pointed out, one can use ArabTeX with the cp1256 encoding provided by arabi. Why would someone want to? Because arabi is not yet perfect; the fonts for ArabTeX are nicer; because that's the way it used to be done. The point is, we used to support it, and we shouldn't just drop that support if it's not absolutely necessary. The latter was bug 2927 which is now fixed for a while but since then the arabi-package is required for Arabic. But once arabi is installed on your system, it will be used automatically, so ArabTeX is already not an option. Even if you load it in the preamble it will only be used for Arabic-only documents. As soon you have another language in your document arabi will be used again due to its babel interface. So why not using my solution? Note that I created it by the requests of Mostafa. Because it breaks ArabTeX, that's all. The patch I sent in earlier tonight (see below) separates between ArabTeX and Arabi, so that on top of it, it should be possible to implement your patch without breaking ArabTeX. Then your solution can be used. Please see the email I sent earlier tonight for a patch which separates between arabi and arabtex. This patch should be implemented over the arabic_arabi language. I haven't received an email from you yet. Yes, there seems to be a problem with email to the newsgroup --- look at the times of the emails I (and others) sent to the newsgroup tonight, vs. their arrival times. If it doesn't arrive by tomorrow, I'll resend it... regards Uwe
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
Uwe Stöhr schrieb: I don't understand. Currently ArabTeX is never loaded by LyX for Arabic documents so it is also not used. Bug 2928 is the prove for my statement. Bug 2928 occurs because the \R command is not defined. But when ArabTeX was loaded \R would have been defined. When you load ArabTeX with the line \usepackage{arabtex} in the document preamble, bug 2928 doesn't occur. So my patch can't break arabTeX as it is already not used. Correct me when I'm wrong. regards Uwe
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
Uwe Stöhr wrote: Uwe Stöhr schrieb: I don't understand. Currently ArabTeX is never loaded by LyX for Arabic documents so it is also not used. Bug 2928 is the prove for my statement. Bug 2928 occurs because the \R command is not defined. But when ArabTeX was loaded \R would have been defined. When you load ArabTeX with the line \usepackage{arabtex} in the document preamble, bug 2928 doesn't occur. So my patch can't break arabTeX as it is already not used. Correct me when I'm wrong. see http://permalink.gmane.org/gmane.editors.lyx.devel/88100. If you follow the instructions there, ArabTeX works perfectly alright in LyX. regards Uwe Dov
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
Dov Feldstern schrieb: So my patch can't break arabTeX as it is already not used. Correct me when I'm wrong. see http://permalink.gmane.org/gmane.editors.lyx.devel/88100. If you follow the instructions there, ArabTeX works perfectly alright in LyX. You procedure requires a lot of manual changes, and it also proves that arabTeX is currently not used directly by LyX. The goal is that the uses only has to set the document language to Arabic in LyX's document settings menu and can start. With my patch and arabi this is possible. regards Uwe
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
I'm resending this message, as only about half my emails seem to be getting through tonight. My apologies if you're receiving it twice... Uwe Stöhr wrote: Dov Feldstern schrieb: I will test this now. But even if it all works, this is *not* OK as it is --- it breaks ArabTeX Arabic. I don't understand. Currently ArabTeX is never loaded by LyX for Arabic documents so it is also not used. My patch requires arabi for both languages. In the email I sent a few days ago, I explained exactly how to get LyX working in Arabic, with either ArabTeX or Arabi. As explained there, it is possible to perform a small one-time setup of LyX, and then use ArabTeX for Arabic in LyX. So LyX has been supporting Arabic to a certain extent for quite a while, using ArabTeX. The changes you are proposing now break that support. To repeat myself: ArabTeX without arabi can't be used due to the missing babel interface and ArabTeX is not babel-based, and therefore doesn't require a babel interface. , more important, due to the missing input encodings. ArabTeX can use many different encodings, including a transliteration-type encoding which uses plain ASCII, I think. It does not include cp1256 by default, AFAICT. However, (a) it does include other encodings (iso-8859-8, for example), which LyX did support until recently, I'm not sure if it still does; (b) as you have pointed out, one can use ArabTeX with the cp1256 encoding provided by arabi. Why would someone want to? Because arabi is not yet perfect; the fonts for ArabTeX are nicer; because that's the way it used to be done. The point is, we used to support it, and we shouldn't just drop that support if it's not absolutely necessary. The latter was bug 2927 which is now fixed for a while but since then the arabi-package is required for Arabic. But once arabi is installed on your system, it will be used automatically, so ArabTeX is already not an option. Even if you load it in the preamble it will only be used for Arabic-only documents. As soon you have another language in your document arabi will be used again due to its babel interface. So why not using my solution? Note that I created it by the requests of Mostafa. Because it breaks ArabTeX, that's all. The patch I sent in earlier tonight (see below) separates between ArabTeX and Arabi, so that on top of it, it should be possible to implement your patch without breaking ArabTeX. Then your solution can be used. Please see the email I sent earlier tonight for a patch which separates between arabi and arabtex. This patch should be implemented over the arabic_arabi language. I haven't received an email from you yet. Yes, there seems to be a problem with email to the newsgroup --- look at the times of the emails I (and others) sent to the newsgroup tonight, vs. their arrival times. If it doesn't arrive by tomorrow, I'll resend it... regards Uwe
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
Uwe Stöhr wrote: Dov Feldstern schrieb: So my patch can't break arabTeX as it is already not used. Correct me when I'm wrong. see http://permalink.gmane.org/gmane.editors.lyx.devel/88100. If you follow the instructions there, ArabTeX works perfectly alright in LyX. You procedure requires a lot of manual changes, and it also proves that arabTeX is currently not used directly by LyX. Yes, that is correct. For many many years this was how LyX supported Hebrew as well as Arabic. AFAIK, they still don't work "out of the box", and there's a certain amount of setup required by the user. This does *not* prove that there aren't Arabic users out there using ArabTeX in LyX. And those users, if they exist, will not be happy if we break it. The goal is that the uses only has to set the document language to Arabic in LyX's document settings menu and can start. With my patch and arabi this is possible. I applaud that goal, but not at the expense of breaking backwards compatibility. regards Uwe
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
Dov Feldstern schrieb: To repeat myself: ArabTeX without arabi can't be used due to the missing babel interface and ArabTeX is not babel-based, and therefore doesn't require a babel interface. But LyX currenly uses babel to handle the different languages. Therefore arabi will be used when LyX inserts \selectlanguage{arabic}. I don't understand why you don't want to have ONE package for Arabic AND Farsi. This fixes many problems and keeps it maintainable. Having support for two packages side by side requires much more maintaining work and expect it will introduce new problems we are currently not aware of. , more important, due to the missing input encodings. ArabTeX can use many different encodings, including a transliteration-type encoding which uses plain ASCII, I think. It does not include cp1256 by default, AFAICT. However, (a) it does include other encodings (iso-8859-8, for example), which LyX did support until recently, I'm not sure if it still does; Yes but not cp1256 which is required on Windows. (b) as you have pointed out, one can use ArabTeX with the cp1256 encoding provided by arabi. Yes, but then you use automatically arabi not arabTeX, see my explanation above. arabi is not yet perfect; the fonts for ArabTeX are nicer; because that's the way it used to be done. Every pacakge has its disadvantages. I mean we can discuss this again and again without changing this. But we have to decide what package to use arabTeX OR arabi - both is not an option. The point is, we used to support it, and we shouldn't just drop that support if it's not absolutely necessary. We never supported this since the user has to some manual adjustments before it works. When LyX would automatically load arabTeX when the user uses Arabic as document language then you can say we support it. regards Uwe
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
Uwe Stöhr wrote: Dov Feldstern schrieb: To repeat myself: ArabTeX without arabi can't be used due to the missing babel interface and ArabTeX is not babel-based, and therefore doesn't require a babel interface. But LyX currenly uses babel to handle the different languages. Therefore arabi will be used when LyX inserts \selectlanguage{arabic}. I don't understand why you don't want to have ONE package for Arabic AND Farsi. This fixes many problems and keeps it maintainable. Having support for two packages side by side requires much more maintaining work and expect it will introduce new problems we are currently not aware of. I would like to have one. But right now we have two packages for Arabic, neither one of which is perfect; and one of them has been supported for a long time, so suddenly switching to the other and breaking compatibility with the traditionally supported one doesn't seem right. Certainly I see advantages to using Arabi. For example, I don't think it will be possible to mix Hebrew and Arabic using ArabTeX, I think it *will* be possible with Arabi. But until Arabi is the clear winner --- and currently it's not, it still has many problems --- I don't think we should drop ArabTeX. , more important, due to the missing input encodings. ArabTeX can use many different encodings, including a transliteration-type encoding which uses plain ASCII, I think. It does not include cp1256 by default, AFAICT. However, (a) it does include other encodings (iso-8859-8, for example), which LyX did support until recently, I'm not sure if it still does; Yes but not cp1256 which is required on Windows. (b) as you have pointed out, one can use ArabTeX with the cp1256 encoding provided by arabi. Yes, but then you use automatically arabi not arabTeX, see my explanation above. arabi is not yet perfect; the fonts for ArabTeX are nicer; because that's the way it used to be done. Every pacakge has its disadvantages. I mean we can discuss this again and again without changing this. But we have to decide what package to use arabTeX OR arabi - both is not an option. Both *is* an only an option, I think it's a *must*. I have sent a patch which does that, I resent it about three times already, but it doesn't seem to be getting through... The point is, we used to support it, and we shouldn't just drop that support if it's not absolutely necessary. We never supported this since the user has to some manual adjustments before it works. When LyX would automatically load arabTeX when the user uses Arabic as document language then you can say we support it. I don't understand that approach. By that measure, we don't support Hebrew, either, because I still have to setup the keymap myself; because there's a whole Wiki page dedicated to how to get Hebrew working in LyX, which proves that it doesn't exactly work out of the box --- and that's only the Windows instructions, for linux it a bit more complicated; because until about a month ago, RTL was turned off by default, so the user had to turn it on manually. But the fact is that I've been using LyX to write complex Hebrew documents for the past five years, and I would be very upset if suddenly all that was broken "because we didn't really support it up until now". regards Uwe
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
Dov Feldstern schrieb: and one of them has been supported for a long time, so suddenly switching to the other and breaking compatibility with the traditionally supported one doesn't seem right. As I said we never supported arabTeX directly. So we don't switch to something. Mostafa, what is your opinion here? But anyway, to your patch: - better use arabic_arabtex and arabic_arabi. If this is a fileformat change, fine by me. - your patch is incomplete, you miss the fontencoding issue. What font encoding is required by arabTeX? I can add this to your patch when we made a decision what to support. We never supported this since the user has to some manual adjustments before it works. When LyX would automatically load arabTeX when the user uses Arabic as document language then you can say we support it. I don't understand that approach. By that measure, we don't support Hebrew, either, because I still have to setup the keymap myself; Why that? We support now unicode so you should be able to type directly without using a keymap. because there's a whole Wiki page dedicated to how to get Hebrew working in LyX, which proves that it doesn't exactly work out of the box Hmm, you can drop the binding file issue there and when you have a Hebrew keyboard also the keyboard map section. For me it only took 5 minutes to set up Hebrew: Installing the culsmus.exe that brings the Hebrew fonts to LaTeX and that's it. (Btw. it was me who created this wiki page originally.) Of course you have for every language first set up LaTeX by installing packages, but with support I mean that you don't have to use preamble hacks or fiddling with the preferences to make things work when you change the document language. With your recipe in http://permalink.gmane.org/gmane.editors.lyx.devel/88100 I cannot write an Arabic and immediately later an English document. Docments with mixed languages are also not really possible. arabTeX simply don't work correctly for me. I tested this out hardly half a year ago and gave up, but now with arabi, everything concernig the infrastructure works: - no special other package to install to get the input encoding - I can mix languages as much as I want without editing the preamble or ERT - I can type Arabic and Englich documents without modifying the preferences when I switch - when you have an Arabic keyboard, you don't need a keyboard map file - babel support: no need to translate the names manually in the preamble for every document All this is also possible with Hebrew, so this is what I call support. But all this is not possible with arabTeX. Sure also arabi has it's drawbacks but as the infrastructure is ready with it, we can build in small workarounds like the ones for Farsi in my patch. --- Besides this I would commit at least the Farsi related stuff of my patch as this improves typesetting Farsi enormously. regards Uwe
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
Dear Dov and Uwe, ARABI is a good package for Farsi that supports BABEL. But it has some (even many) limitations. About Arabic I believe that I am not the person who should decides at this moment. I need more time for that one. I would like to keep the possibility of using ArabTeX (NOT necessarily SUPPORTING) with LyX. Although ArabTeX is not based on BABEL but I think it can be used beside the BABEL. What I suggest is not to hurry for using ARABI for Arabic. Please give me some time so that I can figure out what is the best way to deal with ArabTeX and ARABI. It is just a matter of time. Please keep the current status of Arabic as it is. Maybe it is not possible (currently) to do all things automatically in writing Arabic and Farsi but at least we can make it easier for the users. Isn't it? Thanks for all your help and support, Mostafa - Take the Internet to Go: Yahoo!Go puts the Internet in your pocket: mail, news, photos & more.
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
> Attached is a better patch where > everything is implemented: > - correct encoding for the fontenc > package. Mostafa tested this and > gave his OK for this issue. This is exteremly OK. > - special quotation mark definitions > for Farsi. Mostafa tested this and > gave his OK for this issue. This is OK. It should be removed later as you have mentioned it. BUT BUT BUT BUT In fact there are three commands not two commands. The command [EMAIL PROTECTED]@R{#1}} will make problems when we want the Arabic based on ARABI as the main language. Therefore remove this one from the file "languages" and keep the other two. We should use this command only if the main language is not Arabic_ARABI. > - fix for bug 2929 - use \textAR > and not \R. Improved version of Dov's > patch. Mostafa also tested this > successfully. No. I have not yet. In fact this is not basically a bug. > - use the document language also for > the TOC. This is a special arabi-package > thing. Please test. It needs more investigation. We should find the correct point in the source at which we could place the command "\TOCLanguage{farsi}". > - To make the TOC settings work and also > for other issues, arabic must be loaded > to babel too when farsi is the document > language. Please test. This fix should be temporarily too and so it should be removed later if the bug is fixed later. I will test it. > Mostafa, when you give your OK I'll commit > it. When there are possible problems with > the two new things, I'll only commit the > uncritical and tested issues from above. > (With this patch now also your > example_lyxified.lyx you sent two days > ago to be added to LyX's examples compiles > for me.) OK. I will prepare a clean patch for all these topics except the issues that are related to Arabic, because Dov and Uwe are working on this topic too. Mostafa - Get the free Yahoo! toolbar and rest assured with the added security of spyware protection.
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
Mostafa Vahedi schrieb: - use the document language also for the TOC. This is a special arabi-package thing. Please test. It needs more investigation. We should find the correct point in the source at which we could place the command "\TOCLanguage{farsi}". I think I have found it out: It must be called after babel. And futhermore to be able to use this, also arabic has to be loaded for babel when \TOClanguage{farsi} is used. Both is implemented in the patch. OK. I will prepare a clean patch for all these topics except the issues that are related to Arabic When you have tested the TOClanguage and the additional arabic call, I could also provide a clean patch out of mine that don't touches Arabic. thanks and regards Uwe
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
(Again, resending because it didn't arrive the first time; sorry if you got it twice) Uwe Stöhr wrote: Dov Feldstern schrieb: The question is, should we really go to all this trouble? (Well, not so much, but (a) should we separate between ArabTeX and Arabi, and (b) should we update lyx2lyx to deal with the language change, for backwards compatibility with something which we don't know whether or not it exists?) Or maybe you're right --- we should just stick with Arabi, that will be the only Arabic support that LyX provides. Sorry about all this mess :( . What do you say? Mostafa said he will investigate further and that he needs some time for it. I propose we postpone this after LyX 1.5.0. When no fileformat change is needed, this can be also implemented to Lyx 1.5.1. If not we can so it in LyX 1.6.0. regards Uwe I am in favor of applying the attached patch, and then applying Uwe's changes on top of the arabic_arabi language. The idea being that we support ArabTeX as much as it is currently supported (but not full support, by any means); and we're beginning to support Arabi for Arabic, too, but there's still work to be done. Hopefully, this will be good enough to draw some real Arabic users in, and they'll be able to help out with how best to continue. And if not, then there's no point in putting any more work into it. But the basis will be in. I don't think that a format change is needed, because we're using new languages, not changing the interpretation of existing ones. If there are users out there who have existing documents in ArabTeX, they'll have to replace all "\lang arabic" with "\lang arabic_arabtex", but that's all. I think this is reasonable. BTW, there's one more stage that I left out of the ArabTeX instructions, which may be why you were having trouble, Uwe. With this, it works, including chapters, etc. (copied straight from Dekel's instructions): 4. Get the file http://cs.haifa.ac.il/~dekelts/lyx/arab-article.layout and put it in ~/.lyx/layouts/ Run LyX and select the edit->reconfigure menu, and exit from LyX 5. Now you are ready to start writing Arabic documents: Start a new document, open the document layout popup (layout->document menu) Select article(Arabic) as the document class, default as the encoding, and English as the language (not Arabic!), and press OK. Use the F12 key to switch between Arabic and English. Index: src/insets/InsetTabular.cpp === --- src/insets/InsetTabular.cpp (revision 18851) +++ src/insets/InsetTabular.cpp (working copy) @@ -2286,6 +2286,9 @@ if (par.getParLanguage(buf.params())->lang() == "farsi") os << "\\textFR{"; + else if (par.getParLanguage(buf.params())->lang() == "arabic_arabi") + os << "\\textAR{"; + // currently, remaning RTL languages are arabic_arabtex and hebrew else os << "\\R{"; } Index: src/Font.cpp === --- src/Font.cpp(revision 18851) +++ src/Font.cpp(working copy) @@ -752,10 +752,18 @@ if (language()->lang() == "farsi") { os << "\\textFR{"; count += 8; + } else if (language()->lang() == "arabic_arabi") { + os << "\\textAR{"; + count += 8; } else if (!isRightToLeft() && + base.language()->lang() == "arabic_arabi") { + os << "\\textLR{"; + count += 8; + } else if (!isRightToLeft() && base.language()->lang() == "farsi") { os << "\\textLR{"; count += 8; + // currently the remaining RTL languages are arabic_arabtex and hebrew } else if (isRightToLeft() != prev.isRightToLeft()) { if (isRightToLeft()) { os << "\\R{"; Index: src/Text.cpp === --- src/Text.cpp(revision 18851) +++ src/Text.cpp(working copy) @@ -369,7 +369,8 @@ if (isPrintable(c)) { Language const * language = font.language(); if (language->rightToLeft()) { - if (language->lang() == "arabic" || + if (language->lang() == "arabic_arabtex" || + language->lang() == "arabic_arabi" || language->lang() == "farsi") { if (Encodings::isComposeChar_arabic(c)) return 0; Index: src/rowpainter.cpp ===
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
On Friday 22 June 2007 17:04:25 Dov Feldstern wrote: > I don't think that a format change is needed, because we're using new > languages, not changing the interpretation of existing ones. If there > are users out there who have existing documents in ArabTeX, they'll have > to replace all "\lang arabic" with "\lang arabic_arabtex", but that's > all. I think this is reasonable. I am sorry Dov, replacing "\lang arabic" by "\lang arabic_arabtex" is a file format change and an easy one to implement in lyx2lyx. Better yet, if I understood what you said it means that all the previous users of Arabic were using arabtex, right? If you, Mostafa and Uwe agree on a set of patches to be applied before rc2 this can proceed. -- José Abílio
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
I applied the following patch: http://www.lyx.org/trac/changeset/18858 Mostafa Vahedi schrieb: Attached is a better patch where everything is implemented: - correct encoding for the fontenc package. Mostafa tested this and gave his OK for this issue. This is exteremly OK. This is included in the committed patch. - special quotation mark definitions for Farsi. Mostafa tested this and gave his OK for this issue. This is OK. It should be removed later as you have mentioned it. I added a comment regarding this. BUT BUT BUT BUT In fact there are three commands not two commands. The command [EMAIL PROTECTED]@R{#1}} will make problems when we want the Arabic based on ARABI as the main language. Therefore remove this one from the file "languages" and keep the other two. We should use this command only if the main language is not Arabic_ARABI. OK, done in the commited patch. - fix for bug 2929 - use \textAR and not \R. Improved version of Dov's patch. Mostafa also tested this successfully. No. I have not yet. In fact this is not basically a bug. OK I droped it from the patch. - use the document language also for the TOC. This is a special arabi-package thing. Please test. It needs more investigation. We should find the correct point in the source at which we could place the command "\TOCLanguage{farsi}". - To make the TOC settings work and also for other issues, arabic must be loaded to babel too when farsi is the document language. Please test. This fix should be temporarily too and so it should be removed later if the bug is fixed later. I will test it. What is your result for these two? Only with them I can compile the example_lyxified.lyx you sent to the list. regards Uwe
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
Uwe Stöhr schrieb: I applied the following patch: http://www.lyx.org/trac/changeset/18858 I corrected it: http://www.lyx.org/trac/changeset/18859 Uwe
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
Attached is Dov's latest patch. I added the font encoding stuff and the fileformat change. It is furthermore needed to load arabtex in the preamble when the language is arabic_arabtex, I added this to the patch. Please test. --- Dov wrote: > +arabic_arabtex arabic "Arabic (ArabTeX)" true cp1256 ar_SA "" When arabtex is used, babel should not be loaded (because babel is not supported by arabtex and to avaoid interferences with arabi), I therefore corrected this line. All other parts of your patch are OK. I tested the last hour all combinations of languages to be save that our patch does not have any drawbacks and I think it is save to apply. --- There is one thing that needs to be improved: As I told you arabtex must be loaded in the preamble, but it must be loaded whenever arabic_arabtex is used in the document, also when arabic is not the document language. I don't know how to implement this. Currently I hae this: if (language->lang() == "arabic_arabtex") { os << "\\usepackage{arabtex}\n"; But this is only the document language. How can I test if one of the languages used in the document is arabic? thanks and regards Uwe Index: lib/languages === --- lib/languages (revision 18861) +++ lib/languages (working copy) @@ -1,7 +1,8 @@ # name babel name GUI name RTL? encoding code latex options afrikaans afrikaans "Afrikaans" false iso8859-15 af_ZA "" americanamerican "American" false iso8859-15 en_US "" -arabic arabic "Arabic" true cp1256 ar_SA "" +arabic_arabtex "" "Arabic (ArabTeX)" true cp1256 ar_SA "" +arabic_arabi arabic "Arabic (Arabi)" true cp1256 ar_SA "" armenian"" "Armenian" false armscii8 hy_AM "" austrianaustrian "Austrian" false iso8859-15 de_AT "" naustrian naustrian "Austrian (new spelling)" false iso8859-15 de_AT "" Index: lib/lyx2lyx/LyX.py === --- lib/lyx2lyx/LyX.py (revision 18861) +++ lib/lyx2lyx/LyX.py (working copy) @@ -77,7 +77,7 @@ ("1_2", [220], generate_minor_versions("1.2" , 4)), ("1_3", [221], generate_minor_versions("1.3" , 7)), ("1_4", range(222,246), generate_minor_versions("1.4" , 4)), - ("1_5", range(246,275), generate_minor_versions("1.5" , 0))] + ("1_5", range(246,276), generate_minor_versions("1.5" , 0))] def formats_list(): Index: lib/lyx2lyx/lyx_1_5.py === --- lib/lyx2lyx/lyx_1_5.py (revision 18861) +++ lib/lyx2lyx/lyx_1_5.py (working copy) @@ -1749,6 +1749,34 @@ r'\end_layout' ] +def convert_arabic (document): +if document.language == "arabic": +document.language = "arabic_arabtex" +i = find_token(document.header, "\\language", 0) +if i != -1: +document.header[i] = "\\language arabic_arabtex" +i = 0 +while i < len(document.body): +h = document.body[i].find("\lang arabic", 0, len(document.body[i])) +if (h != -1): +# change the language name +document.body[i] = '\lang arabic_arabtex' +i = i + 1 + +def revert_arabic (document): +if document.language == "arabic_arabtex": +document.language = "arabic" +i = find_token(document.header, "\\language", 0) +if i != -1: +document.header[i] = "\\language arabic" +i = 0 +while i < len(document.body): +h = document.body[i].find("\lang arabic_arabtex", 0, len(document.body[i])) +if (h != -1): +# change the language name +document.body[i] = '\lang arabic' +i = i + 1 + ## # Conversion hub # @@ -1782,10 +1810,12 @@ [271, [convert_ext_font_sizes]], [272, []], [273, []], - [274, [normalize_font_whitespace_274]] + [274, [normalize_font_whitespace_274]], + [275, [convert_arabic]] ] revert = [ + [274, [revert_arabic]], [273, []], [272, [revert_separator_layout]], [271, [revert_preamble_listings_params, revert_listings_inset, revert_include_listings]], Index: src/Buffer.cpp === --- src/Buffer.cpp (revision 18861) +++ src/Buffer.cpp (working copy) @@ -142,7 +142,7 @@ namespace { -int const LYX_FORMAT = 274; +int const LYX_FORMAT = 275; } // namespace anon Index: src/BufferParams.cpp === --- src/BufferParams.cpp (revision 18861) +++ src/BufferParams.cpp (working copy) @@ -895,9 +895,9 @@ // set font encoding // this one is not per buffer - // for Farsi we also need to load the LAE and LFE encoding + // for arabic_arabi and farsi we
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
Uwe Stöhr wrote: Attached is Dov's latest patch. I added the font encoding stuff and the fileformat change. It is furthermore needed to load arabtex in the preamble when the language is arabic_arabtex, I added this to the patch. Please test. Thanks for taking care of the lyx2lyx stuff! I added the requisite comment to the FORMAT file --- could you just check your name (I'm not sure my accents are correct) and email? Regarding your change for the arabtex preamble --- I don't think this code will ever be used, because one of the specific instructions for using ArabTeX is to *never* switch the document language to "Arabic (ArabTeX)". It doesn't work if you do that. The document language should always be English (or whatever your primary language is, but *not* Arabic (ArabTeX) ). The preamble stuff currently gets added by the manual changes to the preferences file during the one-time setup. So I removed this code in the attached patch. I guess once we know how to activate it even if the language is used anywhere in the document, then we can add it in again, and we won't need this change in the preferences file. --- Dov wrote: > +arabic_arabtex arabic "Arabic (ArabTeX)" true cp1256 ar_SA "" When arabtex is used, babel should not be loaded (because babel is not supported by arabtex and to avaoid interferences with arabi), I therefore corrected this line. I know this is not correct, and I had originally removed it, but then the language switch commands (added to the preferences during the one-time setup for ArabTeX) were not taking effect. It looks like the language switch commands are currently used only if babel is activated. That's why I had added "arabic" to the babel column, just so that the language switch commands would be used. I realize that this is not strictly correct, and may even cause trouble when trying to mix with other languages --- but that's the way it's currently done. We could achieve this by changing the code (in Font.cpp, I think), but I'd rather make more intrusive changes to the actual code only when we really know what we should be doing, and at the moment I don't feel that we do. In the attached patch I added back "arabtex" instead of "arabic" (+ a comment explaining this), just to make it a little clearer. This seems to work. Again, once we know what should really be done, we can fix it in the correct way. All other parts of your patch are OK. I tested the last hour all combinations of languages to be save that our patch does not have any drawbacks and I think it is save to apply. --- There is one thing that needs to be improved: As I told you arabtex must be loaded in the preamble, but it must be loaded whenever arabic_arabtex is used in the document, also when arabic is not the document language. I don't know how to implement this. Currently I hae this: if (language->lang() == "arabic_arabtex") { os << "\\usepackage{arabtex}\n"; But this is only the document language. How can I test if one of the languages used in the document is arabic? I don't know how to do this in the code. Currently, this is not a problem if the required commands are added to the preferences file (during the one-time ArabTeX setup), as explained above. thanks and regards Uwe Thanks for your work and comments! Dov Index: lib/languages === --- lib/languages (revision 18871) +++ lib/languages (working copy) @@ -1,7 +1,10 @@ # name babel name GUI nameRTL? encoding code latex options afrikaans afrikaans "Afrikaans" false iso8859-15 af_ZA "" americanamerican "American" false iso8859-15 en_US "" -arabic arabic "Arabic"true cp1256 ar_SA "" +# ArabTeX doesn't actually use babel, but something is necessary here in order +# to activate the language-switch mechanism +arabic_arabtex arabtex "Arabic (ArabTeX)" true cp1256 ar_SA "" +arabic_arabi arabic"Arabic (Arabi)" true cp1256 ar_SA "" armenian"" "Armenian" false armscii8 hy_AM "" austrianaustrian "Austrian" false iso8859-15 de_AT "" naustrian naustrian "Austrian (new spelling)" false iso8859-15 de_AT "" Index: lib/lyx2lyx/LyX.py === --- lib/lyx2lyx/LyX.py (revision 18871) +++ lib/lyx2lyx/LyX.py (working copy) @@ -77,7 +77,7 @@ ("1_2", [220], generate_minor_versions("1.2" , 4)), ("1_3", [221], generate_minor_versions("1.3" , 7)), ("1_4", range(222,246), generate_minor_versions("1.4" , 4)), - ("1_5", range(246,275), generate_minor_versions("1.5" , 0))] + ("1_5", range(246,276), generate_minor_versions("1.5" , 0))] def formats_list(): Index: lib/lyx2lyx/lyx_1_5.py ===
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
Any thoughts on this? Attached a newer version of the patch, the only difference is incrementing the format number by one, after Jurgen's format change. Dov Feldstern wrote: Uwe Stöhr wrote: Attached is Dov's latest patch. I added the font encoding stuff and the fileformat change. It is furthermore needed to load arabtex in the preamble when the language is arabic_arabtex, I added this to the patch. Please test. Thanks for taking care of the lyx2lyx stuff! I added the requisite comment to the FORMAT file --- could you just check your name (I'm not sure my accents are correct) and email? Regarding your change for the arabtex preamble --- I don't think this code will ever be used, because one of the specific instructions for using ArabTeX is to *never* switch the document language to "Arabic (ArabTeX)". It doesn't work if you do that. The document language should always be English (or whatever your primary language is, but *not* Arabic (ArabTeX) ). The preamble stuff currently gets added by the manual changes to the preferences file during the one-time setup. So I removed this code in the attached patch. I guess once we know how to activate it even if the language is used anywhere in the document, then we can add it in again, and we won't need this change in the preferences file. --- Dov wrote: > +arabic_arabtex arabic "Arabic (ArabTeX)" true cp1256 ar_SA "" When arabtex is used, babel should not be loaded (because babel is not supported by arabtex and to avaoid interferences with arabi), I therefore corrected this line. I know this is not correct, and I had originally removed it, but then the language switch commands (added to the preferences during the one-time setup for ArabTeX) were not taking effect. It looks like the language switch commands are currently used only if babel is activated. That's why I had added "arabic" to the babel column, just so that the language switch commands would be used. I realize that this is not strictly correct, and may even cause trouble when trying to mix with other languages --- but that's the way it's currently done. We could achieve this by changing the code (in Font.cpp, I think), but I'd rather make more intrusive changes to the actual code only when we really know what we should be doing, and at the moment I don't feel that we do. In the attached patch I added back "arabtex" instead of "arabic" (+ a comment explaining this), just to make it a little clearer. This seems to work. Again, once we know what should really be done, we can fix it in the correct way. All other parts of your patch are OK. I tested the last hour all combinations of languages to be save that our patch does not have any drawbacks and I think it is save to apply. --- There is one thing that needs to be improved: As I told you arabtex must be loaded in the preamble, but it must be loaded whenever arabic_arabtex is used in the document, also when arabic is not the document language. I don't know how to implement this. Currently I hae this: if (language->lang() == "arabic_arabtex") { os << "\\usepackage{arabtex}\n"; But this is only the document language. How can I test if one of the languages used in the document is arabic? I don't know how to do this in the code. Currently, this is not a problem if the required commands are added to the preferences file (during the one-time ArabTeX setup), as explained above. thanks and regards Uwe Thanks for your work and comments! Dov Index: src/insets/InsetTabular.cpp === --- src/insets/InsetTabular.cpp (revision 18885) +++ src/insets/InsetTabular.cpp (working copy) @@ -2286,6 +2286,9 @@ if (par.getParLanguage(buf.params())->lang() == "farsi") os << "\\textFR{"; + else if (par.getParLanguage(buf.params())->lang() == "arabic_arabi") + os << "\\textAR{"; + // currently, remaning RTL languages are arabic_arabtex and hebrew else os << "\\R{"; } Index: src/Font.cpp === --- src/Font.cpp(revision 18885) +++ src/Font.cpp(working copy) @@ -756,6 +756,14 @@ base.language()->lang() == "farsi") { os << "\\textLR{"; count += 8; + } else if (language()->lang() == "arabic_arabi") { + os << "\\textAR{"; + count += 8; + } else if (!isRightToLeft() && + base.language()->lang() == "arabic_arabi") { + os << "\\textLR{"; + count += 8; + // currently the remaining RTL languages are arabic_arabtex and
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
> "Dov" == Dov Feldstern <[EMAIL PROTECTED]> writes: Dov> Any thoughts on this? Attached a newer version of the patch, the Dov> only difference is incrementing the format number by one, after Dov> Jurgen's format change. I'll let Jose decide about whether it is good enough now, but I do not like much the hardcoding which is flourishing in he code base. I think it is time to understand what are the interesting features in languages and to put them in the lib/languages file (probably by changing its format to match more closely layout files. On a question more related to the actual patch: is it really necessary to have special code in InsetTabular? This looks very zrong to me. Isn't there some incarnation of Font::latexWriteStartChanges that would work? JMarc
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
Jean-Marc Lasgouttes wrote: "Dov" == Dov Feldstern <[EMAIL PROTECTED]> writes: Dov> Any thoughts on this? Attached a newer version of the patch, the Dov> only difference is incrementing the format number by one, after Dov> Jurgen's format change. I'll let Jose decide about whether it is good enough now, but I do not like much the hardcoding which is flourishing in he code base. I think it is time to understand what are the interesting features in languages and to put them in the lib/languages file (probably by changing its format to match more closely layout files. Jean-Marc, I'm glad you're saying this, because I feel very much the same way, but I felt that I had already held this patch up enough already, and I do think that it should go in for now... In it's current form, the patch only has hardcoding in places where language hardcoding already existed previously, most of it for a very long time. So I think that for now this should go in --- especially since there's also the format change involved, and not laying the foundation for that now will mean waiting until 1.6... I think there are a *lot* of multi-language issues which have to be worked out in LyX, but that's not for now... On a question more related to the actual patch: is it really necessary to have special code in InsetTabular? This looks very zrong to me. Isn't there some incarnation of Font::latexWriteStartChanges that would work? Yeah, I don't know why it should be there... But again, I believe that the basis has been there for a very long time, and now's not the time to start fixing this... JMarc
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
Dov Feldstern schrieb: Thanks for taking care of the lyx2lyx stuff! I added the requisite comment to the FORMAT file --- could you just check your name (I'm not sure my accents are correct) and email? The name is fine there. Btw. Why do you have a [EMAIL PROTECTED] email adress and me not? Regarding your change for the arabtex preamble --- I don't think this code will ever be used, because one of the specific instructions for using ArabTeX is to *never* switch the document language to "Arabic (ArabTeX)". It doesn't work if you do that. OK then, but besides this patch I would use the method to insert Farsi related stuff whenever Fasi is used. So I removed this code in the attached patch. I guess once we know how to activate it even if the language is used anywhere in the document, then we can add it in again, and we won't need this change in the preferences file. OK. > +arabic_arabtex arabic "Arabic (ArabTeX)" true cp1256 ar_SA "" When arabtex is used, babel should not be loaded (because babel is not supported by arabtex and to avaoid interferences with arabi), I therefore corrected this line. I know this is not correct, and I had originally removed it, but then the language switch commands (added to the preferences during the one-time setup for ArabTeX) were not taking effect. It looks like the language switch commands are currently used only if babel is activated. I cannot reproduce this - works fine here or I do need a better recipe to reproduce. We should not commit obiously wrong stuff. What you describe is a bug we should fix and not introducing a workaround. I only changed in your patch the arabic babel entry in "languages" and a bit whitespace. I'll fix the problem you described when I can reproduce it. All other parts of your patch are OK. José gave his OK for the case when we agreed to a solution and we tested it hardly enough the last days I think. Therefore I committed it: http://www.lyx.org/trac/changeset/18887 When there are unseen side effects please shout! Thanks for the good work and the discussions that lead to this solution and best regards Uwe
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
Uwe Stöhr wrote: > Btw. Why do you have a [EMAIL PROTECTED] email adress and me not? You do. Now you just need to learn how to use it ;-) $ ssh [EMAIL PROTECTED] $ cat .forward [EMAIL PROTECTED] -- Angus
Re: [patch] better fix for bug 2928: make Arabic and Farsi correctly usable
Uwe Stöhr wrote: Dov Feldstern schrieb: Thanks for taking care of the lyx2lyx stuff! I added the requisite comment to the FORMAT file --- could you just check your name (I'm not sure my accents are correct) and email? The name is fine there. Btw. Why do you have a [EMAIL PROTECTED] email adress and me not? I don't know, JMarc gave me one when he opened my account. Just ask him, I'm sure he'll let you have one ;) . Or maybe you already have, and just never knew --- see if you have a .forward file in your home directory, and where it's pointing to... > +arabic_arabtex arabic "Arabic (ArabTeX)" true cp1256 ar_SA "" When arabtex is used, babel should not be loaded (because babel is not supported by arabtex and to avaoid interferences with arabi), I therefore corrected this line. I know this is not correct, and I had originally removed it, but then the language switch commands (added to the preferences during the one-time setup for ArabTeX) were not taking effect. It looks like the language switch commands are currently used only if babel is activated. I cannot reproduce this - works fine here or I do need a better recipe to reproduce. Try the attached file. With the current situation (latest svn) I get the output in arabtex2-bad.dvi. Adding arabtex (or anything, it doesn't matter what, it could say 'dummy') to lib/languages as in my patch fixes it, and I get the output as portrayed in arabtex2-good.lyx. We should not commit obiously wrong stuff. What you describe is a bug we should fix and not introducing a workaround. As I explained, I'd rather put in a patch which only changes a configuration file (lib/languages) than start playing around with the actual code, until I'm sure what the correct way to solve this is. And right now I don't feel that I am. Also, I think that what JMarc said last night about the hardcoding is relevant here... I only changed in your patch the arabic babel entry in "languages" and a bit whitespace. I'll fix the problem you described when I can reproduce it. All other parts of your patch are OK. José gave his OK for the case when we agreed to a solution and we tested it hardly enough the last days I think. Therefore I committed it: http://www.lyx.org/trac/changeset/18887 Thanks, Uwe! When there are unseen side effects please shout! Thanks for the good work and the discussions that lead to this solution and best regards Uwe arabtex2.lyx Description: application/lyx arabtex2-bad.dvi Description: TeX dvi file arabtex2-good.dvi Description: TeX dvi file