Re: [XeTeX] additional beginL endL nodes in math
On 2015-04-16, David Carlisle d.p.carli...@gmail.com wrote: On 16 April 2015 at 20:51, Khaled Hosny khaledho...@eglug.org wrote: That was very naive and did not work when inline math is broken over multiple lines, so I reverted this and the whole TeX-XeT business. XeTeX in TeX Live 2015 should behave identical to previous versions in this area. Ah OK, sorry it didn't work out. I do see the problems with \specials and \writes in RTL text is a real problem. It seems to me that this is a consequence of some really bad design decisions in TeX ! I have wondered whether the right way to is to make a new variant of TeX in which there are invisible whatsits (could call them boojums, perhaps) which are ignored by most of TeX's list-(un)building activities. For example, a glue-boojum-glue sequence would be treated as a glue sequence, except that the boojum would remain behind when the glue was discarded. Boojum specials would instantly solve the absurd problem of colouring display maths without screwing up the spacing. -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
On 17 April 2015 at 13:07, Julian Bradfield jcb+xe...@jcbradfield.org wrote: On 2015-04-16, David Carlisle d.p.carli...@gmail.com wrote: On 16 April 2015 at 20:51, Khaled Hosny khaledho...@eglug.org wrote: That was very naive and did not work when inline math is broken over multiple lines, so I reverted this and the whole TeX-XeT business. XeTeX in TeX Live 2015 should behave identical to previous versions in this area. Ah OK, sorry it didn't work out. I do see the problems with \specials and \writes in RTL text is a real problem. It seems to me that this is a consequence of some really bad design decisions in TeX ! I have wondered whether the right way to is to make a new variant of TeX in which there are invisible whatsits (could call them boojums, perhaps) which are ignored by most of TeX's list-(un)building activities. For example, a glue-boojum-glue sequence would be treated as a glue sequence, except that the boojum would remain behind when the glue was discarded. possibly but in the interests of keeping divergence of tex-related engines to a minimum as a first approach (if different approaches were being considered) I'd look to what luatex is doing, where colour and directionality can be specified without introducing new nodes colour being an attribute of the font for example. Boojum specials would instantly solve the absurd problem of colouring display maths without screwing up the spacing. With care it should be possible now to colour maths without affecting spacing (although it's true that more care than one would wish is needed) David -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
David Carlisle wrote: possibly but in the interests of keeping divergence of tex-related engines to a minimum as a first approach (if different approaches were being considered) I'd look to what luatex is doing, where colour and directionality can be specified without introducing new nodes colour being an attribute of the font for example. Colour is already an attribute of the font in XeTeX, is it not ? The syntax certainly suggests that it is : \font \thisfont = Tw Cen MT Condensed:color=968A4D scaled 955 for example. ** Phil. -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
On 17 April 2015 at 15:26, Philip Taylor p.tay...@rhul.ac.uk wrote: David Carlisle wrote: possibly but in the interests of keeping divergence of tex-related engines to a minimum as a first approach (if different approaches were being considered) I'd look to what luatex is doing, where colour and directionality can be specified without introducing new nodes colour being an attribute of the font for example. Colour is already an attribute of the font in XeTeX, is it not ? The syntax certainly suggests that it is : \font \thisfont = Tw Cen MT Condensed:color=968A4D scaled 955 for example. ** Phil. yes but the latex color package doesn't know that (who wrote that package, anyway) David -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
2015-04-17 16:31 GMT+02:00 David Carlisle d.p.carli...@gmail.com: On 17 April 2015 at 15:26, Philip Taylor p.tay...@rhul.ac.uk wrote: David Carlisle wrote: possibly but in the interests of keeping divergence of tex-related engines to a minimum as a first approach (if different approaches were being considered) I'd look to what luatex is doing, where colour and directionality can be specified without introducing new nodes colour being an attribute of the font for example. Colour is already an attribute of the font in XeTeX, is it not ? The syntax certainly suggests that it is : \font \thisfont = Tw Cen MT Condensed:color=968A4D scaled 955 for example. ** Phil. yes but the latex color package doesn't know that (who wrote that package, anyway) Colour is a font attribut in XeTeX but AFAIK it allowx RGB and RGBA only, CMYK is not supported. If I want to print the document by offset, I have to use colour specials, otherwise I risk unwanted result. Zdeněk Wagner http://hroch486.icpf.cas.cz/wagner/ http://icebearsoft.euweb.cz David -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
On Fri, Apr 17, 2015 at 03:31:33PM +0100, David Carlisle wrote: On 17 April 2015 at 15:26, Philip Taylor p.tay...@rhul.ac.uk wrote: David Carlisle wrote: possibly but in the interests of keeping divergence of tex-related engines to a minimum as a first approach (if different approaches were being considered) I'd look to what luatex is doing, where colour and directionality can be specified without introducing new nodes colour being an attribute of the font for example. Colour is already an attribute of the font in XeTeX, is it not ? The syntax certainly suggests that it is : \font \thisfont = Tw Cen MT Condensed:color=968A4D scaled 955 for example. ** Phil. yes but the latex color package doesn't know that (who wrote that package, anyway) And you can’t use it for all cases, e.g. rules will no be colored, nor it can be used to set background color. LuaTeX attributes are generic and can be used anywhere, and some packages use to to pass around color information, but without the ability to process nodes using Lua, its use will be very limited. Regards, Khaled -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
Hi David, Zdenek, and others
On 18/04/2015, at 1:05, Zdenek Wagner zdenek.wag...@gmail.com wrote:
2015-04-17 16:31 GMT+02:00 David Carlisle d.p.carli...@gmail.com:
package, anyway)
Colour is a font attribut in XeTeX but AFAIK it allowx RGB and RGBA only,
CMYK is not supported. If I want to print the document by offset, I have to
use colour specials, otherwise I risk unwanted result.
It is not just colour information that one may want to insert.
Here are some more instances in support of boojums (using pdfTeX , not
XeTeX).
A.
For tagged PDF you may need to tag individual characters with attributes for
accessibility and Copy/Paste, such as to get the PDF page-contents stream
looking like:
/Span /Alt(...)/ActualTextFFEF BDC
... normal TeX-placed material ...
EMC
\pdfliteral page { }can provide the extra PDF coding lines,
but this is 2 extra boojums for each actual character in the math list.
B.
I'm currently writing a paper describing a method to attach tooltips to
mathematical symbols and expressions. After setting the chunks in boxes for
measuring, this ultimately puts
\pdfannot
into the math-stream.
To not affect spacing, this would need to be a boojum surely.
I can supply instances where spacing has changed, by an extra thin space.
Sometimes placing extra {...} avoids this extra space, other cases require a \!
to fix.
Zdeněk Wagner
http://hroch486.icpf.cas.cz/wagner/
http://icebearsoft.euweb.cz
David
Is there a good tool or TeXnique that lets one see the contents of a
math-stream, after all macros are processed, and during the internal
page-construction stage?
I'd like something a bit better than just examining box contents after using
\tracingall .
Cheers,
Ross
--
Subscriptions, Archive, and List information, etc.:
http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
On 17 April 2015 at 21:43, Ross Moore ross.mo...@mq.edu.au wrote:
B.
I'm currently writing a paper describing a method to attach tooltips to
mathematical symbols and expressions. After setting the chunks in boxes for
measuring, this ultimately puts
\pdfannot
into the math-stream.
To not affect spacing, this would need to be a boojum surely.
As with colour, the \pdfannot whatsit shouldn't affect spacing so long as
you
avoid placing it in the middle of a word where it may affect inter-letter
kerns or ligatures.
\pdfannot{..}a should have the same spacing as a
\mathrel{\pdfannot{..}=} should have the same spacing as =
I can supply instances where spacing has changed, by an extra thin space.
Sometimes placing extra {...} avoids this extra space, other cases require
a \! to fix.
David
--
Subscriptions, Archive, and List information, etc.:
http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
Is there a good tool or TeXnique that lets one see the contents of a math-stream, after all macros are processed, and during the internal page-construction stage? I'd like something a bit better than just examining box contents after using \tracingall . $x+y\showlists$ might be what you are looking for? -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
On 16 April 2015 at 01:21, Vafa Khalighi glkv...@gmail.com wrote: I also support Khaled's changes to xetex; I only tested it few months ago and if the issues I reported are fixed now, then it is good. And since when LaTeX has a package/ or test files that uses \beginL \endL? All I remember is that there are only babel's rlbabel.def file and then arabicore package but these packages themselves contain bugs (irrelevant to xetex). bidi is the only big package that uses e-tex bidirectional algorithm for right to left typesetting. With the new changes of xetex, bidi package needs to be written from scrtach but I have no complain against it; the news that the \special and other issues fixed in xetex makes me so happy that I will not complain about changes. I am grateful to Khaled for fixing them. The problem is not so much documents or packages that use \beginL rather those that do not. The document I posted earlier does not use any features of xetex or even etex and is usable with original classic tex82. in tex, etex, pdftex, luatex and xetex 2014 it typesets 700 in xetex 2015 it typesets 0. Whichever way you look at it this is a big incompatibility and I started the thread to gain a better understanding of why the change was made. So now I see that while it's not ideal, given the resources and the seriousness of the bugs that were being fixed, it's an understandable decision to go with this route at the present time. It causes us some problems in latex maintenance, but we'll cope. David -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
On 16 April 2015 at 12:35, Khaled Hosny dr.khaled.ho...@gmail.com wrote: I’m testing a change that removes the direction node from around inline math and instead (ab)uses the presence of \math(on|off) node to output the necessary dvi opcode, and it seems to fix this issue as well. Lets hope that there is no some blackmagic^W deep TeX hackery that plays with the this node as well. Ohh, interesting, thanks. I nearly suggested something like that originally but my knowledge of the tex internals isn't really up to suggesting code changes at that level:-) David -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
On 16 April 2015 at 11:54, Vafa Khalighi glkv...@gmail.com wrote: I have not got TeXLive 2015 but that is just bad. Does the new xetex effect any other of math typesetting that used to work fine with Knuth TeX? yes see the example I posted in the second message I posted to this thread in Knuth TeX it typesets 700 in xetex 2015 it typesets 0, That was engineered to typeset a test value, but real documents that produce different spacing are inevitable, as this later breqn example shows. does it break any package? with breqn the issue is not too bad since breqn is an experimental package and not a lot of people are aware of it or use it. I am not saying that it is ok but that if it does not break any other packages, then it is not too bad. It's hard to know what which packages or documents will change. tex-xet makes math expressions opaque to tests with \lastskip, \unskip and friends as the extra non-removable \endR node that is inserted stops the expression being dis-assembled. As I said in the initial message in this thread that breaks 77 of our test files and affects any document that is dissembling horizontal lists constructed from math. It was easy to guess breqn was doing that kind of thing so I made a small example and as expected it failed in 2015. Other packages, who knows David -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
I’m testing a change that removes the direction node from around inline
math and instead (ab)uses the presence of \math(on|off) node to output the
necessary dvi opcode, and it seems to fix this issue as well. Lets hope
that there is no some blackmagic^W deep
TeX hackery that plays with the this node as well.
On Thu, Apr 16, 2015 at 12:47 PM, David Carlisle d.p.carli...@gmail.com
wrote:
To see a non contrived example where the change breaks existing
documents, take this example
(using a math expression from the breqn package documentation)
\documentclass[]{article}
\usepackage{breqn}
\begin{document}
aaa
\begin{dmath*}
g = 0 + 1 - 1
\end{dmath*}
aaa
\begin{dmath*}
T(n) \hiderel{\leq} T(2^{\lceil\lg n\rceil})
\leq c(3^{\lceil\lg n\rceil}
-2^{\lceil\lg n\rceil})
3c\cdot3^{\lg n}
=3c\,n^{\lg3}
\end{dmath*}.
\end{document}
I attach the output from xelatex 2014 and 2015,
basically breqn doesn't work with xelatex 2015.
I haven't fully traced it but I suspect (since math lists are opaque now)
that this is not fixable by changing macro code.
David
--
Subscriptions, Archive, and List information, etc.:
http://tug.org/mailman/listinfo/xetex
--
Subscriptions, Archive, and List information, etc.:
http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
To see a non contrived example where the change breaks existing
documents, take this example
(using a math expression from the breqn package documentation)
\documentclass[]{article}
\usepackage{breqn}
\begin{document}
aaa
\begin{dmath*}
g = 0 + 1 - 1
\end{dmath*}
aaa
\begin{dmath*}
T(n) \hiderel{\leq} T(2^{\lceil\lg n\rceil})
\leq c(3^{\lceil\lg n\rceil}
-2^{\lceil\lg n\rceil})
3c\cdot3^{\lg n}
=3c\,n^{\lg3}
\end{dmath*}.
\end{document}
I attach the output from xelatex 2014 and 2015,
basically breqn doesn't work with xelatex 2015.
I haven't fully traced it but I suspect (since math lists are opaque now)
that this is not fixable by changing macro code.
David
--
Subscriptions, Archive, and List information, etc.:
http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
I have not got TeXLive 2015 but that is just bad. Does the new xetex effect
any other of math typesetting that used to work fine with Knuth TeX? does
it break any package? with breqn the issue is not too bad since breqn is an
experimental package and not a lot of people are aware of it or use it. I
am not saying that it is ok but that if it does not break any other
packages, then it is not too bad.
On Thu, Apr 16, 2015 at 8:47 PM, David Carlisle d.p.carli...@gmail.com
wrote:
To see a non contrived example where the change breaks existing
documents, take this example
(using a math expression from the breqn package documentation)
\documentclass[]{article}
\usepackage{breqn}
\begin{document}
aaa
\begin{dmath*}
g = 0 + 1 - 1
\end{dmath*}
aaa
\begin{dmath*}
T(n) \hiderel{\leq} T(2^{\lceil\lg n\rceil})
\leq c(3^{\lceil\lg n\rceil}
-2^{\lceil\lg n\rceil})
3c\cdot3^{\lg n}
=3c\,n^{\lg3}
\end{dmath*}.
\end{document}
I attach the output from xelatex 2014 and 2015,
basically breqn doesn't work with xelatex 2015.
I haven't fully traced it but I suspect (since math lists are opaque now)
that this is not fixable by changing macro code.
David
--
Subscriptions, Archive, and List information, etc.:
http://tug.org/mailman/listinfo/xetex
--
Subscriptions, Archive, and List information, etc.:
http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
On 16 April 2015 at 20:51, Khaled Hosny khaledho...@eglug.org wrote: On Thu, Apr 16, 2015 at 12:39:47PM +0100, David Carlisle wrote: On 16 April 2015 at 12:35, Khaled Hosny dr.khaled.ho...@gmail.com wrote: I’m testing a change that removes the direction node from around inline math and instead (ab)uses the presence of \math(on|off) node to output the necessary dvi opcode, and it seems to fix this issue as well. Lets hope that there is no some blackmagic^W deep TeX hackery that plays with the this node as well. Ohh, interesting, thanks. I nearly suggested something like that originally but my knowledge of the tex internals isn't really up to suggesting code changes at that level:-) That was very naive and did not work when inline math is broken over multiple lines, so I reverted this and the whole TeX-XeT business. XeTeX in TeX Live 2015 should behave identical to previous versions in this area. Regards, Khaled Ah OK, sorry it didn't work out. I do see the problems with \specials and \writes in RTL text is a real problem. David -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
Is there no other way that the issues you have could be addressed without introducing TeX-XeT? The more we look at this the more it seems a very unfortunate step, introducing several incompatibilities with pdftex and luatex. Especially if there is a possibility of moving in the end to an Omega/luatex model for bidirectional support introducing a change at this point seems a strange choice. The extra nodes added here really are a backward step, my first suggestion of only conditionally adding them doesn't really work if boxes containing math are saved and used in different contexts. Especially now in the Texlive 2015 pretest period there is so little time to test any changes. Would it be possible to back it out this change at least until after TL2015 is released That would give time to investigate a more compatible way to address the issues? David -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
On Wed, Apr 15, 2015 at 09:07:05PM +0100, David Carlisle wrote: Let me go and read some of those issues, thanks for the pointer, Just in case it is not clear why reviving the long dead TeX-XeT is a fix here; TeX-XeT does not do any actual reversal but rather outputs special opcodes in the DVI file and the DVI driver does the actual reversal, now in the driver we do not “physically” reverse the order of the nodes but instead change their X positions so that they are visually reversed. It is possible to do the X manipulation in TeX, Omega does that, but not for a mere mortal like me (and it does not help that the TeX code is complex, interwoven and written in an archaic language I only superficially understand), it was much easier to do it in the driver. Regards, Khaled -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
I do not understand. If system A performed function Y until now, yet its designer/maintainer wishes it to perform function Y' henceforth Khaled's point is that XeTeX did not work correctly on the issue at hand until he made the change we're discussing. It's unfair to characterise his change as a change in behaviour of the TeX engine, it was a bugfix. Arthur -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
Hi all, I do not want to vote but I feel that TeX behaviour was really wrong. When I type an English paragraph quoting in a middle of a sentence an Urdu text آپ جآمع مسجد کی مینار پر جا کستِ ہِں without special LTR/RTL markup, the output will be incorrect although I see it right in the text editor as well as in this mail mesage and no LTR/RTL markup is needed. I have not examined the new version of XeTeX but I feel that this is the right way to go. Zdeněk Wagner http://hroch486.icpf.cas.cz/wagner/ http://icebearsoft.euweb.cz 2015-04-15 22:26 GMT+02:00 Arthur Reutenauer arthur.reutena...@normalesup.org: I do not understand. If system A performed function Y until now, yet its designer/maintainer wishes it to perform function Y' henceforth Khaled's point is that XeTeX did not work correctly on the issue at hand until he made the change we're discussing. It's unfair to characterise his change as a change in behaviour of the TeX engine, it was a bugfix. Arthur -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
On 15 April 2015 at 21:18, Khaled Hosny khaledho...@eglug.org wrote: On Wed, Apr 15, 2015 at 09:07:05PM +0100, David Carlisle wrote: Let me go and read some of those issues, thanks for the pointer, Just in case it is not clear why reviving the long dead TeX-XeT is a fix here; TeX-XeT does not do any actual reversal but rather outputs special opcodes in the DVI file and the DVI driver does the actual reversal, now in the driver we do not “physically” reverse the order of the nodes but instead change their X positions so that they are visually reversed. It is possible to do the X manipulation in TeX, Omega does that, but not for a mere mortal like me (and it does not help that the TeX code is complex, interwoven and written in an archaic language I only superficially understand), it was much easier to do it in the driver. Regards, Khaled Yes understood. So it seems that an ideal situation would be to change the tex--xet code to not so aggressively reorder nodes. But the number of people who could do that is limited (and doesn't include me) and it's not likely to get implemented/tested in the TL2015 timeframe So given that. I can understand your call to revert to tex--xet. I think the extra nodes in math will affect some people (and certainly causes a non trivial amount of extra work on the latex test suite) but as I'm not in a position to offer a better plan I'm not going to complain too much, I've raised the issue and it's been considered, David -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
On 15 April 2015 at 21:36, Zdenek Wagner zdenek.wag...@gmail.com wrote: Hi all, I do not want to vote but I feel that TeX behaviour was really wrong. When I type an English paragraph quoting in a middle of a sentence an Urdu text آپ جآمع مسجد کی مینار پر جا کستِ ہِں without special LTR/RTL markup, the output will be incorrect although I see it right in the text editor as well as in this mail mesage and no LTR/RTL markup is needed. I have not examined the new version of XeTeX but I feel that this is the right way to go. Zdeněk Wagner Khaled will correct me if I'm wrong, but I don't think the change affects this case, (internally the mechanism is different, but input and output is the same). David -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
On 15 April 2015 at 21:19, Philip Taylor p.tay...@rhul.ac.uk wrote: Khaled Hosny wrote: On Wed, Apr 15, 2015 at 08:52:38PM +0100, Philip Taylor wrote: Might it not be possible to resolve this by making the new behaviour optional, using a new primitive for the purpose ? So this just penalizes the unlucky people who need to enable the special behaviour, I’d rather penalize everyone and have a consistent, if erratic in someways, behaviour. I do not understand. If system A performed function Y until now, yet its designer/maintainer wishes it to perform function Y' henceforth, how can those who wish to exploit Y' be /penalised/ by being asked to invoke the operation specifically, rather than have it just happen and destroy backwards compatibility ? I see no penalty at all, just allowing the informed user to make an explicit choice between the two behaviours. Philip Taylor Phil, it's hard to argue that working right-to-left text should be an opt-in (or even opt-out) feature of xetex. I may wish for an ideal world in which this could be fixed another way, but I don't think an option would help at all. David -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
David Carlisle wrote: Khaled will correct me if I'm wrong, but I don't think the change affects this case, (internally the mechanism is different, but input and output is the same). In that case (and this is partially off-topic), what is the correct mechanism by which one should instruct XeTeX (current version : TL-2014) to reverse that stretch of text ? I have a 544pp book that goes to print on Friday, and it contains Hebrew fragments which this thread has caused me to re-examine; it would indeed seem that they are being set incorrectly, LTR instead of RTL ... XeTeX source : two verses: “foreign direction=RTL language=Hebrewחסר מכאן פסוק את אשפן וכ' ופסוק ואת הארנבת/foreign”; f.≈89r: PDF output : two verses: “ הארנבת ואת ופסוק וכ' אשפן את פסוק מכאן חסר ”; f. 89r: Philip Taylor -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
On 15 April 2015 at 20:05, Khaled Hosny khaledho...@eglug.org wrote: On Wed, Apr 15, 2015 at 04:29:47PM +0100, David Carlisle wrote: Is there no other way that the issues you have could be addressed without introducing TeX-XeT? I first reported this issue in 2008, 7 years ago, so it seems it is not going to go way magically, much to my surprise. Peter Breitenlohner became aware of this issue 2 years ago and a fix were promised but nothing so far (not that I blame him). That was the question really, is there a pointer to which issue this is fixing sorry I don't know my way round the xetex issue control that well:-) AFAICS, a port from Omega model is unlikely to happen, it is a big invasive change and is as untested as the TeX-XeT model and has its own share of problems, and, more importantly, no one seems to be stepping up to do the required work. Yes not surprised, really but still, having two incompatible direction systems causes complications for cross-engine format like latex, and having three incompatible mechanisms instead isn't exactly an ideal change. The extra nodes added here really are a backward step, my first suggestion of only conditionally adding them doesn't really work if boxes containing math are saved and used in different contexts. I fail to see that catastrophe this is causing. In general things that are incompatible between engines are extra maintenance work as we have extensive regression checks that check that things are the same. We can make xetex pass by checking in a xetex-specific result file for every test file that involves math, but that's a far from ideal situation. But that extra work is just for the latex maintainers so not so bad (for other people) but the main problem is that non-removable nodes are a major problem when manipulating TeX boxes, it's bad enough that \mathon\mathoff nodes are not removable, but that can be circumvented by forcing the math list to break, but tex-xet then adds \beginL\endL around each broken fragment which basically makes all boxes generated from math totally opaque to the macro layer. that is the cause of the 0/700 difference in the tex file I posted and in general will cause possible differences in behaviour without warning when documents move between xetex and pdftex/luatex. Especially now in the Texlive 2015 pretest period there is so little time to test any changes. Would it be possible to back it out this change at least until after TL2015 is released That would give time to investigate a more compatible way to address the issues? This was backed out for 2014 release, if I’m going to back it out for each release why bother with it at all? As I said that's the question really, what is the issue that this is fixing and is it worth adding this level of incompatibility between xetex and pdftex? Regards, Khaled Honestly I'm sorry to be sounding so ungrateful, but if I don't comment about things when I download a pretest and it causes nearly 80 of our test files to fail, I wouldn't be much use as a tester:-) David -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
On Wed, Apr 15, 2015 at 08:44:12PM +0100, David Carlisle wrote: On 15 April 2015 at 20:05, Khaled Hosny khaledho...@eglug.org wrote: On Wed, Apr 15, 2015 at 04:29:47PM +0100, David Carlisle wrote: Is there no other way that the issues you have could be addressed without introducing TeX-XeT? I first reported this issue in 2008, 7 years ago, so it seems it is not going to go way magically, much to my surprise. Peter Breitenlohner became aware of this issue 2 years ago and a fix were promised but nothing so far (not that I blame him). That was the question really, is there a pointer to which issue this is fixing sorry I don't know my way round the xetex issue control that well:-) https://sourceforge.net/p/xetex/bugs/11/ (and numerous reports elsewhere) But basically, eTeX’s TeX--XeT reverses all the nodes between \beginR/\endR unconditionally, so if you are using specials for e.g. color or hyperlinks, the “end” whatsit nodes will end up before the “start” ones, causing all sorts of breakage. The same is true for \openin/\read or \openout/\write pairs, they will be reversed and executed in the wrong order. AFAICS, a port from Omega model is unlikely to happen, it is a big invasive change and is as untested as the TeX-XeT model and has its own share of problems, and, more importantly, no one seems to be stepping up to do the required work. Yes not surprised, really but still, having two incompatible direction systems causes complications for cross-engine format like latex, and having three incompatible mechanisms instead isn't exactly an ideal change. It is either that or live with dysfunctional right-to-left support in XeTeX, it is obvious which one I’m going to chose, sorry. Regards, Khaled -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
On Wed, Apr 15, 2015 at 08:52:38PM +0100, Philip Taylor wrote: Might it not be possible to resolve this by making the new behaviour optional, using a new primitive for the purpose ? So this just penalizes the unlucky people who need to enable the special behaviour, I’d rather penalize everyone and have a consistent, if erratic in someways, behaviour. Regards, Khaled -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
It is either that or live with dysfunctional right-to-left support in XeTeX, it is obvious which one I’m going to chose, sorry. Yes sure, sometimes you just have to make choices and live with them, I understand that. Let me go and read some of those issues, thanks for the pointer, David -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
Khaled Hosny wrote: On Wed, Apr 15, 2015 at 08:52:38PM +0100, Philip Taylor wrote: Might it not be possible to resolve this by making the new behaviour optional, using a new primitive for the purpose ? So this just penalizes the unlucky people who need to enable the special behaviour, I’d rather penalize everyone and have a consistent, if erratic in someways, behaviour. I do not understand. If system A performed function Y until now, yet its designer/maintainer wishes it to perform function Y' henceforth, how can those who wish to exploit Y' be /penalised/ by being asked to invoke the operation specifically, rather than have it just happen and destroy backwards compatibility ? I see no penalty at all, just allowing the informed user to make an explicit choice between the two behaviours. Philip Taylor -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
On Wed, Apr 15, 2015 at 04:29:47PM +0100, David Carlisle wrote: Is there no other way that the issues you have could be addressed without introducing TeX-XeT? I first reported this issue in 2008, 7 years ago, so it seems it is not going to go way magically, much to my surprise. Peter Breitenlohner became aware of this issue 2 years ago and a fix were promised but nothing so far (not that I blame him). The more we look at this the more it seems a very unfortunate step, introducing several incompatibilities with pdftex and luatex. That is unfortunate. Especially if there is a possibility of moving in the end to an Omega/luatex model for bidirectional support introducing a change at this point seems a strange choice. AFAICS, a port from Omega model is unlikely to happen, it is a big invasive change and is as untested as the TeX-XeT model and has its own share of problems, and, more importantly, no one seems to be stepping up to do the required work. The extra nodes added here really are a backward step, my first suggestion of only conditionally adding them doesn't really work if boxes containing math are saved and used in different contexts. I fail to see that catastrophe this is causing. Especially now in the Texlive 2015 pretest period there is so little time to test any changes. Would it be possible to back it out this change at least until after TL2015 is released That would give time to investigate a more compatible way to address the issues? This was backed out for 2014 release, if I’m going to back it out for each release why bother with it at all? Regards, Khaled -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
Might it not be possible to resolve this by making the new behaviour optional, using a new primitive for the purpose ? Philip Taylor -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
I also support Khaled's changes to xetex; I only tested it few months ago and if the issues I reported are fixed now, then it is good. And since when LaTeX has a package/ or test files that uses \beginL \endL? All I remember is that there are only babel's rlbabel.def file and then arabicore package but these packages themselves contain bugs (irrelevant to xetex). bidi is the only big package that uses e-tex bidirectional algorithm for right to left typesetting. With the new changes of xetex, bidi package needs to be written from scrtach but I have no complain against it; the news that the \special and other issues fixed in xetex makes me so happy that I will not complain about changes. I am grateful to Khaled for fixing them. -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
In that case (and this is partially off-topic), what is the correct mechanism by which one should instruct XeTeX (current version : TL-2014) to reverse that stretch of text ? \beginR, \endR, it's in the subject of this thread. I have a 544pp book that goes to print on Friday, and it contains Hebrew fragments which this thread has caused me to re-examine; it would indeed seem that they are being set incorrectly, LTR instead of RTL ... No comment. Arthur -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
Fair enough. My thought was simply that if a sufficiently large number of users are dependent (even tacitly) on the current behaviour, then opting-in to the new should not be out of the question ... That's the eternal question, isn't it - to which extend should a buggy behaviour be preserved for backward compatibility? In this case I'm rather glad that Khaled didn't hesitate to break said compatibility. Arthur -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
On Wed, Apr 15, 2015 at 09:45:32PM +0100, David Carlisle wrote: On 15 April 2015 at 21:36, Zdenek Wagner zdenek.wag...@gmail.com wrote: Hi all, I do not want to vote but I feel that TeX behaviour was really wrong. When I type an English paragraph quoting in a middle of a sentence an Urdu text آپ جآمع مسجد کی مینار پر جا کستِ ہِں without special LTR/RTL markup, the output will be incorrect although I see it right in the text editor as well as in this mail mesage and no LTR/RTL markup is needed. I have not examined the new version of XeTeX but I feel that this is the right way to go. Zdeněk Wagner Khaled will correct me if I'm wrong, but I don't think the change affects this case, (internally the mechanism is different, but input and output is the same). That is right, though I was considering adding automatic bidirectional support the way Zdenek suggests (i.e. applying the Unicode Bidirectional algorithm to the whole paragraph as it should), but not anymore, I learnt my lesson. Regards, Khaled -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
\font\x=Arial at 12pt \x two verses:\beginR ק את אשפן וכ' ופסוק ואת הארנבת\endR; f.≈89r: \bye On 15 April 2015 at 21:54, Philip Taylor p.tay...@rhul.ac.uk wrote: David Carlisle wrote: Khaled will correct me if I'm wrong, but I don't think the change affects this case, (internally the mechanism is different, but input and output is the same). In that case (and this is partially off-topic), what is the correct mechanism by which one should instruct XeTeX (current version : TL-2014) to reverse that stretch of text ? I have a 544pp book that goes to print on Friday, and it contains Hebrew fragments which this thread has caused me to re-examine; it would indeed seem that they are being set incorrectly, LTR instead of RTL ... XeTeX source : two verses: “foreign direction=RTL language=Hebrewחסר מכאן פסוק את אשפן וכ' ופסוק ואת הארנבת/foreign”; f.≈89r: PDF output : two verses: “ הארנבת ואת ופסוק וכ' אשפן את פסוק מכאן חסר ”; f. 89r: Philip Taylor -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex -- http://dpcarlisle.blogspot.com/ -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
On Wed, Apr 15, 2015 at 09:19:04PM +0100, Philip Taylor wrote: Khaled Hosny wrote: On Wed, Apr 15, 2015 at 08:52:38PM +0100, Philip Taylor wrote: Might it not be possible to resolve this by making the new behaviour optional, using a new primitive for the purpose ? So this just penalizes the unlucky people who need to enable the special behaviour, I’d rather penalize everyone and have a consistent, if erratic in someways, behaviour. I do not understand. If system A performed function Y until now, yet its designer/maintainer wishes it to perform function Y' henceforth, how can those who wish to exploit Y' be /penalised/ by being asked to invoke the operation specifically, rather than have it just happen and destroy backwards compatibility ? I see no penalty at all, just allowing the informed user to make an explicit choice between the two behaviours. It is not possible (not with a reasonable effort anyway) to support both behaviours, so the option would boil down to having right-to-left “extension” enabled or not. Regards, Khaled -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
On 14 April 2015 at 17:47, David Carlisle d.p.carli...@gmail.com wrote:
As noted in the release notes direction support now works in math
which is a good thing but a side effect seems to be that beginL endL
nodes are added to every math list
The change introduces an incompatibility between xetex and pdftex/luatex.
the following plain tex example is obviously slightly contrived but is
not untypical of the kinds
of tests formats like latex commonly do, and shows that having
incompatible directional support
between xetex and pdftex will cause maintenance problems for latex
packages that try to offer
a unified interface.
\setbox0\vbox{\raggedright \hsize35pt$a + b$\par
\global\setbox1\lastbox
\unskip\unpenalty
\global\setbox1\lastbox
}
\setbox0\hbox{\unhbox1\unskip\xdef\tst{\the\lastpenalty}}
\tst
\bye
produces 700 in pdftex or luatex and 0 in xetex.
--
Subscriptions, Archive, and List information, etc.:
http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
On 14 April 2015 at 21:14, Khaled Hosny khaledho...@eglug.org wrote: On Tue, Apr 14, 2015 at 05:47:29PM +0100, David Carlisle wrote: As noted in the release notes direction support now works in math which is a good thing but a side effect seems to be that beginL endL nodes are added to every math list This is part of the TeX-XeT code actually and not related to the mentioned change. This is the part that ensure math is always set left-to-right even if the surrounding text is set right-to-left. I suspected as much but basically no one has used tex-xet for years as they have used the xetex variant, so this is effectively introducing a third incompatible directional system making pdftex, xetex and luatex mutually incompatible, which is a rather scary prospect for supporting cross-engine formats like latex. Would it be possible for the automatic beginL node _not_ to be added if the current context was already left to right? That should be theoretically possible, but I don’t know how. My naïve attempt below does not work (in the sense that the condition is always false even if the text was surrounded by \beginR/\endR. So if someone can come up with a working patch, I’ll happily apply it. Thanks for looking, I hope someone can, and this can be changed before texlive 2015 is frozen... David -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] additional beginL endL nodes in math
have used the xetex variant, sorry, I meant to write have used the etex variant, -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
