Re: [XeTeX] The future of XeTeX
On 12-07-30 21:00, Zdenek Wagner wrote: The question is: is it easier to find a volunteer for maintaining XeTeX or a volunteer who will implement missing features for luaotfload and port important packages as Polyglossia to luatex? It is definitely easier to replace the ICU Layout integration in XeTeX with HarfBuzz integration, and to tweak the XeTeX code to potentially remove the native Mac OS X font engine integration (or at least make it optional) than to write full-scale OpenType Layout shapers in Lua from scratch. HarfBuzz is still not perfect but it definitely developing towards becoming the only sensible OpenType Layout engine within the open-source realm. Developing some of the shapers, especially for Indic languages, is not trivial per se (it is taking the HarfBuzz developers quite a long time, and it is after all an intensive effort). Doing everything from scratch in Lua will be a similarly long-winded effort: Lua is not a very popular language, and even if an complete OTL solution is created in Lua some day, it will need maintenance. The great thing about the XeTeX concept is that it doesn't try to do everything itself. It uses the best available components to do the heavy-lifting. When Jonathan Kew created it, ICU Layout was the best OpenType Layout library available. Soon, HarfBuzz will be the best one, guaranteed to be maintained. As I've written on this list previously, integrating HarfBuzz into XeTeX (as an alternative to the existing engines, i.e. ICU Layout, Graphite 1 and ATSUI) would be very desirable. I think such a project could be funded by some of the existing TeX groups. Best, Adam -- May success attend your efforts, -- Adam Twardoch (Remove list. from e-mail address to contact me directly.) -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] The future of XeTeX
2012/7/30 Philip TAYLOR p.tay...@rhul.ac.uk: case. But differences at the syntactic level are a far greater concern : I think one should accept that if one passes an extant XeTeX source through LuaTeX, line and page breaks may well differ, but if LuaTeX barfs on valid XeTeX source, that is (for me, at least) a far greater concern (and a reason against adoption, to be honest). That's why you should use ConTeXt or LaTeX, which will isolate your document from differences in the engine. Best Martin -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] The future of XeTeX
2012/7/31 Martin Schröder mar...@oneiros.de: 2012/7/30 Philip TAYLOR p.tay...@rhul.ac.uk: case. But differences at the syntactic level are a far greater concern : I think one should accept that if one passes an extant XeTeX source through LuaTeX, line and page breaks may well differ, but if LuaTeX barfs on valid XeTeX source, that is (for me, at least) a far greater concern (and a reason against adoption, to be honest). That's why you should use ConTeXt or LaTeX, which will isolate your document from differences in the engine. Not always, Examples: 1. Polyglossia (XeLaTeX only) 2. Accented characters in PDF outlines (works fine in XeLaTeX, difficult to do in LaTeX) 3. Input file in UTF-8 (no problem in XeLaTeX and LuaLaTeX, difficult to do in LaTeX - the inputenc package sets the \catcode of the first part of a character to 13, therefore \futurelet\nextchar\dosomething fails, placing a macro requesting a single parameter to \everypar for geting and positioning the first character of a paragraph in a different font will also fail; enxTeX does not have these problems but may cause other problems elsewhere) 4. Setting PDF version: does XeTeX support \pdfminorversion and does it send a proper command to the xdv-pdf driver? I have not found such feature in the dvipdfmx manual. Best Martin -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex -- Zdeněk Wagner http://hroch486.icpf.cas.cz/wagner/ http://icebearsoft.euweb.cz -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] The future of XeTeX
Adam Twardoch (List) wrote: I believe that the TeX world needs a policy on naming engine-specific commands. This is akin to the CSS browser-specific prefixes, such as -webkit-text-security or -moz-font-features. XeTeX already does this: all the XeTeX-specific commands are prefixed with XeTeX. Some of those commands are of general use (in such case, the XeTeX and LuaTeX developers should communicate to standardize a new command) while others may really not be of general use at all, as they're more-less hacks or ways to achieve certain effects in a way tied very much to the specific implementation of the engine. I agree. And I also think (although this is a spur-of-the-moment idea, and I may regret it tomorrow) that perhaps we also need a TeX Council, to co-ordinate naming conventions across disparate TeX-based developments. ** Phil. -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] The future of XeTeX
Am 31.07.2012 um 00:40 schrieb Zdenek Wagner: 4. Setting PDF version: does XeTeX support \pdfminorversion and does it send a proper command to the xdv-pdf driver? I have not found such feature in the dvipdfmx manual. XeTeX cannot because it produces XDV, but xdvipdfmx (invoke it with --help) can. Within limits. -- Greetings Pete The day Microsoft makes something that doesn't suck is the day they start selling vacuum cleaners. – Ernest Jan Plugge -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] The future of XeTeX
2012/7/31 Peter Dyballa peter_dyba...@web.de:
Am 31.07.2012 um 00:40 schrieb Zdenek Wagner:
4. Setting PDF version: does XeTeX support \pdfminorversion and does
it send a proper command to the xdv-pdf driver? I have not found such
feature in the dvipdfmx manual.
XeTeX cannot because it produces XDV, but xdvipdfmx (invoke it with --help)
can. Within limits.
Yes, I know. Similarly as XeTeX can set \pdfpageheight and
\pdfpagewidth (or use \special{papersize=...}) there might be a
similar \special for setting PDF version and compression but such
\special's do not exist. It is more difficult to create PDF/X
compliant file because I have to create XDV and then specify the PDF
version when calling xdvipdfmx but pdftex can set it directly. The
same problem appears with LaTeX + dvips. These are cases when LaTeX
does not help, usage is engine specific.
--
Greetings
Pete
The day Microsoft makes something that doesn't suck is the day they start
selling vacuum cleaners.
- Ernest Jan Plugge
--
Subscriptions, Archive, and List information, etc.:
http://tug.org/mailman/listinfo/xetex
--
Zdeněk Wagner
http://hroch486.icpf.cas.cz/wagner/
http://icebearsoft.euweb.cz
--
Subscriptions, Archive, and List information, etc.:
http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] The future of XeTeX
On 31/07/2012 05:05, Adam Twardoch (List) wrote: As I've written on this list previously, integrating HarfBuzz into XeTeX (as an alternative to the existing engines, i.e. ICU Layout, Graphite 1 and ATSUI) would be very desirable. I have been looking at XeTeX guts (after the discussion of shifting from ATSUI to Core Text), and also Harfbuzz guts for another typesetting project, (still under wraps at the moment but more details soon!) and would be very happy to give some time to make this happen. Unfortunately this month is insane for me but I should have some time in September/October. -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] The future of XeTeX
Martin Schröder wrote: That's why you should use ConTeXt or LaTeX, which will isolate your document from differences in the engine. Yes, and I should drive an automatic so I don't have to worry about which gear I should be in. In fact, I should employ a chauffeur, so I don't have to worry about driving at all. But I /enjoy/ driving, and I /enjoy/ using my gearbox; in fact, I enjoy taking full responsibility for everything I do with my car (short of servicing it, which I leave to the experts). And I feel exactly the same about TeX. So, thanks but no thanks : canned solutions may be fine for some, perhaps even for most; but for some of us, finding our own solutions is a far far far more rewarding way of life. ** Phil. -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] The future of XeTeX
On Tue, Jul 31, 2012 at 12:57:16AM +0200, Peter Dyballa wrote: Am 31.07.2012 um 00:40 schrieb Zdenek Wagner: 4. Setting PDF version: does XeTeX support \pdfminorversion and does it send a proper command to the xdv-pdf driver? I have not found such feature in the dvipdfmx manual. XeTeX cannot because it produces XDV, but xdvipdfmx (invoke it with --help) can. Within limits. It would be as easy as adding \specials, both for the XeTeX side and for xdvipdfmx. Also settings for compression level, object stream compression, ... would be useful. Example to generate uncompressed PDF: xe(la)tex --output-driver=xdvipdfmx -V4 -z0 But the most annyouning bug is the broken/crappy/unspecified interface to transfer binary bytes to PDF output structures. Yours sincerely Heiko Oberdiek -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] The future of XeTeX
On Mon, Jul 30, 2012 at 05:15:11PM +0200, Dominik Wujastyk wrote: Not long ago, Jonathan noted about a particular problem that, The best chance of fixing this will be if xetex is updated some day to use the new harfbuzz engine, but there's no immediate prospect of that, I'm afraid. Like most of us on this list, I use XeTeX, value it enormously, and depend on it for day-to-day work. But it's becoming clear to me that the long-term maintenance of XeTeX is not assured. I’m tentatively maintaining XeTeX. TUG is supporting me this year to continue working on completing the OpenType math engine, look at the occasional bug report, and consider merging xdvipdfmx into dvipdfmx. I also intend to look into HarfBuzz integration, but no concrete plans yet. Jonathan, though not directly maintaining the code, is still “supervising” the development activity in some sort. Regards, Khaled -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] The future of XeTeX
On Mon, Jul 30, 2012 at 02:51:12PM -0400, William Adams wrote: On Jul 30, 2012, at 11:35 AM, Richard Koch wrote: This worries me as well, since A LOT of people use XeTeX for production level work. On the Macintosh, XeTeX uses a deprecated font library which Apple never converted to 64 bits. The latest documentation from Apple says that this library will be removed entirely in a future system. That's scary for such a central component of the TeX distribution. I guess AAT/ATSUI is only used even when not using the xdv2pdf back-end to load fonts --- wouldn't one be able to compile it for Mac OS X using the settings / options for Linux so that while one wouldn't be able to easily access system fonts, one would be able to access fonts by file reference, or to use those fonts which are available to xquartz? ATSUI is used by the engine on mac to support AAT fonts, also corresponding services are used for finding system fonts. It is possible to disable all Mac specific code and build XeTeX as if it was another Unix (I think MacPorts does this, on 64 bit at least), but that might not be acceptable for every one. Regards, Khaled -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] The future of XeTeX
On Mon, Jul 30, 2012 at 10:40:17PM +0100, Philip TAYLOR wrote: Adam Twardoch (List) wrote: * Even with the same core set of commands, if using OpenType fonts, the results between LuaTeX and XeTeX will necessarily vary. LuaTeX and XeTeX use different mechanisms when it comes to extracting glyph metrics, kerning, other positioning commands, and also different mechanisms when it comes to processing things like OpenType contextual alternates etc. -- and by using different mechanisms, it by necessity arrives at results that differ slightly. No known OpenType Layout engine out there (Microsoft Uniscribe, Monotype WorldType, Bitstream Panorama, Adobe World composer, ICU Layout, HarfBuzz, Pango, or the LuaTeX engine) is 100% compatible with any other, so the same line, or even word, may be typeset slightly differently with each of those layout engines. This will, in the end, necessarily result in different glyphs being used at times, different line-breaking being generated etc. When it comes to Unicode and OpenType, it's much more complex than the original 8-bit Western world, and cross-platform compatibility is no longer a goal that can be achieved at this time. I'd say the situation is similar to the world of web browsers: HTML, CSS and JavaScript are being actively developed, but some snapshots of the development are strictly documented by the W3C, yet other factors come into play so that a 100% pixel compatibility between Mozilla Firefox, WebKit (Chrome or Safari), Microsoft Internet Explorer and Opera is not achievable, and probably never will be. OK, thank you Adam. I think perhaps I was being unrealistic in asking whether the two PDFs would be visually identical; for the very reasons you adduce, it is clear that this can never be the case. But differences at the syntactic level are a far greater concern : I think one should accept that if one passes an extant XeTeX source through LuaTeX, line and page breaks may well differ, but if LuaTeX barfs on valid XeTeX source, that is (for me, at least) a far greater concern (and a reason against adoption, to be honest). The extended XeTeX \font syntax is supported by luaotfload (which can be loaded under plain as well, it has no LaTeX dependency), but other XeTeX specific features might not (e.g. the graphics syntax is pdftex like), but I imagine for most part that a compatibility layer of some sort can be written. Regards, Khaled -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] The future of XeTeX
On Mon, Jul 30, 2012 at 01:03:01PM -0400, Behdad Esfahbod wrote: On 07/30/2012 12:56 PM, Shriramana Sharma wrote: CC-ing the developer of HarfBuzz in case he's not on this list. On Mon, Jul 30, 2012 at 9:05 PM, Richard Koch k...@math.uoregon.edu wrote: On the Macintosh, XeTeX uses a deprecated font library which Apple never converted to 64 bits. Does HB support the latest Macintosh systems? Would porting XeTeX to HB solve the above problem? Yes, it will. Alternatively you can wait till someone (me?) ports iculayout to HarfBuzz. Then someone needs to adjust Jonathan's custom version of iculayout to the HarfBuzz version... Does this mean HarfBuzz will support AAT fonts (even if only on Mac through CoreText)? Because if not, we will still need to port XeTeX from ATSUI to CoreText (as that is deprecated API being referred to) and no body who knows how to do this seems interested. Regards, Khaled -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] The future of XeTeX
On 07/30/2012 08:23 PM, Khaled Hosny wrote: Does this mean HarfBuzz will support AAT fonts (even if only on Mac through CoreText)? Because if not, we will still need to port XeTeX from ATSUI to CoreText (as that is deprecated API being referred to) and no body who knows how to do this seems interested. The CoreText backend in HarfBuzz does that indeed. behdad -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] The future of XeTeX
On Tue, Jul 31, 2012 at 1:35 AM, Adam Twardoch (List) list.a...@twardoch.com wrote: optional) than to write full-scale OpenType Layout shapers in Lua from scratch. ... which would still leave people who are heavily banking on Graphite for their work into old Indic scripts (which are unlikely to be supported [well] in any OT implementation) high and dry. The probability of Graphite support being written in Lua is much lesser than a successful support for OT. (Do I understand correctly that the Lua code has to itself parse the font tables? No external library linkups allowed? So simple bindings to the SIL-distributed Graphite library will not be possible?) HarfBuzz is still not perfect but it definitely developing towards becoming the only sensible OpenType Layout engine within the open-source realm. While HB has its own OT shapers, it also supports the Gr2 library so for people like me HB support is the only answer. HB also supports the Uniscribe backend (for those who want it) and from Behdad's recent comment, the latest Mac backend too. So it's not only YAOTI (yet another OpenType implementation) but a global solution -- kudos to Behdad and Jonathan and others who are working on it! Developing some of the shapers, especially for Indic languages, is not trivial per se (it is taking the HarfBuzz developers quite a long time, and it is after all an intensive effort). Doing everything from scratch in Lua will be a similarly long-winded effort: Lua is not a very popular language, and even if an complete OTL solution is created in Lua some day, it will need maintenance. Hm -- over on the ConTeXT list I wrote Please don't tell me you are writing yet another OT engine, and I got the semi-jocular (?) reply OK we don't tell you! :-) Frankly as a Indian, and a Sanskrit/Vedic scholar, my use cases (esp complicated svara-markup in Vedic) test the limit of rendering engines, and I really don't want to keep submitting bug reports for each and every open-source OT engine (doesn't that in some way go against the concept of open-source by taking out the collaboration aspect?) -- so switching to ConTeXT mkiv which supports only LuaTeX is NOT an option for me. The great thing about the XeTeX concept is that it doesn't try to do everything itself. It uses the best available components to do the heavy-lifting. And that's what is in line with the Unix philosophy write? Code-reuse? I'd have been happy if ConTeXT with its provisions for fine typographic control would have continued to support XeTeX in its further development, but well the developers have decided to make it monolithic and based on Lua, so that's their call. -- Shriramana Sharma -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] The future of XeTeX
On Mon, Jul 30, 2012 at 08:27:46PM -0400, Behdad Esfahbod wrote: On 07/30/2012 08:23 PM, Khaled Hosny wrote: Does this mean HarfBuzz will support AAT fonts (even if only on Mac through CoreText)? Because if not, we will still need to port XeTeX from ATSUI to CoreText (as that is deprecated API being referred to) and no body who knows how to do this seems interested. The CoreText backend in HarfBuzz does that indeed. I've been under the impression that Uniscribe and CoreText backends were merely for testing, but if they are indented for production use that would help simplifying things greatly. Regards, Khaled -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] The future of XeTeX
Martin Schröder wrote: You are right. Do a 'texdoc xetex' and 'texdoc luatex' and compare. :-) E.g. polyglossia does not work with LuaTeX (yet). OK, fears concerned. But of course polyglossia is a macro package; do you happen to know what is different at the TeX/XeTeX/LuaTeX primitive level, which from my perspective is a far more important consideration ? ** Phil. -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] The future of XeTeX
On Tue, Jul 31, 2012 at 06:04:13AM +0530, Shriramana Sharma wrote: The great thing about the XeTeX concept is that it doesn't try to do everything itself. It uses the best available components to do the heavy-lifting. And that's what is in line with the Unix philosophy write? Code-reuse? I'd have been happy if ConTeXT with its provisions for fine typographic control would have continued to support XeTeX in its further development, but well the developers have decided to make it monolithic and based on Lua, so that's their call. LuaTeX was developed for ConTeXt (it was Hans idea, even the very choice of Lua was because he was using it somewhere else and liked it), so no surprises here. If the kind of stuff MkIV is doing was possible without LuaTeX, they wouldn't have started it. That being said, it is not impossible to integrate HarfBuzz and Graphite into LuaTeX (it can use external binary modules), but it is unlikely to be part of LuaTeX proper because they want as few external dependencies as possible (and XeTeX has shown that external dependencies are not without cost). I feel you frustration, I'd like see everyone move to HarfBuzz (even non-free software), and have as less incompatibilities as possible (it is a nightmare to develop a moderately complex OpenType font and get it to work the same everywhere), but different projects have different philosophies and one has to understand this. Regards, Khaled -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] The future of XeTeX
Joel C. Salomon wrote: I know for a fact that LuaTeX enables features of OpenType Math fonts that XeTeX does not. See, e.g., https://github.com/wspr/unicode-math/issues/40. And, of course, there are XeTeX-specific features (most prominently interchartoks) which LuaTeX does not implement. Avoiding math and engine-specific extensions, I personally have not *noticed* any differences between the outputs, but given the differences that exist I doubt anyone would care to guarantee identical results. OK, but how about things such as graphics ? I already know that I cannot use (e.g.,) \pdfximage, \pdfrefximage and \pdflastximage in XeTeX, and have to use instead \XeTeXpicfile; if I were to migrate to LuaTeX, would I be correct in expecting to find a whole new set of graphics primitives ? And what of \XeTeX's extensions to the \font primitive ? ** Phil. -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
Re: [XeTeX] The future of XeTeX
Hi Phil, On 31/07/2012, at 7:35 AM, Philip TAYLOR wrote: OK, but how about things such as graphics ? I already know that I cannot use (e.g.,) \pdfximage, \pdfrefximage and \pdflastximage in XeTeX, and have to use instead \XeTeXpicfile; if I were to migrate to LuaTeX, would I be correct in expecting to find a whole new set of graphics primitives ? I know you don't like using LaTeX, but really people have put a huge amount of effort into that, and its packages. When I occasionally still use Plain TeX, almost always I make use of: \input miniltx.tex This loads a *very limited* subset of LaTeX, but enough to allow \usepackage to work. Some LaTeX packages have been written in TeX, so as to not depend on other parts of LaTeX for loading and doing their stuff. This includes graphics and graphicx, and perhaps color too. (Xy-pic was written this way too!) So you can make use of the \includegraphics command, along with its options and driver support, from within your Plain TeX documents. This then allows you to just use them with all their associated power and effects, without having to worry about how it all works ... OR ... turn on some \tracingall to find out just what primitives are being constructed, to fully satiate your ever-inquiring mind. And what of \XeTeX's extensions to the \font primitive ? Isn't there some documentation about that? ** Phil. Cheers, Ross Ross Moore ross.mo...@mq.edu.au Mathematics Department office: E7A-419 Macquarie University tel: +61 (0)2 9850 8955 Sydney, Australia 2109 fax: +61 (0)2 9850 8114 -- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex
