Hello Simon, Thanks very much for this detailed information. Obviously (for now) there is plenty there that I don’t get, terminology-wise, but no surprise. I’ll keep this handy and work on it as I can.
I never really worked with C (I was a B programmer back in the day) and am now working on Swift (a good option for rewrite if there were to be one, on OS X anyway… though of course that would make Windows-compatability less good), but I can make do. Grew up on Pascal, though :) Deeply appreciated. Getting more of us ‘mere’ users tech/(Xe)TeX-savvy is vital to the future of the project, I think. K On 2015-04-26, 21:34, "Simon Cozens" <si...@simon-cozens.org> wrote: >On 26/04/2015 23:02, Karljurgen Feuerherm wrote: >> b) assuming a certain level of Xe(La)TeX competence at the Œpresenting¹ >> level, what recommendations would experts on this list make to Œupping >>the >> ante¹ I.e. progress toward a more insider understanding of the software? > >OK, I can answer this but I shall add a few other scattered thoughts as >well. :-) > >First I also want to thank Khaled for his fantastic work in keeping >xetex maintained, bugs fixed, and users happy. Until recently I presumed >that Khaled was a professional software and/or typography person; I >didn't realise that xetex was essentially a free time project for him. I >was even more amazed by his dedication and professionalism. > >Bit by bit, I seem to have picked up how TeX and xetex works, and I wish >I could help with the maintainership, but I just don't have the time at >present. (And given that I think the future of TeX is spelt "SILE" I >don't think it's appropriate for me to either. ;-) ) > >How to get to know xetex? I think the first step in moving from a user >to understanding the mechanics has to be Victor Eijkhout's TeX By Topic. >Either buy a hard copy or download it from >https://bitbucket.org/VictorEijkhout/tex-by-topic/src and read it over >four or five times. It's by far the best introduction to how the TeX >engine works. > >After that you should be able to work your way through the TeXBook; read >that until you can understand the double-arrow sections. > >>From there, there are two directions you need to go in: the WEB program >for TeX, and the xetex extensions and all the related font handling code >and libraries that it uses. > >As Joseph and Phillip have mentioned, WEB is not an easy thing to work >with, and WEB2C doesn't make it any better. But in a way there's nothing >you can do about that; TeX is the WEB source. A lot of the design >constraints of TeX in the early 80s don't apply any more; most of the >unpleasantness around WEB comes from the fact that memory is allocated >statically and that structures are hand-rolled with pointers and >offsets. Rewriting the whole thing in another language wouldn't be a >crazy idea (I've done it) and for long-term maintainability I think it's >essential - we can't go on with statically-allocated PASCAL code for >ever - but it would be a major operation outside of the bounds of >maintaining the current *TeX projects. > >But the up side of that is that there's very little of the WEB code that >you actually need to mess with. Most of it Just Works and is never going >to need to change, and most of the time you can assume that if there's >an problem, it's with the xetex-specific bits, rather than with Knuth's >TeX. > >So after you have a conceptual understanding of how TeX works, the next >step is to run weave on source/texk/web2c/xetexdir/xetex.web [1] and >start reading. You can skim over parts 1-19, read the rest normally, and >focus most of your attention on parts 37-46. In particular, you want to >read over the parts which deal with native word nodes, which are >(basically) hboxes containing native font characters. Look up >"native_word_node" in the index at the back and read those sections. > >Many of the XeTeX extensions call out from Pascal into C; these are >defined in the xetex.defines file. This is a bit tricky to match up >because WEB2C (I think) strips the underscores from the names in the WEB >file. So set_native_metrics in xetex.web gets turned into >setnativemetrics, which is defined by xetex.h as measure_native_node, >which you will find defined in XeTeX_ext.c - this is the key function >which, takes a Pascal memory region representing a native word node (a >bunch of Unicode characters), calls the font shaping functions on it, >and fills in the height, width, and depth of that node back into Pascal >so TeX can run its algorithms on it. Start your exploration of the C >sources from that routine, and follow all the function calls until you >understand what it's doing. At some point you will follow it down to the >harfbuzz interface in XeTeXLayoutInterface.cpp and the FontConfig >interface in XeTeXFontMgr_FC.cpp. (My feeling is that the >AAT/Mac-specific stuff is dead now, and at any rate it's easier to >understand FontConfig/harfbuzz anyway.) > >Once you get to that layer, you may be perplexed by the lack of >documentation for both harfbuzz and FontConfig. Hopefully my article at >http://www.simon-cozens.org/content/duffers-guide-fontconfig-and-harfbuzz >might >help with this. > >Finally, about the future of TeX. Obviously my view that a complete >rewrite is a good idea is going to be a minority report for a while yet, >so I'll stick to XeTeX and LuaTeX. > >I don't know as much as I should about LuaTeX. For me, the point of >xetex is not just that it's a Unicode-compatible TeX, but also that it >supports native fonts well and that it both handles native OS fonts in a >simple way and supports shaping of complex scripts (the harfbuzz bit). >Hans has stated that LuaTeX will not include external font shaping in >the core, and Graham Douglas has done some experiments in calling out to >harfbuzz in Lua for shaping, but I don't think it's a done thing yet. > >So if I want to use complex scripts and native fonts, (and I do) then >right now xetex is where it's at for me. > >But in a way that's beside the point. I've come to thinking that the >future of a software project has less to do with its features than with >the size and dedication of its community. If xetex (or luatex) doesn't >have a good core of people who can improve, debug and maintain it, then >no matter how good it is, it isn't going to be the future of TeX. > >So I hope we can make that happen by building more understanding of the >XeTeX internals, and I'm happy to mentor anyone who wants to understand >the code better. If you have any questions about the code, ask on this >list and I will try to answer. > >[1] Giving up on the change file idea was one of the smartest moves the >xetex people made... Incidentally I attach weave-html.pl, which I >thought was a standard TeX thing but turns out to be something I cooked >up myself. It turns WEB files into pretty linked hypertext. -------------------------------------------------- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex