Dear Joris, (and texmacs-dev)
This week I have been playing with your experimental style rewriter
(src/Style, --enable-experimental), aiming at the goal you describe in
the todo list: the rewriter's output should feed the typesetter. While
comparing the two outputs pixel by pixel, I ran into the markers of the
concatenator. Fixing them properly touches the normal typesetter, so I
would like your advice before going further. And if you find the time
to look at what I did with the rewriter, I would be very grateful.
1. THE PROBLEM: MARKERS CHANGE THE LAYOUT
concater_rep::marker puts an empty text_box around every macro call,
every argument and every with. It is a place for the cursor, typed
MARKER_ITEM, in cmr at the script size. Its extents are not empty: it
goes from the baseline to the cmr x-height. So the same markup does not
typeset the same way when it is written out and when a macro produces
it. I tested this systematically: wrap pieces of a document in an
identity macro, print both versions, compare. These are the channels I
found:
- Italic corrections: italic_correct and italic_restore are not
symmetric. Restore asks the concat box again for its corrections, but
those depend on extents that italic_correct has just changed, so a box
with zero-width markers at its edges stays wider. A subscript given
through <arg> in Termes was 0.6 pt off.
- Script levels: the concat box's sub_lo_base, sup_lo_base and the other
limits take the min or max over all children, markers included, in
cmr.
* T<with|color|brown|<rsub|E>> puts the E 3 pt higher than
T<rsub|E>. This is your beamer logo <TeXmacs*>.
* <apply|cE><rsub|N> in witness.tm put the N 1.9 pt higher than the
same markup written out.
- Markers as script bases: a marker can become the base of a script. A
macro <st> for <rsup|o> at the start of a table cell is placed 1 pt
away from <rsup|o> written out.
- Accent and shape: extended_shape and top_accent (in Upsilon, not upstream)
test N(bs)==1, so a single glyph glued to markers loses its accent
position.
- Spaces at the end: with markers after the last item, its space counts
(typeset_as_concat, lazy_paragraph line_print and query).
print (space) puts a space on the last item even when that item is a
marker. kill_spaces stops at markers.
- Marker height: lines or boxes holding only markers get the x-height of
cmr. This is also the only reason cmr is loaded in documents that
don't use it.
2. WHAT I DID
- Italic corrections: italic_restore now undoes exactly what
italic_correct did; the corrections are kept, not asked for again.
- Zero extents: markers have zero extents and use the current font
instead of cmr. The cursor still gets its height from the font in
text_box_rep::find_cursor.
- Groups of boxes: a concat box leaves its markers out of its heights
and ink. It keeps them in its horizontal extents: my first version
also left them out of those, which moved equation numbers after a
tab. Its script levels, accent and shape are those of its single
non-marker box.
- Scripts: a marker is never the base of a script; a script after
markers takes the base the written-out markup would have, or none. The
italic correction goes on the item before the markers.
- Spaces: the spacing rules ignore trailing markers. kill_spaces treats
markers among the final controls as part of them.
Results: every case I cut down now typesets like its written-out form.
I compared old and new on 1690 documents (the manual, your papers and
presentations from texmacs.org, other public documents, my own papers):
- 460 documents move, mostly by less than half a point;
- 5 change page count (among them witness.tm and the Pure plugin
manual), each time through a script that now sits where the
written-out markup puts it, and the page breaks that follow from it.
3. TWO OTHER BUGS I FOUND ON THE WAY
- Format leak: format_width_rep, format_cell_rep, format_vstream_rep and
query_vstream_width_rep set ref_count= 1 in their constructors, and
make_format_* then wraps them in a handle that counts them again. They
were never freed, together with their line items. This hit every table
cell with hyphenation, including numbered section titles through
prefixed-line.
- Table width: lazy_table_rep::produce read table-width without exec,
while format_table does exec it.
4. THE REWRITER
I followed your notes ("Upcoming style rewriting tool", "Increased
laziness"):
- Partial evaluation: the rewriter evaluates the document with residuals
for whatever depends on the typesetter (lengths in units, box-info,
labels, counters seen by the typesetter).
- Input keys: the memorizers record what they read.
- Typesetter on the rewritten tree: with TEXMACS_STYLE_TYPESET=1 the
editor typesets the rewritten tree. The boxes keep the document's
places, so the cursor and typing work.
- Incremental edits: after an edit, the rewritten tree is diffed and
given to the typesetter as patches. A keystroke now costs the same as
in the normal editor.
- Results:
* all 696 English documentation pages print identically to the
normal typesetter;
* about 90 of 1000 corpus documents still differ, for reasons I
mostly understand (see the questions below).
The design and every decision I took are in src/docs/pure-style.md and
src/docs/pure-style-decisions.md on the branch.
5. QUESTIONS
1. Did you mean the markers to have extents (the cmr x-height)? For
instance, to give empty macro lines a height, or to keep the cursor
visible? Is there a reason they should affect layout that I am
missing?
2. Is it right that a marker can never be a script's base? Or did you
intend scripts to attach to the boundary of a macro?
3. Was italic_restore meant to query the corrections again?
4. Argument markers in the rewritten tree: after the rewriter
substitutes the arguments, the typesetter no longer knows where an
argument began and ended, so the cursor boundaries of arguments are
lost. I tried to recover them in the typesetter, and it made things
worse. I think the rewriter should mark substitutions itself. How did
you plan it?
5. Side effects in residual branches: for-each and counter resets inside
undefined macros (footnotes) are not visible to the typesetter. Did
you foresee state in the rewriter beyond assignments?
6. Some differences remain that I cannot explain. One is in your Beff
and lagrange notations, specific to the rewriter and not caused by
markers. I would love your opinion.
If you have time, a look at the branch would help me a lot: even only
the rewriter's design notes and the marker changes in concat_boxes.cpp,
concater.cpp, concat_post.cpp and lazy_paragraph.cpp. The branch is
https://github.com/mgubi/texmacs/tree/wip_pure_style. The marker fixes
for the normal typesetter alone are in
https://github.com/mgubi/texmacs/pull/448.
Best,
Max
_______________________________________________
Texmacs-dev mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/texmacs-dev
