> Can't `org-latex-preview--generic-callback' handle arbitrary converters?
> What do I miss?
It's tricky, let me explain.
In the old system previews were generated one at a time, so there was no
issue matching tex file names to image file names, and %O worked by
- taking the tex file basename
- appending ".png" or ".svg" to it as appropriate.
This does not work in the new system, because one tex file can generate
many image files. So we need some way of
1. collecting all these image files in the image-converter callback,
2. and matching latex fragments to the right image files.
How this collection and matching works depends on whether we are using a
process filter or the sentinel to process the output images, and whether
the process reports the file names of images it has generated.
----
Under the fast golden path, we collect these images as they are produced
from the process filters (not sentinels), and because the images might
be produced out of order, this requires passing the format for image
file names as an argument to the image converter.
For example: We use "%B-%%9p.svg" with dvisvgm, where %B is the basename
(interpolated by us), and "%9p" is interpreted by dvisvgm as a command
line argument, consumed by dvisvgm (not us).
Then in the process filter we notice that dvisvgm has reported that
org-tex-abcd-0000000143.svg is done, and we pick up that svg file and
display it immediately.
The crucial part is that dvisvgm/dvipng report that they have finished
processing a file and mention the file by name.
----
Under the slow fallback path (--generic-callback), this is a somewhat
easier problem to solve, with some unresolved edge cases. Here is the
logic we are using in --generic-callback:
(let ((images (file-expand-wildcards
(concat texfile-basename "*."
(plist-get extended-info :image-output-type))
'full)))
;; `images' is sorted by string<
;; just assign tex fragments to images and cache them:
(cl-loop
for tex-fragment in (plist-get extended-info :fragments)
for image-file in images
do (cache image-file tex-fragment)))
so the generic callback will work correctly so long as the following
conditions are met:
1. The image-converter generates one image per dvi/pdf page,
corresponding to one tex fragment. This means it should generate many
images for a typical run over the Org buffer/section.
2. When sorted by string<, the image file names are in the same order as
the pdf/dvi pages.
3. There are as many image files as pdf/dvi pages -- no missing image
files.
If condition 1 is not met, previews will simply not be generated for
fragments 2-# in a run.
If condition 2 is not met, previews will be jumbled, applied to the
wrong latex fragments.
If condition 3 is not met, some fragments will not have previews and
some others will have their previews jumbled in an off-by-one way.
This is actually how the imagemagick path works right now. The reason
we can't improve on it is that imagemagick does not provide image file
names that it has finished processing in its stdout. (There may be a
way to do this with CLI flags but we didn't find it when we looked a few
years ago.)
If the user wants to use some other program, say "plumber", as their
image-converter, it will work with the --generic-callback if it
satisfies the above constraints. But even if plumber does more, like
report image file names in its stdout, it is not sufficient to modify
org-latex-preview-process-alist. The user will also have to write a new
callback or process filter specifically for plumber that handles its
stdout format.
>>> However, nothing prevents users from, say, putting dvisvgm command
>>> into dvipng (exaggerated example).
>>
>> Users who mess with org-latex-preview-process-alist do so at their own
>> peril. Honestly I don't see any reason to modify it at all, except for
>> use with other libraries like ob-latex.
>
> I am concerned about users who already modified it. Their config may
> stop working after the update.
As mentioned, the primary issue is that the dvipng/dvisvgm paths require
special suffixes (like "-%9p.svg") to work correctly.
If they bring over the default image-converter command line strings for
dvipng/dvisvgm from the (now obsoleted and separate) variable
org-preview-latex-process-alist into org-latex-preview-process-alist,
everything should still work correctly because %O now maps to the
correct prefix-numbered-suffix format of "%B-%%9p.png", etc.
I changed it now to dispatch on dvipng/dvisvgm/imagemagick/<custom
processing type> instead of on the file extension (.png/.svg). If
required, this mapping can be moved into org-preview-latex-process-alist
itself.
I also tested it out by calling org-latex-preview in an Org buffer with
this configuration:
--8<---------------cut here---------------start------------->8---
#+begin_src elisp :lexical t
(cl-pushnew '(png :programs ("latex" "dvipng")
:description "dvi > png"
:message "you need to install the programs: latex and dvipng."
:image-input-type "dvi" :image-output-type "png"
:image-size-adjust (1.0 . 1.0) :latex-compiler
("latex -interaction nonstopmode -output-directory %o %f"
"latex -interaction nonstopmode -output-directory %o %f"
"latex -interaction nonstopmode -output-directory %o %f")
:image-converter ("dvipng -D %D -T tight -o %O %f")
:transparent-image-converter
("dvipng -D %D -T tight -bg Transparent -o %O %f"))
org-latex-preview-process-alist
:test #'equal)
(setq-local org-latex-preview-process-default 'png)
#+end_src
This is a test:
\[ \int_{0}^{\infty} \exp \left( - y^2 \right) \mathrm{d}y =
\frac{\sqrt{\pi}}{2} \]
--8<---------------cut here---------------end--------------->8---
and it works correctly now. You can test it at your end too. (This is
also a test for the triple latex call.)
But for the above reasons, it cannot work for arbitrary customizations!
I thought we disabled the aliasing of org-preview-latex-process-alist
to org-latex-preview-process-alist for this reason. Users have to
manually edit the latter to break things now.
>>>>>> - %O placeholder in org-latex-preview-process-alist
>>>>>
>>>>> This is yet to be discussed. Pending.
>>
>> I have now added support for %O to the image-converter commands. The
>> reason I was hesitant to add it is that the exact form depends on the
>> image output type, and it requires hard-coding the behaviors of dvipng,
>> dvisvgm and imagemagick. Your suggestion was this:
>>
>>> %O is "absolute output file name"
>>> There is nothing stopping us from
>>> doing spec =
>>> (?O . ,(shell-quote-argument
>>> (concat (expand-file-name texfile-base temporary-file-directory)
>>> "%09d.png"))
>>
>> But the suffix "%09d.png" is different for the three image converter
>> programs (dvipng and imagemagick can use the same format). I've now
>> added these special cases to org-latex-preview--create-image-async.
>> This means that, as mentioned above, it is not possible to drop in a new
>> image-converter program into org-latex-preview-process-alist and expect
>> it to work out of the box.
>
> I'd say that we should link the ?O value to the dvipng/dvisvgm program,
> not to the output type, but I think that before discussing this we
> should settle on possibility to use something other than dvipng/dvisvgm
> in the alist (above).
I've changed it to dispatch on dvisvgm/dvipng/imagemagick/<other>, as
mentioned above. But it can be moved into
org-latex-preview-process-alist if required.
>> I've also fixed one of the other remaining issues, of specifying a chain
>> of processes to org-async-call in a less awkward way. The new API is as
>> follows:
>>
>> org-async-call accepts as its PROC argument a list where the first item
>> is the symbol org-async-chain, and the rest are processes to run in
>> sequence until one of them fails. Each process can be a string or list,
>> with the meanings specified above.
>>
>> (org-async-call
>> '(org-async-chain "cmd1" "cmd2" "cmd3")
>> :success #'success-callback
>> :failure #'failure-callback
>> :buffer (get-buffer-create "*Output Buffer*"))
>>
>> This form allows for easy specification of an async linear process
>> chain. In this case the SUCCESS and BUFFER arguments apply only to the
>> final process in the chain, and the other arguments apply to all
>> processes.
>
> I think the way org-compile-file works now is that output of all the
> command in the chain are dumped into the same buffer. It is handy when
> processes throw warnings/errors - one can see problems from all commands
> in the chain.
This will break all our process filters, whose logic depend on text
parsing of the incomplete process stdout.
>> As a result the png path of org-babel-execute:latex -- which can call
>> latex thrice to generate the dvi file -- now works correctly with
>> org-latex-preview.
>>
>>
>> [...]
>>
>>
>> But in the end I implemented the more explicit
>>
>> '(org-async-chain "cmd1" "cmd2" ...)
>>
>> specification. Let me know if you have any suggestions.
>
> I like that it is explicit. [] vs () will create confusion.
Marking this issue as closed.
>> Since ob-latex now works with org-latex-preview, I want to ask if I can
>> change 'png to 'dvipng for uniformity here in ob-latex:
>>
>> (defcustom org-babel-latex-process-alist
>> `((png :programs ("latex" "dvipng") :description "dvi > png"
>> :message
>> "you need to install the programs: latex and dvipng."
>> :image-input-type "dvi" :image-output-type "png"
>> :image-size-adjust (1.0 . 1.0) :latex-compiler
>> ,(if (executable-find "latexmk")
>> '("latexmk -f -pdf -latex -interaction=nonstopmode
>> -output-directory=%o %f")
>> '("latex -interaction nonstopmode -output-directory %o %f"
>> "latex -interaction nonstopmode -output-directory %o %f"
>> "latex -interaction nonstopmode -output-directory %o %f"))
>> :image-converter ("dvipng -D %D -T tight -o %O %f")
>> :transparent-image-converter
>> ("dvipng -D %D -T tight -bg Transparent -o %O %f")))
>> ...)
>>
>> To be clear, it works as-is with 'png, but using a fallback path in
>> org-latex-preview via org-latex-preview--generic-callback.
>
> I think you can rename it, but we need to make sure that we do not break
> backwards-compatibility.
On second thought, I won't rename it in this patchset. It can be done
at any time later and is not a blocker.
>> The only remaining issues before merging are the following:
>>
>> 1. Removing caption support for LaTeX displaymath from ox-html breaks
>> use of fuzzy links in some way that I don't understand. Waiting for
>> Timothy to look into it as he made this change.
>>
>> 2. ox-odt feedback needs to be addressed.
>>
>> 3. Final cleanup tasks before merging:
>> - Rebase branch and clean up/squash some commits
>> - if-let -> if-let* etc
>> - temporary-file-directory -> (temporary-file-directory)
>> - Mark all org-latex-preview defcustom package versions as Org 10.0,
>> not Org 9.7/9.8.
>> - Remove test-org-latex-preview.el
>> - Add page breaks and section headers to org-latex-preview for
>> forward-page/outline-mode support
>> - Final byte-compile/checkdoc warning checks
>>
>> Apart from your feedback on the %O and process-chain issues from above,
>> that's everything I have in my TODO list. If I missed something please
>> let me know.
>
> The new requirement that preview.sty should be installed needs to be
> documented. (https://list.orgmode.org/orgmode/875x90ecsm.fsf@localhost/)
Done.
> Also, inlining images. The current design may interfere with future
> global inlining. (https://list.orgmode.org/orgmode/874iogkwcu.fsf@localhost/)
This does not seem actionable to me. There is no planned design for
future global inlining that I'm aware of, so I don't know what change to
make here.
Doesn't it make sense to change the code for inlining LaTeX previews
instead when global inlining is designed in ox-html in the future?
> In https://list.orgmode.org/orgmode/87qzrkje80.fsf@localhost/, I wanted
> to see a bit more information about 'live cache in the docstrings.
Done. Also updated the docstring for org-latex-preview-cache, it was wrong.
> We need to document the caveats with indirect buffers in the manual.
> (https://list.orgmode.org/orgmode/87o6moje00.fsf@localhost/)
Done.
----
Marking remaining issues in every email now, since it is getting
difficult to keep the email info in sync with my Org tasks
1. "%O" handling to minimize breakage.
2. process output handling when executing an org-async-chain. (Retain
all process stdout in one buffer?)
3. caption support/fuzzy links to captions for LaTeX fragments in
ox-html
4. ox-odt feedback
5. Final cleanup tasks before merging
6. Global image inlining in ox-html (non-actionable?)
7. `make repro' produces warnings
8. Insertion tracking is over-eager, leading to spurious LaTeX fragment
generation in programmatic editing (multiple insert calls in one
function)
Issues 7 and 8 are being discussed in
https://list.orgmode.org/87cy2ngcr0.fsf@localhost, the rest will be
addressed in this chain.
Karthik
