This meetup started a bit late (~20 min). I messed up the time zone I am in and thought that the start time is an hour later than it actually was. Luckily, we did not start full hour late, thanks to Dave Marquardt, Jeff Trull, and William Denton who followed up on Mastodon and the mailing list.
- As usual, at beginning it was Sacha's News: https://sachachua.com/blog/2024/07/2024-07-08-emacs-news/ - =Demo= reported about Emacs slowing down on large (~100k lines) Org file. The slower, the longer Emacs session is. He was not able to showcase the problem on screen as the problematic file contains sensitive info, but we still troubleshooted by voice. - He also briefly talked about his workflow and file structure, which includes 1. Heavy usage of sparse trees to see what to do next (no agenda, AFAIU) 2. Up to 6 levels nesting with subtrees representing individual projects/subprojects - My immediate reaction for "slower the longer Emacs session is" is problems with Emacs redisplay. - Emacs redisplay works by examining the buffer text between first char at the beginning of the current window and the last char - Usually, it means a very small portion of the buffer (Emacs window cannot normally fit too much text) - ... but not for Org mode - In Org mode, large chunks of text may be invisible, making Emacs redisplay engine go through the whole document (megabytes of text in Demo's example), which may be slow in some scenarios - More specifically, the redisplay scales with the number of /intervals/ in the displayed buffer range - Each /interval/ is a segment of the buffer where no single text property changes - The number of /intervals/ tends to grow over time because of how Emacs fontification works - Each time Emacs needs to apply fonts/colors in buffer, it puts a number of text properties, but only onto the text currently visible (not onto folded, invisible text) - The fontification keeps accumulating as more and more parts of the buffer gets displayed and this may cause slowing down over time if the buffer gets particularly large + fontification is particularly complex (many text properties) - If the above is what is really happening, it should be sufficient to disable fontification via M-x font-lock-mode, use a more radical M-x fundamental-mode, or an even more radical M-x clean-mode (clean-mode is only for Emacs 29+) - Demo tried this, and it turned out that the slowdown was not solved even with fundamental-mode (I forgot about clean-mode during the meetup) - So, something else was the culprit - not fontification - We then tried emacs -Q opening the same file and figured that the lags disappeared - So, as it usually goes, some pesky package / setting is creating the fuss. We sent Demo to bisect the config :) - Dave Marquardt shared his experience exporting from Org mode to docx - He used ox-pandoc: https://github.com/emacsorphanage/ox-pandoc - When exporting, he was able to use "reference document" to derive in-document styles from. Such references are extremely useful when one needs to follow specific document format without manually editing styles in the Word/LibreOffice - We went on comparing ox-pandoc with built-in ox-odt - Some people are ok with ox-odt defaults (they are reasonably good), but I also heard from other people who strongly believe that Pandoc produces more beautiful documents out of the box (without tweaking) - Aside, built-in ox-odt has quite a history - Its original author is... (to say) no longer interested to deal with FSF copyright assignment (it was quite a drama at some point) - So, he maintains his own work of built-in ox-odt with a number of additional features: https://github.com/kjambunathan/org-mode-ox-odt - He even puts it into FAQ https://github.com/kjambunathan/org-mode-ox-odt?tab=readme-ov-file#faqs : Will you merge this repo to upstream Org mode or GNU Emacs? : Never : Will you put this repo on MELPA or GNU ELPA? : Never - It is a pity that the additional features in the fork are not portable upstream, but we never force anyone to contribute or sign the paperwork. So, it is what it is. (I am not 100% sure is the problem with MELPA (or non-GNU ELPA?) though - it does not enforce copyright or licensing) - Dave Marquardt pointed one missing feature in ox-odt - it cannot export inline code blocks: #+begin_src emacs-lisp (defun org-odt-inline-src-block (_inline-src-block _contents _info) "Transcode an INLINE-SRC-BLOCK element from Org to ODT. CONTENTS holds the contents of the item. INFO is a plist holding contextual information." (error "FIXME")) #+end_src - Same for the fork - I am not very familiar wit Open Document format, but I suspect that it might be some kind of technical limitation with ODT spec. Will look into this in the future. - As a reminder, inline src blocks are blocks like src_emacs-lisp{(format "Hello %s" user-login-name)} {{{results(=Hello yantar92=)}}}, which can be right inside a paragraph, unlike src code blocks. - We had a quick question about the difference between Org mode version on GNU ELPA (bugfix branch in git repo) and the development version (main branch) - GNU ELPA contains our official releases - Apart from major releases that are announced and introduce new features, we only put simple/critical bug fixes there. So new "bugfix" versions like Org 9.7.3 -> Org 9.7.4 never contain new features and should not introduce regressions (we are quite paranoid about not introducing regressions when doing bugfix releases, even with critical fixes) - Main branch (also available from GNU ELPA devel, https://protesilaos.com/codelog/2022-05-13-emacs-elpa-devel/) is different - It is a WIP branch where we work on the next release - We try our best, but we can break things there sometimes - We also introduce experimental features, seeking for early feedback from people tracking cutting edge development - Here is a more formal description of the branches in our internal docs: https://orgmode.org/worg/org-maintenance.html#release-types - Jeff Trull shared his WIP on writing Org -> Keynote exporter - In this exporter, he wants to automatically represent Org tables as charts in the exported keynote documents - The idea is to utilize #+PLOT keyword to get hints on what to put onto the charts - His concern was that #+PLOT is designed to work with Gnuplot and Gnuplot may not have all the appropriate plot types - Then, #+PLOT semantics may not be good enough as it will not cover things not supported by Gnuplot - As an example, he stated that pie charts are not possible in Gnuplot - (technically, they are possible, but people often use hacks like https://stackoverflow.com/questions/31896718/generation-of-pie-chart-using-gnuplot) - Since Gnuplot 6 (the recent release), pie charts are, in fact, available - http://www.gnuplot.info/demo/sectors.html - In any case, #+PLOT keyword is extendable. I may contain non-standard entries as needed. See ~org-plot/preset-plot-types~. As long as general semantics is followed, having a #+PLOT value that is not recognized is ok. - I shared the ongoing work on improving org-crypt library - Daniel Clemente recently reminded about the known problem with org-crypt sometimes leaking the unencrypted data onto the disk when saving file, saving backups, or during auto-save https://list.orgmode.org/cajkahpajcy3wwbr_yixcdds9xn6rsoeea7r94tsnq7nnlof...@mail.gmail.com/ - The problem is indeed quite annoying, especially because org-crypt does provide some half-baked workarounds to deal with it - ~org-crypt-use-before-save-magic~ - Unfortunately, ~org-crypt-use-before-save-magic~ has problems: 1. It has to be called manually 2. It uses ~before-save-hook~ and Emacs removes hooks from ~before-save-hook~ if there is any error thrown while running the hook functions. So, if there is any problem with encryption, the hook gets removed (silently!), and the data starts leaking 3. ~before-save-hook~ does not get called during auto-save So, org-crypt employs ~org-crypt-disable-auto-save~ that (by default) keeps asking to disable auto-save completely (without mentioning the customization name) - fairly annoying query when one does not know that customization exists 4. When ~before-save-hook~ is set to ~'encrypt~, it only actually encrypts things before auto-save in current buffer, even when auto-save saves multiple buffers 5. The above two options for auto-encryption make the :crypt: headings disappear in Emacs buffers as well, leading to annoying scenario when pressing C-x C-s while editing :crypt: heading contents encrypts it in front of the user - I am now working on improving reliability of org-crypt to avoid unexpected data leaks and to make the UI less annoying The WIP branch is https://git.sr.ht/~yantar92/org-mode/log/feature/org-crypt-refactor Planned changes: 1. Enable ~org-crypt-use-before-save-magic~ by default 2. Avoid encryption inside Emacs buffers - even when the encrypted headings are auto-saved to disk, there is no reason to encrypt everything in buffer itself 3. Provide interactive commands to disable/enable encryption to avoid blocking saving when encryption is broken for some reason (say, GnuPG does not work) 4. Improve performance (org-crypt is quite slow on large buffers now because it scans through every single heading) 5. Improve UI when encrypting multiple headings symmetrically. The current design is to encrypt each heading separately, using a separate password - it is hardly useful to anyone. If passwords are not same, how would one go around remembering them? - The last thing I was fighting with was (5) - On the surface, it is not too hard to hook into Emacs EPG and query password only once for all the encrypted headings - However, the problem comes when there are existing symmetrically encrypted headings in the buffer. What if they are encrypted using a different password? - =emacs-user= suggested trying decrypting the already encrypted headings with a given new encryption password and if it does not work, suggest to re-encrypt using the new password - Jeff Trull asked why not to display decrypted text as an overlay and avoid all the above trickery with data leaks and whatnot - Overlaying decrypted text is all fine until one needs to actually edit that text (using normal Org mode commands!) - So, we do need to put the decrypted text into actual buffer one way or another. And we go back to the same problem with leakage, albeit less frequent. - Since leakage should not happen at all, we should solve it even with overlay approach - So, overlay idea just complicates things - =emacs-user= asked about calling function from Library of Babel from elisp - The general answer is ~org-sbe~ - However, as =emacs-user= pointed, it triggers warning when used outside Org buffer - Oh well. ~org-sbe~ is a hack (on top of ~org-babel-execute-src-block~) in reality and not a properly designed API - One day we need to implement something better - Patches welcome! At least, it is a good idea to start a discussion on the mailing list. See https://orgmode.org/manual/Feedback.html#Feedback :comments: [18:15] Welcome to <b>[[bbb:OrgMeetup]]</b>!<br /><br />For help on using BigBlueButton see these (short) <a href="https://www.bigbluebutton.org/html5"; target="_blank"><u>tutorial videos</u></a>.<br /><br />To join the audio bridge click the phone button. Use a headset to avoid causing background noise for others.<br /><br />This server is running <a href="https://docs.bigbluebutton.org/"; target="_blank"><u>BigBlueButton</u></a>. [18:16] William Denton : Hello! [18:17] Dave Marquardt : Hi [18:17] cryptk : hello [18:19] Ihor Radchenko : As usual, the latest edition of Emacs News: https://sachachua.com/blog/2024/07/2024-07-08-emacs-news/ [18:19] fnat : Hello! (Still sorting out audio on my end.) [18:19] William Denton : No problem. [18:23] Demo : Hihi! Listening only for the moment. [18:23] fnat : Same here, listening mode. I was able to work around the audio issues using a second device. [18:25] William Denton : Nothing new from me, and no problems ... I'm just lurking. [18:26] Demo : I think my only problem is i'm still suffering from slowdowns in huge files. [18:26] Demo : if you want to discuss that, i just can't show the file :P [18:26] Dave Marquardt : I recently used the Pandoc exporter to create a nice Word document, mostly avoiding using Word or LibreOffice, except for fixing a reference document. Cool stuff! [18:26] Demo : let me get mic [18:30] Dave Marquardt : :) [18:42] Ihor Radchenko : ox-pandoc? [18:42] Dave Marquardt : ox-pandoc [18:43] Ihor Radchenko : https://github.com/emacsorphanage/ox-pandoc [18:43] Dave Marquardt : Yes, I used ox-pandoc [18:44] Ihor Radchenko : https://github.com/kjambunathan/org-mode-ox-odt [18:44] Ihor Radchenko : ox-odt fork [18:45] Ihor Radchenko : from the original author [18:46] Ihor Radchenko : https://github.com/kjambunathan/org-mode-ox-odt?tab=readme-ov-file#faqs [18:46] Ihor Radchenko : Will you merge this repo to upstream Orgmode or GNU Emacs? Never [18:46] Ihor Radchenko : Will you put this repo on MELPA or GNU ELPA? Never [18:46] Dave Marquardt : Yes, I remember the drama! [18:48] Dave Marquardt : IIRC, the ODT exporter doesn't support inline code snippets, which affects me [18:48] Dave Marquardt : The original one [18:48] Dave Marquardt : src_lang{foo} [18:49] Dave Marquardt : Yeah, the FIXME error! [18:49] Demo : its very clear! [18:49] Demo : so glad to see you here too davemq ;] [18:51] Dave Marquardt : Most exporters do nice font and color stuff with inline code [18:51] Dave Marquardt : Thanks [18:52] William Denton : I need to drop out for something at the top of the hour ... thanks for organizing, Ihor, and as always, thanks for all your Org work. [18:58] fnat : Thanks Ihor and all. I also need to switch to another meeting in a few mins. This is very interesting and useful, thanks for organising it, I'll be back next time. [19:00] Jeff Trull : I have a general discussion topic about plots if that's interesting [19:02] Ihor Radchenko : https://stackoverflow.com/questions/31896718/generation-of-pie-chart-using-gnuplot [19:03] Ihor Radchenko : http://www.gnuplot.info/demo/sectors.html [19:03] Ihor Radchenko : with sectors [19:04] Ihor Radchenko : Gnuplot 6 [19:11] Jeff Trull : Actually please do share [19:11] Jeff Trull : I was reading about gnuplot 6 features and spaced out for a moment [19:13] Jeff Trull : Nice [19:23] emacs-user : why not do a (optional)decrypt-encrypt for all entries upon save? it may not be performant, but it would avoid this issue [19:24] emacs-user : yes [19:24] Jeff Trull : Could we *not* decrypt them for real, but just display an overlay with the decrypted data? [19:24] Jeff Trull : Yeah, decrypt on editing [19:25] emacs-user : doing a decrypt-encrypt for ALL entries would also allow org to prompt user when newly entered key differs from the key used in another block in the file [19:29] emacs-user : if we don't want to prompt users for key when their changes are in unencrypted regions, then it would be necessary to track whether the modified changes include a new encrypted entry or not. at that point decryption of all would need to be attempted. the other place would be when an existed encrypted entry is being modified [19:33] Jeff Trull : Not for me but as usual it's very motivating to attend thanks! [19:34] emacs-user : i joined late, but any updates on the new latex preview work? [19:34] Jeff Trull : it's very impressive work [19:35] emacs-user : thanks. so maybe 9.8 but could be later [19:35] emacs-user : heh [19:36] emacs-user : completely random, but what is the recommended way to call a function from "library of babel" via elisp? [19:38] emacs-user : org-sbe seems to provide a mechanism, however, it gives a warning unless invoked from an org-mode buffer [19:38] emacs-user : org-element (if i recall correctly) gives a warning with org-sbe [19:39] Ihor Radchenko : please post a feature request on the mailing list [19:40] emacs-user : (with-temp-buffer (org-mode) ...) seems to work, but i wasn't sure if that was recommended [19:40] emacs-user : understood, thanks [19:42] Jeff Trull : Excellent thanks for holding this meeting [19:43] emacs-user : thank you for organizing, as well as continuing to develop and maintain org-mode [19:43] cryptk : thank you! :end: -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92>
