Re: [PATCH] Highlight ANSI sequences in the whole buffer (was [PATCH] ANSI color on example blocks and fixed width elements)
Ihor Radchenko writes: >> Some comments about the patch, as it currently stands, follow. >> >> - I've introduced two text properties `org-ansi' and >> `org-ansi-context'. >> >> The first is placed on the regions that actually contain ANSI >> sequences and holds information about the sequence that is useful to >> keep around to detect when a sequence has been modified or deleted >> between fontification cycles, as well as information about whether >> or not a sequence should be revealed due to modifications or because >> of visibility changes. > > Let's drop the part with modifications/visibility changes. It should not > be a job for fontification function, so let's not complicate things (as > I mentioned above). I believe that 'org-ansi may no longer be needed > once we drop this. > You want me to remove the code that is responsible for keeping the sequences visible according to `org-fold-show-set-visibility` and according to whether or not the sequence is currently being edited then? What should I do instead? If I remove the code for the modification changes, then it would be difficult to edit the sequences when `org-ansi-hide-sequences` is t since they would remain invisible while editing. -- Nathaniel
Re: [PATCH] Highlight ANSI sequences in the whole buffer (was [PATCH] ANSI color on example blocks and fixed width elements)
Ihor Radchenko writes: > Thanks for the update! And for the resilience working on this difficult > patch :) It has been giving me some trouble :) > I believe that `font-lock-extend-region-functions' should be able to > tackle this situation. Is there any problem with it? I think I have been able to come up with a solution using it. -- Nathaniel
Re: [PATCH] Highlight ANSI sequences in the whole buffer (was [PATCH] ANSI color on example blocks and fixed width elements)
Ihor Radchenko writes: > Ihor Radchenko writes: > >> Nathaniel Nicandro writes: >> >>> Feedback appreciated! >> >> Thanks for the update! >> ... >>> I've finally implemented a solution to what I've discussed previously, >> ... > > It has been a while since the last update in this thread. > Nathaniel, may I know if you are still working on this? Hello Ihor, Yes I'm still working on this. Attached is an updated patch with some tests this time. It's still a work in progress. Below are responses to your previous comments about my last update and some comments about this current patch. > This is very fragile. > I believe that hooking into `org-fold-check-before-invisible-edit' > would lead to simpler implementation. Thank you for the feedback. I indeed was able to come up with a more simpler solution by hooking into that function. To integrate with `org-fold-check-before-invisible-edit' I had to introduce two variables, `org-fold-visibility-detail' which is set to the argument of `org-fold-show-set-visibility' when that function is called and `org-ansi-fontify-begin' to determine the start of the fontification region to see if it's close to the beginning of an invisible sequence that should be turned visible. Let me know if this is an OK approach. I ran into an issue when trying to hook into `org-fold-check-before-invisible-edit' in that when it revealed a sequence at the end of a line, there would be an extra fontification cycle that would occur after the reveal which would cause the sequence to be re-hidden again. To counteract this I had to use `buffer-chars-modified-tick' in the way I do. I couldn't figure out why redisplay was causing that extra fontification cycle when there were no modifications to the buffer. > 1. Open the file and move to the end of the headline "Greater elements" > 2. > 3. Observe fontification extending past the title. This is fixed. I think it was due to specifying the contents-end position as the end of the region to highlight instead of the line-end-position for headlines. > I also edited it around in various places and I managed to trigger > parser errors when the parser lost track of the modifications. This > was presumably because your patch edited the buffer. I no longer make edits to the buffer. The ANSI sequences are no longer accompanied by the zero width spaces from the idea that I had before. With this patch, editing around sequences should be more stable and non-surprising. Basically if a sequence is invisible around point and you edit it, the sequence remains visible. It is only after the first edit outside of a sequence that should make the sequence invisible. Whenever a sequence is being edited, it should always be visible and not turn invisible while in the middle of editing it, e.g. due to an invalid sequence turning valid. Some comments about the patch, as it currently stands, follow. - I've introduced two text properties `org-ansi' and `org-ansi-context'. The first is placed on the regions that actually contain ANSI sequences and holds information about the sequence that is useful to keep around to detect when a sequence has been modified or deleted between fontification cycles, as well as information about whether or not a sequence should be revealed due to modifications or because of visibility changes. The second property holds the ANSI context, as defined by `ansi-color-context-region', for regions that actually have been highlighted or processed by `org-ansi-process-region'. Storing the ANSI context is done so that on fontifying some new region, the context that should be used can be determined simply by examining the property on an appropriate region before the start of the fontification. The property is also used to determine the extent of a context or sequence, how far forward into the buffer its effects last. The extent of a context is useful for extending the region being fontified to include the extent of a sequence which has been modified or deleted between fontification cycles. Currently I only extend the fontification region to include the extent when there has been a deletion or modification of a sequence in the region up for fontification (`org-ansi-extend-region'). I've not found a way to extend the fontification to a region including the full extent of a newly inserted sequence, in such cases the code as it stands now will fontify past the limit of fontification to the end of the element. - The `org-ansi-process-*' functions boil down to calls to `org-ansi-process-region' which does the actual highlighting and bookkeeping of text properties on the regions. Each of the process functions are just aware of the varying types of element structure in an Org document. They are supposed to process an element's region from point to some lim
Re: [PATCH] Highlight ANSI sequences in the whole buffer (was [PATCH] ANSI color on example blocks and fixed width elements)
Ihor Radchenko writes: > Nathaniel Nicandro writes: Hello, I've finally implemented a solution to what I've discussed previously, inserting zero width spaces as boundary characters after an ANSI sequence to act as a separator from the text after the sequence. This would handle the scenario where deleting into the end byte of a sequence causes ansi-color to recognize the partially deleted sequence plus the character directly after the end byte to be a new sequence. This looked like the invisible region containing a sequence eating up other characters not intended to be part of the region. So for example, suppose you had a control sequence, ^[[42m, where m is the end byte that says the sequence is a color sequence. Let point be signified by *. If we have ^[[42m*text then deletion into the end byte would result in ^[[42*text t is still a valid end byte so the fontification process will recognized the whole thing as a valid sequence still and the t would then become part of the invisible region containing the sequence. To avoid this from happening I have introduced the rule that any valid sequence shall have a zero width space immediately after it and this space remains in the buffer even on deleting into it with, for example, backward-delete-char. Let the zero width space be signified by |. If we have ^[[42m|*text then deletion into the space would now result in ^[[42*|text i.e., the effect is that the deletion went past the space, leaving it alone, and deleted the end byte of the control sequence. Since the control sequence is no longer valid, due to the space being at the position of the end byte, it becomes visible. If you then insert a valid end byte, e.g. m, then the effect is ^[[42m|*text i.e., point moved past the space character. So the implementation of that rule of maintaining a zero width space after valid sequences and the rules around deleting into the space or insertion in front of a space are the main changes in this patch compared to previous versions. > > I tried to test your newest patch with the example file you provided and > I notice two things that would be nice: > > 1. It is a bit confusing to understand why one or other text is colored >without seeing the escape characters. Some customization like >`org-link-descriptive' and a command like `org-toggle-link-display' >would be nice. I can see some users prefer seeing the escape codes. I've gone ahead and implemented the toggling of the visibility of the escapes sequences. The variable is `org-ansi-hide-sequences` and the function is `org-toggle-ansi-display`. I just used buffer-invisibility-spec for this. > > 2. Using overlays for fontification is problematic. In your example >file, table alignment becomes broken when escape sequences are hidden >inside overlays: > >| [31mcell 1 | cell 2 | >| cell 3 | cell 4 | > >looks like > >| cell 1 | cell 2 | >| cell 3 | cell 4 | > >Using text properties would make table alignment work without >adjustments in the org-table.el code. > I've gone ahead and used text properties instead of overlays. >> One thing I would like to start doing is writing some tests for this >> feature. It would be great if someone could point me to some tests >> that I can peruse so that I can get an idea of how I can go about >> writing some of my own. Also, are there any procedures or things I >> should be aware of when trying to write my own tests? > > Check out testing/README file in the Org repository. > > Unfortunately, we do not yet have any existing tests for font-locking in > Org tests. You may still refer to the files in testing/lisp/ to see some > example tests. > > Also, Emacs has built-in library to help writing font-lock tests - > faceup.el. You may consider using it. Its top comment also contains a > number of references to various tools that could be useful to diagnose > font-locking code. I have not looked into testing this feature yet. Feedback appreciated! >From ea2345ab218d3bc9c07452b2171afc1361b74b9d Mon Sep 17 00:00:00 2001 From: Nathaniel Nicandro Date: Tue, 9 May 2023 19:58:11 -0500 Subject: [PATCH] Highlight ANSI escape sequences * etc/ORG-NEWS: Describe the new feature. * lisp/org.el (org-fontify-ansi-sequences): New customization variable and function which does the work of fontifying the sequences. (org-ansi-hide-sequences) (org-ansi-highlightable-elements) (org-ansi-highlightable-objects): New customization variables. (org-ansi--before-command, org-ansi--after-command) (org-ansi--before-control-seq-deletion) (org-ansi--after-control-seq-deletion) (org-ansi-zero-width-space, org-ansi-is-zero-width-space) (org-ansi-new-context, org-ansi-process-region) (org-ansi-process-block, org-ansi-process-paragraph) (org-ansi-process-fixed-width) (org-fontify-an
Re: [PATCH] Handle block-type when checking `org-src-fontify-natively'
Ihor Radchenko writes: > Nathaniel, is there any particular problem you encountered that your > patch was aiming to solve? If no, we should probably re-add fontifying > example blocks as undocumented feature people rely on in practice. No there wasn't any particular problem, I was just browsing the code and saw that it was undocumented behavior. -- Nathaniel
Re: [PATCH] Highlight ANSI sequences in the whole buffer (was [PATCH] ANSI color on example blocks and fixed width elements)
Ihor Radchenko writes:
Hello, attached is another updated patch with the following changes:
- Made it possible to add headlines or inline tasks
to `org-ansi-highlightable-elements', these are added by default now.
- To tackle the issue discussed previously about highlights spanning
multiple lines (or elements) being removed when a line is modified I
went ahead and used the font-lock-multiline property (see
font-lock-extend-region-multiline and
font-lock-extend-region-functions) across those regions so that on
any edit of one of the lines, the region including all of the ANSI
sequences that affect that line will be re-fontified. This was the
easier solution, but the downside is that it can cause large regions
to be re-fontified when really all we want to do is apply the
highlighting face to a small line change, for example. An
alternative solution would, when no ANSI sequences are being edited
in the region being fontified and assuming a previous fontification
cycle has applied highlights due to ANSI sequences already, only
apply the highlighting face to the edited region instead of
expanding the region before fontification. The expansion
unnecessarily wastes the fontification cycle on a region larger than
what it needs to be since the information needed for highlighting
the region according to ANSI sequences has already been computed on
a previous fontification cycle. In practice I don't think this
inefficiency will matter much since I would assume most of these
ANSI sequences will be inserted due to the results of code block
execution or inserted by users who want to highlight small regions
of the document so I would consider this problem solved by using
font-lock-multiline for the time being. WDYT?
- To tackle the issue of editing around the invisible ANSI sequences I
left it up to the font-lock process to catch the invisible edits.
Whenever an edit deletes a character of the sequence that renders
the sequence invalid, the font-lock process will reveal the partial
sequence. But I had to limit what was considered a valid ANSI
sequence to get it working in a somewhat acceptable way.
The problem that I found was that if the buffer contains something
like
^[[43mfoo
(where ^[ is the ESC character and can be inserted with "C-q ESC" and
the whole sequence ^[[43m is the ANSI sequence) what was happening was
that deleting into the hidden sequence would leave the region in the
state
^[[43foo
and because the end byte of the ANSI sequence can be any character
in the ASCII range [@A-Z[\]^_`a–z{|}~], ^[[43f would still be a
valid ANSI sequence and would be hidden during the fontification
process after the edit. Since `ansi-color-apply-on-region' only
really handles the sequences that end in an m byte, just rendering
all other ones invisible, I limited the ANSI sequences handled by
this patch to be only those sequences that end in m. This way,
after deleting into the sequence like in the above example the
fontification process would not recognize the region as containing
any sequence. The downside to this solution is that sequences that
end in any other end byte won't get conveniently hidden and the
problem still persists if you have text that starts with an m and
you delete into a hidden sequence.
An alternative solution that doesn't constrain the end byte could be
to add in some extra invisible character like a zero width space and
then use something like the `modification-hooks' text property on
the character to signify that a deletion at the boundary between the
sequence and the text should really delete part of the sequence
instead of the zero width space. I haven't really worked out the
details of this, for example how would it be detected which
direction a deletion is coming from, the front or behind, but I'm
throwing it out there to see if there are any other solutions other
people might be aware of for a similar problem.
- Finally, code has been added to delete the overlays on the hidden
sequences in `org-unfontify-region' so that multiple overlays are not
created on re-fontifying regions containing those sequences.
Other than that, the code is the same as the last patch.
> P.S. I am not yet commenting on the details in the code.
Please let me know what you think of this patch and where I should be
focusing my efforts moving forward to get this submitted to Org.
One thing I would like to start doing is writing some tests for this
feature. It would be great if someone could point me to some tests
that I can peruse so that I can get an idea of how I can go about
writing some of my own. Also, are there any procedures or things I
should be aware of when trying to write my own tests?
>From 506e8c1e5a177b797a541b1541ea98c95668d5e1 Mon Sep 17 00:00:00 2001
From: Nathaniel Nicandro
Date: Tue, 9 May 2023 19:58:11 -0500
Subject: [PATCH
Re: [PATCH] Highlight ANSI sequences in the whole buffer (was [PATCH] ANSI color on example blocks and fixed width elements)
Matt writes: > Thank you for bringing this to my attention and thank you Nathaniel for your > work on this. No problem, I'm glad to contribute to Org :) > Nathaniel, if you and I happen to cross paths in one of Ihor's "office > hours," I would enjoy learning more about what you're doing. Sure. With this patch I'm attempting to fontify the regions bounded by ANSI escape sequences (just the color codes) in an Org buffer using the built-in ansi-color package to do the processing of the sequences. The challenge, for me, seems to be making ansi-color aware of Org element/object boundaries. I am aware of other ANSI escape codes that would be useful to process such as the carriage return and which appear, as you mentioned, when dealing with progress bars in a shell session. Those escape codes are not being handled at the moment. Although, I do have some experience in processing them in an Org buffer when developing my Emacs-Jupyter project. I would be glad to attempt handling these kinds of sequences in Org proper as well. -- Nathaniel
Re: [PATCH] Highlight ANSI sequences in the whole buffer (was [PATCH] ANSI color on example blocks and fixed width elements)
Ihor Radchenko writes: > It runs fine on my side, although I am not sure if things are working as > expected for all the cases. For example, headlines never got fontified > on my side when I tried your patch on top of the latest main branch. Yes, the headlines are currently not fontified since, with this patch, I was mainly looking to satisfy the initial set of rules you laid out which didn't mention headlines as being a fontifiable element for ANSI sequences, or at least I didn't get that from my reading of them. It does make sense, however, to be able to add headlines to the `org-ansi-highlightable-elements` variable so that they are fontified if the user wishes. Although doing so with this patch wouldn't work. > Also, it looks like you simply made the ASCII sequences invisible, which > causes funky behaviour when trying to edit the text around. What we may > need instead is something similar to hidden parts of the links that get > revealed when we try to edit the invisible text. See > `org-catch-invisible-edits' variable and the functions that examine it. I agree that we should reveal the invisible sequence when trying to edit it. Thanks for the tip about `org-catch-invisible-edits', it led me to `org-fold-show-set-visibility' which I think is the appropriate place to reveal a hidden sequence. > You may use `org-fontify-extend-region' to handle such scenarios if you > mark the ANSI highlights with a special text property. Thanks for the tip, I'll look into it. -- Nathaniel
Re: [PATCH] Highlight ANSI sequences in the whole buffer (was [PATCH] ANSI color on example blocks and fixed width elements)
source blocks.
An inline source block src_python{return 1+ 1 without an
end. src_python{return "t[43mest[0ming"}.
[42m Paragraph =[43mone=
_underlined [43m text *bold [42m text [0m* underlined [0m text_
_underlined [43m text_ plain[32m text _underlined [42m text_
_underlined [43m text *bold [42m te /ita[31mlic/ xt [0m end* underlined [0m text_
_underlined [43m text *bold [42m te /ita[31mlic/ xt* underlined [0m text_
_underlined [43m text_ plain _underlined [0m text_
* Greater elements with RESULTS keyword
#+RESULTS:
:drawer:
[42mLorem upsum =valor=. Some more text.
Another paragraph inside drawer.
:end:
#+RESULTS:
:RESULTS:
Paragraph [42mone.
#+begin_example
- [32mtest[0m
- [31mtest[0m
#+end_example
Paragraph [43mtwo.
:END:
#+RESULTS:
- [42mlist
- one
- three
- two
- three
#+RESULTS:
- [ ] [42mCheckbox
-
-
#+RESULTS:
- [42mList item
- [@5] List item
:drawer:
Interior
- list inner
- one two three
four five six
:end:
- tag :: description
#+RESULTS:
- [42mItem 1
- Item 2
| cell 1 | cell 2 |
| cell 3 | cell 4 |
>From 66baf6e1d435974fb4c51cc47eb5b3ace3feb22c Mon Sep 17 00:00:00 2001
From: Nathaniel Nicandro
Date: Tue, 9 May 2023 19:58:11 -0500
Subject: [PATCH] Highlight ANSI escape sequences
* etc/ORG-NEWS: Describe the new feature.
* org.el (org-fontify-ansi-sequences): New customization variable and
function which does the work of fontifying the sequences.
(org-ansi-highlightable-elements)
(org-ansi-highlightable-objects): New customization variables.
(org-ansi-new-context, org-ansi-process-region)
(org-ansi-process-block, org-ansi-process-paragraph)
(org-ansi-process-fixed-width)
(org-fontify-ansi-sequences-1): New functions.
(org-set-font-lock-defaults): Add the `org-fontify-ansi-sequences`
function to the font-lock keywords.
(org-ansi-mode): New minor mode to enable/disable highlighting of the
sequences. Enabled in Org buffers by default.
---
etc/ORG-NEWS | 12 +++
lisp/org.el | 236 +++
2 files changed, 248 insertions(+)
diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 1207d6f..76a81e3 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -492,6 +492,18 @@ Currently implemented options are:
iCalendar programs support this usage.
** New features
+*** ANSI escape sequences are now highlighted in the whole buffer
+
+A new customization ~org-fontify-ansi-sequences~ is available which
+tells Org to highlight all ANSI sequences in the buffer if non-nil and
+the new minor mode ~org-ansi-mode~ is enabled.
+
+To disable highlighting of the sequences you can either
+disable ~org-ansi-mode~ or set ~org-fontify-ansi-sequences~ to ~nil~
+and =M-x org-mode-restart RET=. Doing the latter will disable
+highlighting of sequences in all newly opened Org buffers whereas
+doing the former disables highlighting locally to the current buffer.
+
*** =ob-plantuml.el=: Support tikz file format output
=ob-plantuml.el= now output =tikz= :file format via
diff --git a/lisp/org.el b/lisp/org.el
index d2cd0b9..64a853c 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -81,6 +81,7 @@ (eval-when-compile (require 'gnus-sum))
(require 'calendar)
(require 'find-func)
(require 'format-spec)
+(require 'ansi-color)
(condition-case nil
(load (concat (file-name-directory load-file-name)
@@ -3608,6 +3609,12 @@ (defcustom org-fontify-whole-block-delimiter-line t
:group 'org-appearance
:type 'boolean)
+(defcustom org-fontify-ansi-sequences t
+ "Non-nil means to highlight ANSI escape sequences."
+ :group 'org-appearance
+ :type 'boolean
+ :package-version '(Org . "9.7"))
+
(defcustom org-highlight-latex-and-related nil
"Non-nil means highlight LaTeX related syntax in the buffer.
When non-nil, the value should be a list containing any of the
@@ -5598,6 +5605,208 @@ (defun org-fontify-extend-region (beg end _old-len)
(cons beg (or (funcall extend "end" "]" 1) end)))
(t (cons beg end))
+(defcustom org-ansi-highlightable-elements
+ '(plain-list drawer
+example-block export-block fixed-width paragraph)
+ "A list of element types that will have ANSI sequences processed."
+ :type '(list (symbol :tag "Element Type"))
+ :version "9.7"
+ :group 'org-appearance)
+
+(defcustom org-ansi-highlightable-objects
+ '(bold code export-snippet italic macro
+strike-through table-cell underline verbatim)
+ "A list of object types that will have ANSI sequences processed."
+ :type '(list (symbol :tag "Object Type"))
+ :version "9.7"
+ :group 'org-appearance)
+
+(defun org-ansi-new-context (pos)
+ (list (list (make-bool-vector 8 nil)
+ nil nil)
+(copy-marker pos)))
+
+(defun org-ansi-process-region (beg end context)
+ (or context (setq context (org-ansi-new-context beg)))
+ (move-marker (cadr context) beg)
+ (let ((ansi-col
Re: [PATCH] Highlight ANSI sequences in the whole buffer (was [PATCH] ANSI color on example blocks and fixed width elements)
Ihor Radchenko writes: > Hi, Hi Ihor, > A few months have passed since the last activity in this thread. > May I know if you are still interested in the idea? I apologize for being unresponsive all these months. Yes I'm still interested in this idea, although I have not had time to work on it recently. Life events caused me to have to stop working on it completely a few months back, I'm hoping to be able to put in more time now. I haven't even been able to put that much time into my more popular personal projects recently either! > Should you need any help, feel free to ask. I have been working on some code to satisfy the set of rules you provided in a previous email of this thread. I've made some progress, but the code is a little messy and buggy. I would like to clean it up first before I present it. Where I'm having some trouble is processing the contents of greater elements. My approach for them is basically to define an ansi-context (see `ansi-color-context-region`) for each greater element and process the inner elements using that context. This seems to work except for plain-list elements which can have other plain-list elements within them, e.g. #+RESULTS: - List item 1 - Sub-list item 1 - List item 2 - List item 3 Should the sub-list's sequence affect the rest of list elements in the parent list? If that's the case, then I think I can keep with my approach and define an ansi-context for the outermost plain-list which is used by all the other plain-list elements contained within it. Otherwise I think I would have to do something like copy the ansi-context for each inner plain-list and use the copy to process the sequences in the inner-list so that the context of the outer-list is unaffected. WDYT? -- Nathaniel
Re: [PATCH] Highlight ANSI sequences in the whole buffer (was [PATCH] ANSI color on example blocks and fixed width elements)
Ihor Radchenko writes: > Nathaniel Nicandro writes: > >>> 1. Do not allow ANSI sequences to intersect markup boundaries of the >>>same AST depth: >>>*bold * plain text should not trigger fontification >>>*bold /italic/ * should trigger >>>plain text *bold* plain text also should >> >> Just to make sure I'm getting you right. You're saying that >> fontification should trigger if the sequences live in the same >> org-element-context? > >> What about cases like: >> >> *bold* plain text >> plain text *bold * paragraph end >> >> In the first case, should only "bold" be fontified? Since the sequence >> lives in the bold context. > >> In the second, should only "text"? Since the sequence lives at a higher >> depth (the paragraph context, compared to the bold context). Or should >> it be that the fontification should extend to the end of the paragraph >> because the sequence lives at a higher depth? > > I completely missed the point that codes are not > pairs, but switches; this is completely different from Org syntax. > > So, let me re-consider where codes are likely to be used in > practice: > > 1. Inside shell code blocks (src-block element) > 2. Inside results of evaluation, which are usually fixed-width element, >but might also be example-block, export-block, drawer, table, or >other element. > 3. Inside shell inline code blocks (inline-src-block object) > 4. Inside results of evaluation of an inline code block - usually >code/verbatim markup. > > I think that the most reasonable approach to fontify ANSI sequences will > be the following: > > 1. We will consider ANSI within (a) all greater elements and lesser >elements that have RESULTS affiliated keyword (indicating that they >are result of code block evaluation); (b) otherwise, just lesser >elements, like paragraph, src block, example block, export block, >etc., but _not_ tables (c) otherwise, within verbatim-like objects, >like code, export-snippet, inline-src-block, table-cell, verbatim. > >The three groups above should be declared via variables, so that >users can tweak them as necessary. > > 2. If ANSI sequence is encountered inside a verbatim-like object and we >did not see any ANSI sequences within parent element or greater >element, limit ANSI triggers to the current object. > >Example: > >#+RESULTS: >Lorem upsum =valor=. Some more text. > >(only "valor" will be affected) > > 3. If the first ANSI sequence is encountered inside element and outside >verbatim-like object, the rest of the element is affected, including >all the objects. > >Example: > >#+RESULTS: >Lorem upsum =valor=. Some more text. > >(the first ANSI affects everything, including verbatim; the second >ANSI also affects everything) > > 4. If the first ANSI sequence is encountered inside greater element with >RESULTS affiliated keyword, all the lesser elements inside will be >affected. > >Example: > >#+RESULTS: >:drawer: >Lorem upsum =valor=. Some more text. > >Another paragraph inside drawer. >:end: > >(everything down to :end: is affected) > >or > >#+RESULTS: >- list >- one >- two >- three > >(everything is affected down to the end of the list) > > Does it make sense? > Sounds good to me. >>>> +(cl-letf (((symbol-function #'delete-region) >>>> + (lambda (beg end) >>>> + (add-text-properties beg end '(invisible t >>> >>> This is fragile and relies on internal implementation details of >>> ansi-color.el. Is there another way? >> >> Since the context in which the sequences live in need to be considered, >> it doesn't look like ansi-color-apply-on-region can be used any more >> since it isn't aware of Org objects. >> >> I've come up with a function that calculates the highlightable regions >> (considering contexts) and fontifies them, but it requires the use of >> private functions from ansi-color. Specifically >> ansi-color--face-vec-face, ansi-color--update-face-vec, and >> ansi-color--code-as-hex (used internally by ansi-color--face-vec-face). >> Does it make sense to copy over these functions into Org for the >> purposes of handling ANSI escapes? There would be some backward >> compatibility issues, e.g. ansi-color only started using faces as colors >> in Emacs 28. > > If we really need to, we can propose an extension of > ansi-color-apply-on-region upstream for Emacs itself. -- Nathaniel
Re: [PATCH] Highlight ANSI sequences in the whole buffer (was [PATCH] ANSI color on example blocks and fixed width elements)
Ihor Radchenko writes: > Nathaniel Nicandro writes: > >> The attached patch now uses `org-element-at-point' and >> `org-element-context' to query for the bounds of elements. > > Thanks! > >> Note, I've also attached an updated example file which shows that the >> escape sequences in inline source blocks are now handled similarly to >> regular source blocks, i.e. they are not fontified. > > I do not think that a single exception - source blocks is good enough. > When having something like > ANSI opening term is ==, and closing term is == > it will be not expected to get things fontified. > > A better approach will be: > 1. Do not allow ANSI sequences to intersect markup boundaries of the >same AST depth: >*bold * plain text should not trigger fontification >*bold /italic/ * should trigger >plain text *bold* plain text also should Just to make sure I'm getting you right. You're saying that fontification should trigger if the sequences live in the same org-element-context? What about cases like: *bold* plain text plain text *bold * paragraph end In the first case, should only "bold" be fontified? Since the sequence lives in the bold context. In the second, should only "text"? Since the sequence lives at a higher depth (the paragraph context, compared to the bold context). Or should it be that the fontification should extend to the end of the paragraph because the sequence lives at a higher depth? > 2. Disallow fontification is certain contexts - 'inline-src-block What I will do then is not consider sequences in inline-src-block, code, or verbatim contexts. Are there any other elements or objects that I should not consider (other than the greater elements as you mention below)? For verbatim (and code) contexts, if there are regions like plain == text ANSIy will not get considered and the region between ANSIx and ANSIz will get highlighted using ANSIx's settings. So the verbatim object gets highlighted as well. For inline source blocks, I'll do what I did in the last patch and decompose a paragraph into regions that exclude inline source blocks and only consider those regions when processing the sequences. That way the highlighting doesn't spill over into the inline source blocks (and not interfere with the syntax highlighting of them). > > Further, your current code will do something weird when encountering > greater element: > > :DRAWER: > Paragraph > > Another paragraph > :END: > > You should not consider greater elements when fontifying. > Thanks. In the case of greater elements, then, I will only consider their contents. For plain-lists and tables I will: 1. (for plain-lists) only consider the contents of the list items 2. (for tables) only consider the table-cells of each table-row >> +(cl-letf (((symbol-function #'delete-region) >> + (lambda (beg end) >> + (add-text-properties beg end '(invisible t > > This is fragile and relies on internal implementation details of > ansi-color.el. Is there another way? Since the context in which the sequences live in need to be considered, it doesn't look like ansi-color-apply-on-region can be used any more since it isn't aware of Org objects. I've come up with a function that calculates the highlightable regions (considering contexts) and fontifies them, but it requires the use of private functions from ansi-color. Specifically ansi-color--face-vec-face, ansi-color--update-face-vec, and ansi-color--code-as-hex (used internally by ansi-color--face-vec-face). Does it make sense to copy over these functions into Org for the purposes of handling ANSI escapes? There would be some backward compatibility issues, e.g. ansi-color only started using faces as colors in Emacs 28. -- Nathaniel
Re: [PATCH] Highlight ANSI sequences in the whole buffer (was [PATCH] ANSI color on example blocks and fixed width elements)
Ihor Radchenko writes: > Nathaniel Nicandro writes: > >>> Ideally, fontifying ANSI sequences should be fully controlled by users: >>> 1. We may not want to touch src blocks by default, when >>>`org-src-fontify-natively' is set to t. Only, maybe, provide an >>>option. Or you may better publish a minor mode that does this for >>>shell scripts. >>> 2. We may allow all the ANSI sequences to be fontified in the whole >>>buffer. >> >> I've updated my patch to be a combination of (1) and (2), see the >> attached patch. Essentially every sequence is fontified except those in >> source blocks and a minor mode has been created to allow users to >> disable or enable fontification whenever they want. >> >> I've also attached an example Org file with some ANSI sequences in it >> for testing purposes that you can try out. > > Thanks! > >> One issue that remains is how to handle sequences within inline source >> blocks. Those don't have a src-block property so any sequences within >> an inline source block are currently fontified. > > You should not use 'src-block property at all. There are scenarios when > jit-lock defers source block fontification (in particular, when source > block spans beyond the screen) and 'src-block property is not yet > applied. > > Instead, you should query `org-element-at-point' or > `org-element-context'. The attached patch now uses `org-element-at-point' and `org-element-context' to query for the bounds of elements. Note, I've also attached an updated example file which shows that the escape sequences in inline source blocks are now handled similarly to regular source blocks, i.e. they are not fontified. > >> +*** ANSI escape sequences are now highlighted in the whole buffer >> + >> +A new customization ~org-fontify-ansi-sequences~ is available which >> +tells Org to highlight all ANSI sequences in the buffer if non-nil and >> +the new minor mode ~org-ansi-mode~ is enabled. >> + >> +To disable highlighting of the sequences you can either >> +disable ~org-ansi-mode~ or set ~org-fontify-ansi-sequences~ to ~nil~ >> +and =M-x revert-buffer RET=. Doing the latter will disable >> +highlighting of sequences in all newly opened Org buffers whereas >> +doing the former disables highlighting locally to the current buffer. > > Rather than asking to use revert-buffer, we usually suggest M-x > org-mode-restart. Done. > >> +(defun org-fontify-ansi-sequences (limit) >> + "Fontify ANSI sequences." >> + (when (and org-fontify-ansi-sequences org-ansi-mode) >> +(let (end) >> + (while (/= (point) limit) > > Instead of this strict condition and later juggle with > `narrow-to-region', just use the usual (while (< (point) limit) ...). > Done. >> +(cond >> + ((get-text-property (point) 'src-block) > > As I mentioned above, please use `org-element-at-point'. This function > will also give you information about the block boundaries. > >> +(ansi-color-apply-on-region (point) end t) > > We should probably limit ANSI colour pairs to a single Org element. It > does not make much sense to have text in-between the quotes below > coloured: > > #+begin_quote > ... ... > #+end_quote > > > > > #+begin_quote > ... ... > #+end_quote > Makes sense. Done. >> +;; Reset the context before every fontification cycle. This >> +;; avoids issues where `ansi-color-apply-on-region' attempts to >> +;; use an old starting point that may be from a different part >> +;; of the buffer, leading to "wrong side of point" errors. >> +(setq ansi-color-context-region nil) > > This looks fragile. AFAIU, `ansi-color-context-region' is used to track > currently active ANSI colour settings. Since your fontification function > may be called with various LIMITs, depending on what is displayed on the > user screen, the fontification results might be unpredictable for ANSI > defs spanning across multiple screens. > It seems to be safe to reset `ansi-color-context-region' now given that org-element is used to find the bounds of the element at `point'. Although the fontification limits are dependent on screen size, the org-element functions are not and so the bounds used when applying the fontification for the ANSI sequences won't depend on screen size either. Also, re-setting `ansi-color-context-region' has the effect of not propagating previously applied color settings to other Org elements. >> +(defvar org-ansi-colors >> + '(black red green yellow blue purple cyan w
[PATCH] Highlight ANSI sequences in the whole buffer (was [PATCH] ANSI color on example blocks and fixed width elements)
Ihor Radchenko writes:
> Nathaniel Nicandro writes:
>
>> Attached is the patch. Without this patch, ANSI escape sequences
>> generated by the output of a source block will be left in the buffer
>> without any fontification. With this patch, the escaped text is nicely
>> colored and escape sequences hidden using overlays.
>>
>> It works for Emacs versions which have the `PRESERVE-SEQUENCES` argument
>> to the `ansi-color-apply-on-region` function. It's a bit slow due to
>> the use of overlays. My implementation of this feature in Emacs-Jupyter
>> supports older versions of Emacs without that argument, it relies on a
>> custom version of that function though and uses text properties instead
>> of overlays.
>>
>> Let me know what else could be done on my end to get this patch in.
>> Thanks.
>
> Thanks for the patch!
>
> This is an interesting idea, but I am not sure if we want to use this
> colouring by default. At least, it should be a minor mode. Probably
> enabled by default. Because not every possible user may want to have the
> escape sequences hidden away.
>
> Further, your patch only allows fontifying ANSI sequences in fixed-width
> elements, example blocks, export blocks, and src blocks without known
> major mode that does the fontification. I doubt that fontifying ANSI
> sequences in this specific subset of elements always makes sense -
> example blocks are not always used as src block output; bash code blocks
> may purposely contain escape sequences, but your patch will not handle
> them; inline src block output is not covered at all.
>
> Ideally, fontifying ANSI sequences should be fully controlled by users:
> 1. We may not want to touch src blocks by default, when
>`org-src-fontify-natively' is set to t. Only, maybe, provide an
>option. Or you may better publish a minor mode that does this for
>shell scripts.
> 2. We may allow all the ANSI sequences to be fontified in the whole
>buffer.
I've updated my patch to be a combination of (1) and (2), see the
attached patch. Essentially every sequence is fontified except those in
source blocks and a minor mode has been created to allow users to
disable or enable fontification whenever they want.
I've also attached an example Org file with some ANSI sequences in it
for testing purposes that you can try out.
One issue that remains is how to handle sequences within inline source
blocks. Those don't have a src-block property so any sequences within
an inline source block are currently fontified.
> 3. We may limit ANSI sequence fontification to results and only results.
>Or just certain types of results.
>
> The easiest will be implementing fontification in the whole buffer,
> early during fontification (and early in org-font-lock-keywords; see
> org-font-lock-set-keywords-hook).
* This is a [42mtest[0m
Of [31mANSI[0m [33mcolor[0m sequences
#+begin_src python
for x in y:
print(x + "this is a [43mtest[0m")
#+end_src
: this [42mis a[0m td
=testing=
In paragraph a ~color [44msequ[0mence~ is [41mhere[0m.
[43mThis is a sequence that covers a block
#+begin_example
should be colored
#+end_example
there should be an end here[0m there is the end.
begin [43m
sequence
without end
#+begin_src python
1 + 1
#+end_src
Inline source blocks will have sequences highlighted because we only
look for a src-block text property.
src_python{return "t[43mest[0ming"}
>From c9b505d022410a481210928ecc4cce1f199ec53b Mon Sep 17 00:00:00 2001
From: Nathaniel Nicandro
Date: Thu, 13 Apr 2023 15:06:35 -0500
Subject: [PATCH] Highlight ANSI escape sequences
* etc/ORG-NEWS: Describe the new feature.
* org.el (org-fontify-ansi-sequences): New customization variable and
function which does the work of fontifying the sequences.
(org-set-font-lock-defaults): Add the `org-fontify-ansi-sequences`
function to the font-lock keywords.
(org-ansi-mode): New minor mode to enable/disable highlighting of the
sequences. Enabled in Org buffers by default.
---
etc/ORG-NEWS | 12 ++
lisp/org.el | 112 +++
2 files changed, 124 insertions(+)
diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index b7c88fd..8690540 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -169,6 +169,18 @@ official [[https://clojure.org/guides/deps_and_cli][Clojure CLI tools]].
The command can be customized with ~ob-clojure-cli-command~.
** New features
+*** ANSI escape sequences are now highlighted in the whole buffer
+
+A new customization ~org-fontify-ansi-sequences~ is available which
+tells Org to highlight all ANSI sequences in the buffer if non-nil and
+the new minor mode ~org-ansi-mode~ is enabled.
+
+To disable highlighting of the sequences you can either
+disable ~org-ansi-mode~ or set ~org-fontify-ansi-sequences~
[PATCH] Handle block-type when checking `org-src-fontify-natively'
Hello, I think I found a bug where the condition for `org-src-fontify-natively` in `org-src-fontify-meta-lines-and-blocks-1` wasn't handling the block type. `org-src-fontify-natively` says it should fontify src blocks only, but the condition didn't have that constraint. Attached is a patch that adds in the constraint. >From 2f1342ad5b13f75387f6a26f0c68aac054903bcc Mon Sep 17 00:00:00 2001 From: Nathaniel Nicandro Date: Thu, 6 Apr 2023 08:49:20 -0500 Subject: [PATCH] Handle block-type when checking `org-src-fontify-natively` * lisp/org.el (org-fontify-meta-lines-and-blocks-1): Only fontify natively for blocks of src type. --- lisp/org.el | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/lisp/org.el b/lisp/org.el index a6b8a30..d6c5803 100644 --- a/lisp/org.el +++ b/lisp/org.el @@ -5414,8 +5414,9 @@ (defun org-fontify-meta-lines-and-blocks-1 (limit) beg end-of-endline '(font-lock-fontified t font-lock-multiline t)) (org-remove-flyspell-overlays-in beg bol-after-beginline) (org-remove-flyspell-overlays-in nl-before-endline end-of-endline) - (cond - (org-src-fontify-natively +(cond + ((and org-src-fontify-natively + (string= block-type "src")) (save-match-data (org-src-font-lock-fontify-block (or lang "") block-start block-end)) (add-text-properties bol-after-beginline block-end '(src-block t))) -- 2.39.1 -- Nathaniel
[PATCH] ANSI color on example blocks and fixed width elements
Hello, Attached is the patch. Without this patch, ANSI escape sequences generated by the output of a source block will be left in the buffer without any fontification. With this patch, the escaped text is nicely colored and escape sequences hidden using overlays. It works for Emacs versions which have the `PRESERVE-SEQUENCES` argument to the `ansi-color-apply-on-region` function. It's a bit slow due to the use of overlays. My implementation of this feature in Emacs-Jupyter supports older versions of Emacs without that argument, it relies on a custom version of that function though and uses text properties instead of overlays. Let me know what else could be done on my end to get this patch in. Thanks. diff --git a/lisp/org.el b/lisp/org.el index 4d12084..24617ad 100644 --- a/lisp/org.el +++ b/lisp/org.el @@ -81,6 +81,7 @@ (eval-when-compile (require 'gnus-sum)) (require 'calendar) (require 'find-func) (require 'format-spec) +(require 'ansi-color) (condition-case nil (load (concat (file-name-directory load-file-name) @@ -5326,6 +5327,10 @@ (defsubst org-activate-links (limit) (defun org-activate-code (limit) (when (re-search-forward "^[ \t]*\\(:\\(?: .*\\|$\\)\n?\\)" limit t) (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0)) +(let ((ansi-color-apply-face-function + (lambda (beg end face) + (font-lock-prepend-text-property beg end 'face face + (ansi-color-apply-on-region (match-beginning 0) (match-end 0) t)) (remove-text-properties (match-beginning 0) (match-end 0) '(display t invisible t intangible t)) t)) @@ -5421,7 +5426,12 @@ (defun org-fontify-meta-lines-and-blocks-1 (limit) (let ((face-name (intern (format "org-block-%s" lang (append (and (facep face-name) (list face-name)) - '(org-block))) + '(org-block)) + (let ((ansi-color-apply-face-function + (lambda (beg end face) + (font-lock-prepend-text-property beg end 'face face +(ansi-color-apply-on-region + bol-after-beginline beg-of-endline t))) ((not org-fontify-quote-and-verse-blocks)) ((string= block-type "quote") (add-face-text-property -- Nathaniel
Re: [PATCH] org-user-idle-seconds: Add support for logind
Ihor Radchenko writes: > Nathaniel Nicandro writes: > >> I would like to submit a patch that adds support for logind to >> `org-user-idle-seconds`. This patch has been working for me for a long >> while now and I thought it time for me to submit it. I've contributed >> to Emacs in the past so my paperwork should be on file. > > Thanks! > > Though I do not see any commits associated with Nathaniel Nicandro or > your email in Emacs git repo. > Bastien, may you check FSF records? > >> Let me know if any changes should be made. >> >> lisp/org-clock.el | 29 + >> 1 file changed, 29 insertions(+) > > Please provide the commit message. See > https://orgmode.org/worg/org-contribute.html#commit-messages Done in the attached updated patch. > >> +(defvar org-logind-dbus-session-path >> + (when (and (boundp 'dbus-runtime-version) >> + (require 'dbus nil t) >> + (member "org.freedesktop.login1" >> (dbus-list-activatable-names))) >> +(dbus-call-method >> + :system "org.freedesktop.login1" >> + "/org/freedesktop/login1" >> + "org.freedesktop.login1.Manager" >> + "GetSessionByPID" (emacs-pid > > Please provide a docstring for the new variable. > > Also, this new feature is worth documenting in ORG-NEWS. Done. > > Otherwise, the patch looks good. >From 1cc887d1378839c3322c93ebb5a524499b8d3073 Mon Sep 17 00:00:00 2001 From: Nathaniel Nicandro Date: Wed, 22 Mar 2023 14:54:57 -0500 Subject: [PATCH] lisp/org-clock.el: Add support for logind * lisp/org-timer.el (org-logind-dbus-session-path): New variable. (org-logind-user-idle-seconds): New function. (org-user-idle-seconds): Use them. --- etc/ORG-NEWS | 7 +++ lisp/org-clock.el | 30 ++ 2 files changed, 37 insertions(+) diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS index 87ecd77..ef3a2ab 100644 --- a/etc/ORG-NEWS +++ b/etc/ORG-NEWS @@ -87,6 +87,13 @@ a date range in the agenda. It inherits from the default face in order to remain backward-compatible. ** New features +*** Add support for ~logind~ idle time in ~org-user-idle-seconds~ + +When Emacs is built with =dbus= support and +the =org.freedesktop.login1= interface is available, fallback to +checking the =IdleSinceHint= property when +determining =org-user-idle-seconds= as the penultimate step. + *** ~org-metaup~ and ~org-metadown~ now act on headings in region When region is active and starts at a heading, ~org-metaup~ and diff --git a/lisp/org-clock.el b/lisp/org-clock.el index f869b0b..323efa4 100644 --- a/lisp/org-clock.el +++ b/lisp/org-clock.el @@ -51,6 +51,9 @@ (declare-function org-table-goto-line "org-table" (n)) (declare-function org-dynamic-block-define "org" (type func)) (declare-function w32-notification-notify "w32fns.c" ( params)) (declare-function w32-notification-close "w32fns.c" ( params)) +(declare-function dbus-list-activatable-names "dbus" ( bus)) +(declare-function dbus-call-method "dbus" (bus service path interface method args)) +(declare-function dbus-get-property "dbus" (bus service path interface property)) (defvar org-frame-title-format-backup nil) (defvar org-state) @@ -1214,6 +1217,26 @@ (defun org-x11-idle-seconds () "Return the current X11 idle time in seconds." (/ (string-to-number (shell-command-to-string org-clock-x11idle-program-name)) 1000)) +(defvar org-logind-dbus-session-path + (when (and (boundp 'dbus-runtime-version) + (require 'dbus nil t) + (member "org.freedesktop.login1" (dbus-list-activatable-names))) +(dbus-call-method + :system "org.freedesktop.login1" + "/org/freedesktop/login1" + "org.freedesktop.login1.Manager" + "GetSessionByPID" (emacs-pid))) + "D-Bus session path for the elogind interface.") + +(defun org-logind-user-idle-seconds () + "Return the number of idle seconds for the user according to logind." + (- (float-time) + (/ (dbus-get-property + :system "org.freedesktop.login1" + org-logind-dbus-session-path + "org.freedesktop.login1.Session" "IdleSinceHint") +1e6))) + (defun org-user-idle-seconds () "Return the number of seconds the user has been idle for. This routine returns a floating point number." @@ -1222,6 +1245,13 @@ (defun org-user-idle-seconds () (org-mac-idle-seconds)) ((and (eq window-system 'x) org-x11idle-exists-p) (org-x11-idle-seconds)) + ((and + org-logind-dbus-session-path + (dbus-get-property + :system "org.freedesktop.login1" + org-logind-dbus-session-path + "org.freedesktop.login1.Session" "IdleHint")) +(org-logind-user-idle-seconds)) (t (org-emacs-idle-seconds -- 2.39.1 -- Nathaniel
[PATCH] org-user-idle-seconds: Add support for logind
Hello, I would like to submit a patch that adds support for logind to `org-user-idle-seconds`. This patch has been working for me for a long while now and I thought it time for me to submit it. I've contributed to Emacs in the past so my paperwork should be on file. Let me know if any changes should be made. lisp/org-clock.el | 29 + 1 file changed, 29 insertions(+) diff --git a/lisp/org-clock.el b/lisp/org-clock.el index f869b0b..9d34a72 100644 --- a/lisp/org-clock.el +++ b/lisp/org-clock.el @@ -51,6 +51,9 @@ (declare-function org-table-goto-line "org-table" (n)) (declare-function org-dynamic-block-define "org" (type func)) (declare-function w32-notification-notify "w32fns.c" ( params)) (declare-function w32-notification-close "w32fns.c" ( params)) +(declare-function dbus-list-activatable-names "dbus" ( bus)) +(declare-function dbus-call-method "dbus" (bus service path interface method args)) +(declare-function dbus-get-property "dbus" (bus service path interface property)) (defvar org-frame-title-format-backup nil) (defvar org-state) @@ -1214,6 +1217,25 @@ (defun org-x11-idle-seconds () "Return the current X11 idle time in seconds." (/ (string-to-number (shell-command-to-string org-clock-x11idle-program-name)) 1000)) +(defvar org-logind-dbus-session-path + (when (and (boundp 'dbus-runtime-version) + (require 'dbus nil t) + (member "org.freedesktop.login1" (dbus-list-activatable-names))) +(dbus-call-method + :system "org.freedesktop.login1" + "/org/freedesktop/login1" + "org.freedesktop.login1.Manager" + "GetSessionByPID" (emacs-pid + +(defun org-logind-user-idle-seconds () + "Return the number of idle seconds for the user according to logind." + (- (float-time) + (/ (dbus-get-property + :system "org.freedesktop.login1" + org-logind-dbus-session-path + "org.freedesktop.login1.Session" "IdleSinceHint") +1e6))) + (defun org-user-idle-seconds () "Return the number of seconds the user has been idle for. This routine returns a floating point number." @@ -1222,6 +1244,13 @@ (defun org-user-idle-seconds () (org-mac-idle-seconds)) ((and (eq window-system 'x) org-x11idle-exists-p) (org-x11-idle-seconds)) + ((and + org-logind-dbus-session-path + (dbus-get-property + :system "org.freedesktop.login1" + org-logind-dbus-session-path + "org.freedesktop.login1.Session" "IdleHint")) +(org-logind-user-idle-seconds)) (t (org-emacs-idle-seconds -- 2.39.1 -- Nathaniel
Re: Feedback on Emacs-Jupyter
> Thanks a lot for your work! Thanks a lot for maintaining the wonderful Org package! > Hope it helps. Thanks, I've previously seen the video and it does help with figuring out the issues with emacs-jupyter as it currently stands. > Also, it appears that Jupyter is getting a lot of traction in research > community and it is distributed via GPL-compatible license (modified BSD > license). I believe that we should consider including the Org-related > part to Org core if the rest can be distributed via ELPA. WDYT? I would be happy to contribute the Org related parts of emacs-jupyter to Org core. I think that would mainly mean the ob-jupyter.el file that is similar to the other ob-*.el files. How would I go about contributing ob-jupyter.el in this specific situation? With regards to distributing the rest via ELPA, I would like to get the code base into a more stable state as I am planning to integrate changes that I've been working on this past year which may cause some breaking changes in people's configs. Once I've shaken out the bugs after I merge those changes, I'll be more than willing to submit emacs-jupyter as an ELPA package, if they would have it. Regards, -- Nathaniel
Re: Feedback on Emacs-Jupyter
> Let me echo others’ comments by saying how much I appreciate your work > on emacs-jupyter. I use the package daily in my work and it’s been > really fantastic. Thank you very, very much for your huge efforts and > time investment in creating and maintaining this package. Your welcome, I've enjoyed my time learning about Emacs, Org, and Jupyter in the process. > (is there an emacs-jupyter mailing list or preferred venue?) No mailing list. Currently just using GitHub. A mailing list sounds interesting though. I wonder if people would be more willing to contribute pull requests in that case? > It would be great to be able to re-establish the connection to the > REPL without discarding the buffer. I like this idea. When a connection to a kernel is lost for some reason we can reset the connection. For ZMQ based connections I think it would be as simple as shutting down and starting up new ZMQ sockets. For notebook based connections, closing and then re-opening the WebSocket that represents the connection. > 2) Getting results from a running block after >disconnecting/reconnecting. Currently it’s difficult to manage >long-running code blocks using emacs-jupyter if I disconnect from >the remote server while the code block is running. The work around >in #1 above to re-connect to a remote kernel doesn’t work if the >kernel is still executing a block. If the kernel has completed >executing the block, then the results are not populated back into >the notebook (the execution UUID populated when the code block >began executing remains). I wonder if the issue has to do with the fact that a new session ID is created every time you reconnect to the kernel using workaround #1. The function jupyter-api-get-kernel-ws is responsible for returning a new WebSocket when connecting to a kernel and using your workaround would mean the session ID associated with the WebSocket connection would be different after reconnecting. That may be why the messages don't get handled. That's just my initial guess though. Could you open up an issue over at emacs-jupyter about this? > 3) Similarly, improving feedback from a long-running block would be >very helpful. Currently you can use print statements in a long >running block to report progress, but these are lost after >disconnecting (maybe they are buffered on the jupyter server side >and could be replayed on reconnect?). Further, I don’t know if >this is feasible, but it would be amazing to have support for >progress bars for long-running tasks (e.g., have a tqdm progress >bar render in the org notebook). I'll have to investigate the buffering of messages on the Jupyter server side of things. I do think there is some message buffering going on and that they get replayed to the client, but I'll have to double check. I did play play around with tqdm, and it works mostly. What about the behavior with tqdm do you not see. When I was testing it, it seemed that the previous progress bars would stack up instead of being removed from the buffer. But then I M-x eval-defun on jupyter-org--handle-control-codes and everything worked as it should, that is only one progress bar gets refreshed on every iteration. Not sure what is happening there. Thanks for your feedback, -- Nathaniel
Re: Feedback on Emacs-Jupyter
Ken, > First, thank you (many times) for maintaining emacs-jupyter. It is one of the > most-used tools on my computer. I've been using your software daily for the > past few years to develop code and write papers. Your welcome, I'm glad that my efforts on the project mean that there are people who have integrated it into their daily workflow! > issues inputting tables into Python code blocks The issue that I found was related to how, with a cached result, the result is eventually read by org-babel-read-result, which in turn calls org-babel-read-element. And org-babel-read-element returns nil for results in a drawer. I'm assuming because there is not a standard way to interpret drawer results as Lisp. Perhaps interpreting the drawer as a list of results? So that in the particular case mentioned in https://github.com/nnicandro/emacs-jupyter/issues/267, org-babel-read-result would return a list of one element containing the corresponding table. > I throw it out there as a concept/idea I do like the idea of a standalone document that contains everything without the need of external resources. So there would be a "* Graphics" section containing base64 encoded images and then maybe on opening the document, those get translated to image links under their respective locations. Maybe by default we treat the "* Graphics" section similar to a section tagged as ARCHIVE so that a user doesn't have to weary themselves with all those encoded images. I would suggest opening up a new issue requesting the feature so we can talk more about it. > Thanks again for the excellent software, Your welcome, -- Nathaniel
Feedback on Emacs-Jupyter
Hello all, I'm the maintainer of the emacs-jupyter project (https://github.com/nnicandro/emacs-jupyter) which essentially integrates Jupyter kernels (https://jupyter.org) with Org mode source blocks. I wanted to make an introduction to the Org community. So...hello! And thanks for promoting the project on https://orgmode.org/features.html. I believe a lot of users of the project use it mainly for the Org integration. I thought it would be a good idea to get some feedback from the community on how their experience using emacs-jupyter has been. I'm getting back into active maintenance of the project and am looking for feedback to get a better idea of what the future of the project could look like. What features of standard Org source blocks do you find Jupyter source blocks are lacking? What potential features do you think would be useful for Jupyter source blocks to support, given the capabilities of Jupyter? What would it mean to see Emacs-Jupyter and Org more integrated? Of course, any other thoughts are welcome. -- Nathaniel
Re: [O] Bug: clocktable error in org-agenda [9.2.2 (release_9.2.2-245-g9d7b1e @ /home/dortmann/src/git-org-mode/lisp/)]
I am also experiencing this issue. I'm assuming =org-agenda-sticky= is set to =t= in your configuration since that is what I have in mine. I've tracked the problem down to (1) the handling of =org-agenda-sticky= in =org-agenda-goto-date=, (2) how this affects what =org-agenda-mode= (called by =org-agenda-redo=) does with =org-agenda-local-vars=, and (3) the subsequent modification of variables in =org-agenda-local-vars= before another call to =org-agenda-redo=. 1. =org-agenda-goto-date= temporarily sets =org-agenda-sticky= to =nil= before it calls =org-agenda-redo=. 2. None of the variables in =org-agenda-local-vars= are made buffer local when =org-agenda-redo= calls =org-agenda-mode= during the call to =org-agenda-goto-date=. 3. Setting one of the variables in =org-agenda-local-vars= after (2) happens actually sets its global value instead of its local value. So, for example, calling =org-agenda-log-mode= will set the global value of =org-agenda-show-log= to =t=. Once =org-agenda-redo= is called, it will once again be a local variable, but now the default global value is =t=. The main reason why log mode stays on when you try to toggle it off is because the =reset-saved= function in =org-agenda-mode= (which restores the saved values of =org-agenda-local-vars= after all local variables have been killed and re-made as local variables) does not set the value of any local variable whose saved value is =nil=. Since the local value of =org-agenda-show-log= is =nil= when you try to toggle log mode off, its buffer local value is not reset to =nil= by =reset-saved=. This means that it keeps the default global value it had when it was made into a local variable and this value is now =t= due to (3). So I think the main issue here is that =org-agenda-goto-date= does not handle =org-agenda-sticky= correctly since none of the =org-agenda-local-vars= are turned back into local variables during =org-agenda-redo= when =org-agenda-sticky= is =t=. Another possible issue is that the =reset-saved= function in =org-agenda-mode= does not set the value of variables if they are =nil=, but this is not an issue if the variables it resets are guaranteed to always have a global default of =nil=. I haven't looked into why =org-agenda-goto-date= needs to temporarily set =org-agenda-sticky= to =nil. Maybe someone can answer that, but a solution until that gets figured out is to allow =reset-saved= to set =nil= values. - Nathaniel
[O] [PATCH] Honor the :python header argument in python src-blocks
Hello, This is my first post to the mailing list and I just wanted to say thank all of you for making such a great piece of code! Below is a patch that should fix the problem where a python session was being initiated without first checking the :python header argument. --- PATCH --- From a721c97924b5b965179a2dd90e54d63c7de00317 Mon Sep 17 00:00:00 2001 From: Nathaniel Nicandro <nr...@uic.edu> Date: Thu, 27 Apr 2017 18:16:41 -0500 Subject: [PATCH] Honor the :python header argument in python src-blocks * lisp/ob-python.el (org-babel-execute:python): `org-babel-python-command` should be set before calling `org-babel-python-initiate-session`. TINYCHANGE --- lisp/ob-python.el | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lisp/ob-python.el b/lisp/ob-python.el index b838a48..f533658 100644 --- a/lisp/ob-python.el +++ b/lisp/ob-python.el @@ -73,15 +73,15 @@ This will typically be either `python' or `python-mode'." (defun org-babel-execute:python (body params) "Execute a block of Python code with Babel. This function is called by `org-babel-execute-src-block'." - (let* ((session (org-babel-python-initiate-session + (let* ((org-babel-python-command + (or (cdr (assq :python params)) org-babel-python-command)) + (session (org-babel-python-initiate-session (cdr (assq :session params (result-params (cdr (assq :result-params params))) (result-type (cdr (assq :result-type params))) (return-val (when (and (eq result-type 'value) (not session)) (cdr (assq :return params (preamble (cdr (assq :preamble params))) - (org-babel-python-command - (or (cdr (assq :python params)) org-babel-python-command)) (full-body (org-babel-expand-body:generic (concat body (if return-val (format "\nreturn %s" return-val) "")) -- 2.3.8
