[PATCH] Highlight ANSI sequences in the whole buffer (was [PATCH] ANSI color on example blocks and fixed width elements)

[Nathaniel Nicandro]

Ihor Radchenko  writes:

> Nathaniel Nicandro  writes:
>
>> Attached is the patch.  Without this patch, ANSI escape sequences
>> generated by the output of a source block will be left in the buffer
>> without any fontification.  With this patch, the escaped text is nicely
>> colored and escape sequences hidden using overlays.
>>
>> It works for Emacs versions which have the `PRESERVE-SEQUENCES` argument
>> to the `ansi-color-apply-on-region` function.  It's a bit slow due to
>> the use of overlays.  My implementation of this feature in Emacs-Jupyter
>> supports older versions of Emacs without that argument, it relies on a
>> custom version of that function though and uses text properties instead
>> of overlays.
>>
>> Let me know what else could be done on my end to get this patch in.
>> Thanks.
>
> Thanks for the patch!
>
> This is an interesting idea, but I am not sure if we want to use this
> colouring by default. At least, it should be a minor mode. Probably
> enabled by default. Because not every possible user may want to have the
> escape sequences hidden away.
>
> Further, your patch only allows fontifying ANSI sequences in fixed-width
> elements, example blocks, export blocks, and src blocks without known
> major mode that does the fontification. I doubt that fontifying ANSI
> sequences in this specific subset of elements always makes sense -
> example blocks are not always used as src block output; bash code blocks
> may purposely contain escape sequences, but your patch will not handle
> them; inline src block output is not covered at all.
>
> Ideally, fontifying ANSI sequences should be fully controlled by users:
> 1. We may not want to touch src blocks by default, when
>`org-src-fontify-natively' is set to t. Only, maybe, provide an
>option. Or you may better publish a minor mode that does this for
>shell scripts.
> 2. We may allow all the ANSI sequences to be fontified in the whole
>buffer.

I've updated my patch to be a combination of (1) and (2), see the
attached patch.  Essentially every sequence is fontified except those in
source blocks and a minor mode has been created to allow users to
disable or enable fontification whenever they want.

I've also attached an example Org file with some ANSI sequences in it
for testing purposes that you can try out.

One issue that remains is how to handle sequences within inline source
blocks.  Those don't have a src-block property so any sequences within
an inline source block are currently fontified.

> 3. We may limit ANSI sequence fontification to results and only results.
>Or just certain types of results.
>
> The easiest will be implementing fontification in the whole buffer,
> early during fontification (and early in org-font-lock-keywords; see
> org-font-lock-set-keywords-hook).

* This is a test

Of ANSI color sequences

#+begin_src python
for x in y:
print(x + "this is a test")
#+end_src

: this is a td

=testing=

In paragraph a ~color sequence~ is here.

This is a sequence that covers a block

#+begin_example
should be colored
#+end_example

there should be an end here there is the end.

begin  
sequence
without end
#+begin_src python
1 + 1
#+end_src

Inline source blocks will have sequences highlighted because we only
look for a src-block text property.

src_python{return "testing"}
>From c9b505d022410a481210928ecc4cce1f199ec53b Mon Sep 17 00:00:00 2001
From: Nathaniel Nicandro 
Date: Thu, 13 Apr 2023 15:06:35 -0500
Subject: [PATCH] Highlight ANSI escape sequences

* etc/ORG-NEWS: Describe the new feature.
* org.el (org-fontify-ansi-sequences): New customization variable and
function which does the work of fontifying the sequences.
(org-set-font-lock-defaults): Add the `org-fontify-ansi-sequences`
function to the font-lock keywords.
(org-ansi-mode): New minor mode to enable/disable highlighting of the
sequences.  Enabled in Org buffers by default.
---
 etc/ORG-NEWS |  12 ++
 lisp/org.el  | 112 +++
 2 files changed, 124 insertions(+)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index b7c88fd..8690540 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -169,6 +169,18 @@ official [[https://clojure.org/guides/deps_and_cli][Clojure CLI tools]].
 The command can be customized with ~ob-clojure-cli-command~.
 
 ** New features
+*** ANSI escape sequences are now highlighted in the whole buffer
+
+A new customization ~org-fontify-ansi-sequences~ is available which
+tells Org to highlight all ANSI sequences in the buffer if non-nil and
+the new minor mode ~org-ansi-mode~ is enabled.
+
+To disable highlighting of the sequences you can either
+disable ~org-ansi-mode~ or set ~org-fontify-ansi-sequences~ to ~nil~
+and =M-x revert-buffer RET=.  Doing the latter will disable
+highlighting of sequences in all newly opened Org buffers whereas
+doing the former disables highlighting locally to the current 

Re: [BUG] Propertized space in Agenda's mode-name [9.5.2 (release_9.5.2-3-geb9f34 @ /usr/local/share/emacs/28.0.91/lisp/org/)]

[Ihor Radchenko]

Gustavo Barros  writes:

> I get that. But please look at it from my perspective too. This is a
> reiteration of a previous report. Now you are asking me to make a
> third one, which would have to start from scratch.

Not from scratch, if you link this discussion.

> Also, if Emacs devs are requested to chime in by Org devs it is
> different from "just another user's report". Besides, I don't even
> understand why you say this is an Emacs problem, so I'd be at a loss
> at even trying to explain what the general problem is.

I suspect that it might be Emacs problem.
It will be enough for you to mention that I asked you to report to Emacs
bug tracker.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [BUG] Propertized space in Agenda's mode-name [9.5.2 (release_9.5.2-3-geb9f34 @ /usr/local/share/emacs/28.0.91/lisp/org/)]

[Gustavo Barros]

Hi Ihor,

On Thu, 13 Apr 2023 at 12:10, Ihor Radchenko  wrote:

> Basically, we need to bring Emacs devs in. Because I have no clue what
> is going on. And reporting to Emacs bug tracker is the way to involve
> Emacs devs.

I get that. But please look at it from my perspective too. This is a
reiteration of a previous report. Now you are asking me to make a
third one, which would have to start from scratch.

Also, if Emacs devs are requested to chime in by Org devs it is
different from "just another user's report". Besides, I don't even
understand why you say this is an Emacs problem, so I'd be at a loss
at even trying to explain what the general problem is.

Best,
Gustavo.

Re: [BUG] Propertized space in Agenda's mode-name [9.5.2 (release_9.5.2-3-geb9f34 @ /usr/local/share/emacs/28.0.91/lisp/org/)]

[Ihor Radchenko]

Gustavo Barros  writes:

>> It looks like Emacs bug then. I suspect some funny staff going on during 
>> compilation.
>> May you report it to Emacs bug tracker, linking to this bug report?
>
> Mhm, I don't know why you say it is an Emacs bug. From my side, it
> only affects Org Agenda. So I've reported it to Org. Besides, it would
> be a legitimate place to report this even if it was a more general
> issue.

Basically, we need to bring Emacs devs in. Because I have no clue what
is going on. And reporting to Emacs bug tracker is the way to involve
Emacs devs.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [BUG] Propertized space in Agenda's mode-name [9.5.2 (release_9.5.2-3-geb9f34 @ /usr/local/share/emacs/28.0.91/lisp/org/)]

[Gustavo Barros]

Hi Ihor,

On Thu, 13 Apr 2023 at 11:52, Ihor Radchenko  wrote:

> I inspected `mode-name'.
> I tried harder, and I was able to reproduce using literally emacs -Q.

Thanks, and I'm glad to know I'm not crazy then.

> It looks like Emacs bug then. I suspect some funny staff going on during 
> compilation.
> May you report it to Emacs bug tracker, linking to this bug report?

Mhm, I don't know why you say it is an Emacs bug. From my side, it
only affects Org Agenda. So I've reported it to Org. Besides, it would
be a legitimate place to report this even if it was a more general
issue.

Best regards,
Gustavo.

Re: [BUG] Propertized space in Agenda's mode-name [9.5.2 (release_9.5.2-3-geb9f34 @ /usr/local/share/emacs/28.0.91/lisp/org/)]

[Ihor Radchenko]

Gustavo Barros  writes:

>> I still cannot reproduce on bugfix.
>
> thanks for checking this again. When you say you can't reproduce you
> mean that you get no blank gaps in the mode line, or that, if you
> inspect `mode-name`, the space is not propertized? (Just tested the
> latter and I do find it propertized even in `emacs -Q`).

I inspected `mode-name'.
I tried harder, and I was able to reproduce using literally emacs -Q.
In contrast, I cannot reproduce when running from Emacs git directory
(not using built-in Org version), even when I use the same release tag.

It looks like Emacs bug then. I suspect some funny staff going on during 
compilation.
May you report it to Emacs bug tracker, linking to this bug report?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [BUG] Propertized space in Agenda's mode-name [9.5.2 (release_9.5.2-3-geb9f34 @ /usr/local/share/emacs/28.0.91/lisp/org/)]

[Gustavo Barros]

Hi Ihor,

On Thu, 13 Apr 2023 at 11:24, Ihor Radchenko  wrote:
>
> Gustavo Barros  writes:
>
> > For the record, I'm trying the new Emacs pretest with Org 9.6.3 and
> > the `mode-name` is still has the propertized space and, in some
> > situations (e.g. `org-tags-view`) the blank gap in the mode line still
> > creeps in for me.
>
> I still cannot reproduce on bugfix.

thanks for checking this again. When you say you can't reproduce you
mean that you get no blank gaps in the mode line, or that, if you
inspect `mode-name`, the space is not propertized? (Just tested the
latter and I do find it propertized even in `emacs -Q`).

Best,
Gustavo.

Re: [BUG] Propertized space in Agenda's mode-name [9.5.2 (release_9.5.2-3-geb9f34 @ /usr/local/share/emacs/28.0.91/lisp/org/)]

[Ihor Radchenko]

Gustavo Barros  writes:

> For the record, I'm trying the new Emacs pretest with Org 9.6.3 and
> the `mode-name` is still has the propertized space and, in some
> situations (e.g. `org-tags-view`) the blank gap in the mode line still
> creeps in for me.

I still cannot reproduce on bugfix.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [BUG] Propertized space in Agenda's mode-name [9.5.2 (release_9.5.2-3-geb9f34 @ /usr/local/share/emacs/28.0.91/lisp/org/)]

[Gustavo Barros]

Hi All,

On Sat, 22 Oct 2022 at 07:51, Gustavo Barros  wrote:
>
> On Sat, 22 Oct 2022 at 04:14, Ihor Radchenko  wrote:
>
> > I am unable to reproduce with the latest Org.
>
> thanks for checking this report.
> I've retested things here and, though I can still reproduce it with
> Org 9.5.5, I can confirm the issue is gone on 9.6-pre with the latest
> commit on master.
> Lucky us! :-)

For the record, I'm trying the new Emacs pretest with Org 9.6.3 and
the `mode-name` is still has the propertized space and, in some
situations (e.g. `org-tags-view`) the blank gap in the mode line still
creeps in for me.

Best regards,
Gustavo.

Re: how to omit section numbers from theindex.html when publishing a project

[Ihor Radchenko]

"Christopher W. Ryan"  writes:

> Well, my IT department did some major damage to my computer system, and
> now the output of publishing a project (my only one) to html is
> different. Specifically, the numbers next to each alphabetical letter in
> theindex.html are shown, whereas before they were not. I can't recall if
> this is some sort of setting that I changed once and now need to change
> back. How can I prevent thos numbers from appearing in theindex.html,
> but still have them appear in the main document for which theindex.html
> is the index?

It would help if you provided more detailed demonstration about what you
expected to happen and what happened. See
https://orgmode.org/manual/Feedback.html#Feedback

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [PATCH v2] Add compat.el support to Org (was: [POLL] Use compat.el in Org? (was: Useful package? Compat.el))

[Ihor Radchenko]

Max Nikulin  writes:

>> +;; Package-Requires: ((emacs "26.1") (compat "29.1.4.1"))
>
> Is there a way to express (or (compat "29.1.4.1") (emacs "28.1")) to 
> avoid installing compat in the case of sufficiently new emacs? E.g. 
> dpkg/apt allows such alternatives.

No, AFAIK:

D.8 Conventional Headers for Emacs Libraries

‘Package-Requires’
 If this exists, it names packages on which the current package
 depends for proper operation.  *Note Packaging Basics::.  This is
 used by the package manager both at download time (to ensure that a
 complete set of packages is downloaded) and at activation time (to
 ensure that a package is only activated if all its dependencies
 have been).

 Its format is a list of lists on a single line.  The ‘car’ of each
 sub-list is the name of a package, as a symbol.  The ‘cadr’ of each
 sub-list is the minimum acceptable version number, as a string that
 can be parsed by ‘version-to-list’.  An entry that lacks a version
 (i.e., an entry which is just a symbol, or a sub-list of one
 element) is equivalent to entry with version "0".  For instance:

  ;; Package-Requires: ((gnus "1.0") (bubbles "2.7.2") cl-lib (seq))

 The package code automatically defines a package named ‘emacs’ with
 the version number of the currently running Emacs.  This can be
 used to require a minimal version of Emacs for a package.

> Early I asked concerning compat-29.1.3. I would prefer to install 
> elpa-compat system package to avoid network activity in response to 
> make. Required version matters for those who builds packages for 
> backport repositories for various linux distributions. It allows to 
> decide if it is necessary to provide newer elpa-compat or it is enough 
> to package elpa-org.

We can put older version require now, indeed. However, later we should
not constrain ourselves about bumping compat version as necessary.

Considering Debian lifecycle, it is pretty much guaranteed that we will
require the compat version that is not installed by Debian, eventually.

In any case, see the attached additional patch.

>> compat.el is required for "compile" target. Compilation will simply fail
>> with older Emacs. And "test" triggers "compile".
>
> There are different types of build systems. Some of them allows to 
> specify which dependencies should be pulled, some do not. My expectation 
> is that make does not attempt to manage dependencies. For me it is OK to 
> type an additional command to install them and to fail otherwise.

Sure. And you will have such option (EFLAGS).
However, I decided to enable auto-downloading by default to not break
the previous working compilation instructions.

It may, however, be worth documenting EFLAGS in WORG. See the attached
patch.

> In my opinion
>
>> +@$(FIND) $(pkgdir) \(  -name \*.elc \) -exec $(RM) {} +
>
> command tells that package management capabilities in Emacs are rather 
> rudimentary (in comparison to e.g. python toolchain). That is why I am 
> in favor to more manual dependency management. Notice that I am not 
> against an option to do it from make.

May you elaborate what you mean by "more manual"? What concrete change
in the patch do you have in mind?

> Untested:
>
> $(FIND) $(pkgdir) -name \*.elc -delete
>
> "@" for silencing is intentionally dropped, parenthesis are unnecessary 
> for single condition.

Looks fine, except that now we have part of the targets using $RM
variable and part of the targets ignoring it. I am not sure if it is a
problem. I am slightly in favour of keeping the existing approach with
$RM.

>>> default.mk is included from top level Makefile only.
>> 
>> Not sure here. I just noticed that GITVERSION got re-calculated many
>> times when looking at debug output of make. (It was slowing things down
>> significantly). GITVERSION is in targets.mk though.
>
> GITVERSION is defined as ?=, so it is a variable with deferred 
> evaluation. Immediately evaluated variable may defined using :=

That's what I did in one of the patches (use :=).

>>> pkgdir = $(top_builddir)/pkg-deps
>> 
>> How to define top_builddir? If also via `pwd`, I see not much difference.
>
> I consider it as self-documenting code. Intermediate variable makes it 
> apparent for readers that the directory is relative to the top of the 
> package file tree.

Agree. Will do.
See the attached.

> Since out of source tree builds are not supported, I would consider 
> adding emacs version to path. The idea is to allow keeping installed 
> packages when switching between several emacs versions.

Done.

>From c5ba5773e69d16930fc12db6fbe0d907fae926cd Mon Sep 17 00:00:00 2001
Message-Id: 
From: Ihor Radchenko 
Date: Sat, 8 Apr 2023 19:05:37 +0200
Subject: [PATCH 1/4] mk/default.mk (pkgdir): Make it more clear that we indent
 to use top git dir

* mk/default.mk (top_builddir): New variable storing top-level directory.
(pkgdir): Use the new variable.
---
 mk/default.mk | 3 ++-
 1 file changed, 

Re: [BUG] org-latex-packages-alist type specification [9.6.3 (release_9.6.3-2-gf2949d @ /usr/local/share/emacs/29.0.90/lisp/org/)]

[General discussions about Org-mode.]

> On Apr 13, 2023, at 19:23, Ihor Radchenko  wrote:
> 
> Ruijie Yu via "General discussions about Org-mode."
>  writes:
> 
>> From 0be8702357ddf699c5ff1814a5fa57a6443b10de Mon Sep 17 00:00:00 2001
>> From: Ruijie Yu 
>> Date: Thu, 13 Apr 2023 13:14:36 +0800
>> Subject: [PATCH] * lisp/org.el org-latex-packages-alist: fixed type 
>> definition
> 
> Thanks!
> May I know if you got any reply about your FSF assignment?

Yes, it is now complete.  I presume you want to check with Bastien, so I took 
the liberty of CC’ing him myself. 

>> ---
>> lisp/org.el | 3 ++-
>> 1 file changed, 2 insertions(+), 1 deletion(-)
>> diff --git a/lisp/org.el b/lisp/org.el
>> index 26d2a8610..5bad16fd9 100644
>> --- a/lisp/org.el
>> +++ b/lisp/org.el
>> @@ -3490,7 +3490,8 @@ Make sure that you only list packages here which:
>>  (list :tag "options/package pair"
>>(string :tag "options")
>>(string :tag "package")
>> - (boolean :tag "Snippet"))
>> + (boolean :tag "snippet")
>> + (repeat :tag "compilers" string))
> 
> Compiler list may also be nil.

This should be covered by the repeat case?  Unless you want to be explicit 
about this, in which case you can wrap it in a choice, whose first choice being 
(const :tag “description” nil), and the second choice being the repeat case.  
Can’t change the patch myself because I don’t have access to a computer ATM. 

> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at .
> Support Org development at ,
> or support my work at 

--
Best,


RY

Re: [BUG] org-latex-packages-alist type specification [9.6.3 (release_9.6.3-2-gf2949d @ /usr/local/share/emacs/29.0.90/lisp/org/)]

[Ihor Radchenko]

Ruijie Yu via "General discussions about Org-mode."
 writes:

> From 0be8702357ddf699c5ff1814a5fa57a6443b10de Mon Sep 17 00:00:00 2001
> From: Ruijie Yu 
> Date: Thu, 13 Apr 2023 13:14:36 +0800
> Subject: [PATCH] * lisp/org.el org-latex-packages-alist: fixed type definition

Thanks!
May I know if you got any reply about your FSF assignment?

> ---
>  lisp/org.el | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/lisp/org.el b/lisp/org.el
> index 26d2a8610..5bad16fd9 100644
> --- a/lisp/org.el
> +++ b/lisp/org.el
> @@ -3490,7 +3490,8 @@ Make sure that you only list packages here which:
>  (list :tag "options/package pair"
>(string :tag "options")
>(string :tag "package")
> -  (boolean :tag "Snippet"))
> +  (boolean :tag "snippet")
> + (repeat :tag "compilers" string))

Compiler list may also be nil.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: #o544 is user rw?

[Ihor Radchenko]

Timothy  writes:

> Any objections?

Fixed, by Timothy, on bugfix.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=6a96054f0

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [BUG] Killing link located at beginning of line does not kill full link if region is selected to beginning or end of line by either mouse or arrow keys [9.5.2 (release_9.5.2-13-gdd6486 @ c:/Progra

[Ihor Radchenko]

"Miguel Garza"  writes:

> Yes, I think there could be arguments for different behaviors, but I suspect 
> the expected behavior should be the same as when you select the link by other 
> methods, such as starting a region at one end and then jumping to the 
> beginning or end of the line with move-beginning-of-line or move-end-of-line. 
> When killing the line or region after that movement, the entire text (URL, 
> description, all brackets) is killed.

I am not sure.
Consider a user who wants to select on the description part of
[[https://orgmode.org][link]]

Currently, it is possible to place region start at "l" by moving cursor
leftwards. Rightwards will place region start before the first "[".

What you suggest will make the described use case impossible.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at