Re: [Bug] 'org-font-lock-extra-keywords' appear next to the parent heading when its subtree is folded.

[Philipp Kiefer]

Even though it can be toggled via the 'hide' variable? Okay, will do!

On 18.02.2023 11:41, Ihor Radchenko wrote:

Philipp Kiefer  writes:


Please peruse attached to reproduce another glitch that occurs with
(setq org-hide-emphasis-markers t). The screencast illustrates what
happens in the sample file. The glitch (the pointer automatically
dropping to the line below a heading rather than staying at the far
right of the heading) seems to occur only when point is on a line below
the mouse click location (the mouse is clicked in the empty space to the
right of headlines) and it does not seem to happen for headings with
children.

I can reproduce, but I do not think that it is Org's bug.
I recommend you sending this reproducer to Emacs devs (M-x
report-emacs-bug).

Re: [Bug] 'org-font-lock-extra-keywords' appear next to the parent heading when its subtree is folded.

[Ihor Radchenko]

Philipp Kiefer  writes:

> Please peruse attached to reproduce another glitch that occurs with 
> (setq org-hide-emphasis-markers t). The screencast illustrates what 
> happens in the sample file. The glitch (the pointer automatically 
> dropping to the line below a heading rather than staying at the far 
> right of the heading) seems to occur only when point is on a line below 
> the mouse click location (the mouse is clicked in the empty space to the 
> right of headlines) and it does not seem to happen for headings with 
> children.

I can reproduce, but I do not think that it is Org's bug.
I recommend you sending this reproducer to Emacs devs (M-x
report-emacs-bug).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [Bug] 'org-font-lock-extra-keywords' appear next to the parent heading when its subtree is folded.

[Ihor Radchenko]

Philipp Kiefer  writes:

> Here's what I did:
>
>  1. Downloaded org-cycle.el and org-fold-core.el via the link you sent
>  2. Verified those are the correct files (found the changes in the files
> that were highlighted in green on the git page)
>  3. Copied both files over the ones in my
> c:\Users\PK\Documents\emacs_config\.emacs.d\elpa\org-9.6.1\ folder
>  4. Restarted Emacs
>
> Yet, the issue still occurs in the sample file I sent you.
>
> Did I do something wrong?

You did not recompile Org.

You may try to set load-prefer-newer to t to force Emacs loading file
sources if they are modified after the compiled versions.

Or you may open org-fold-core.el manually and run M-x eval-buffer (just
for testing).

Or you may download Org from git
(https://git.savannah.gnu.org/cgit/emacs/org-mode.git/)

Or you may download Org from ELPA devel (http://elpa.gnu.org/devel/).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [Bug] 'org-font-lock-extra-keywords' appear next to the parent heading when its subtree is folded.

[Philipp Kiefer]

Thanks a lot for looking into that, Ihor.

However, I wasn't able to fix the bug here. Very likely, I did not 
"install" the fix correctly as I'm not sure how to go about it - or else 
it's not a complete fix yet.


Here's what I did:

1. Downloaded org-cycle.el and org-fold-core.el via the link you sent
2. Verified those are the correct files (found the changes in the files
   that were highlighted in green on the git page)
3. Copied both files over the ones in my
   c:\Users\PK\Documents\emacs_config\.emacs.d\elpa\org-9.6.1\ folder
4. Restarted Emacs

Yet, the issue still occurs in the sample file I sent you.

Did I do something wrong?

Many thanks!

On 17.02.2023 13:01, Ihor Radchenko wrote:

Philipp Kiefer  writes:


I can reproduce with this minimal init.el using Emacs GNU Emacs 27.2
(build 1, x86_64-w64-mingw32)
   of 2021-03-26 and Org mode version 9.6.1:

Thanks for the reproducer!
It is rather an edge case.
Fixed, on main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=18a146a9d

Re: [Bug] 'org-font-lock-extra-keywords' appear next to the parent heading when its subtree is folded.

[Ihor Radchenko]

Philipp Kiefer  writes:

> I can reproduce with this minimal init.el using Emacs GNU Emacs 27.2 
> (build 1, x86_64-w64-mingw32)
>   of 2021-03-26 and Org mode version 9.6.1:

Thanks for the reproducer!
It is rather an edge case.
Fixed, on main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=18a146a9d

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [Bug] 'org-font-lock-extra-keywords' appear next to the parent heading when its subtree is folded.

[Philipp Kiefer]

Hi Ihor,

Sorry for the delay, but I had to weed out my init.el to make sure none 
of the settings were interfering with the problem I was describing.


Please find attached a sample Org file with the bug description and 
steps to reproduce the issue at the bottom of the file. (Also pasting it 
in here: Collapse all trees, then start at the bottommost heading, and 
use ^c^p repeatedly to go up through the headings to the top. Point will 
jump right over Trees 2 and 3 to 1.)


I can reproduce with this minimal init.el using Emacs GNU Emacs 27.2 
(build 1, x86_64-w64-mingw32)

 of 2021-03-26 and Org mode version 9.6.1:

-

(desktop-save-mode 1)
(setq org-hide-emphasis-markers t)
(with-current-buffer " *load*"
  (goto-char (point-max)))

--

I use the last command ("with...") to skip all the rest of my init.el. 
The problem goes away when I set org-hide-emphasis-markers to nil.


A further and likely related issue is that point often jumps to the 
beginning of the next line when I click the mouse to position point at 
the end of a heading. This can not be reproduced in the sample file, I 
think, but I could send you a screencast showing the problem. This issue 
also goes away when setting org-hide-emphasis-markers to nil.


Thanks for looking into this and please let me know if you can reproduce 
or have any further questions.


Best regards,

Philipp

On 26.01.2023 10:15, Ihor Radchenko wrote:

Philipp Kiefer  writes:


...
If any of the conditions listed above are untrue, e. g. I set
'org-hide-emphasis-markers' to nil, or remove the markers from heading A
the problem goes away.

I hope you can reproduce it based on this description.

I would help if you provided an example of the described heading.
* 1
** 1.1
*** 1.1.1
 1.1.1.1
 1.1.1.2
*** 1.2.1
** 1.2
* *2*
** 2.1
*** 2.1.1
*** 2.1.2
** 2.2
* *3*
** 3.1
*** 3.1.1
*** 3.1.2
*** 3.1.3
** 3.2
* 
* Collapse all trees, then start down here, and use ^c^p repeatedly to go up 
through the headings to the top. Point will jump right over Trees 2 and 3 to 1.

Re: [Bug] 'org-font-lock-extra-keywords' appear next to the parent heading when its subtree is folded.

[Philipp Kiefer]

On 24.01.2023 18:56, Ihor Radchenko wrote:

Philipp Kiefer  writes:


You set 'invisible text property to nil, which tells Emacs - make the
text visible. Emacs obeys.

Well, with Orgmode version 9.5, Emacs was never *this* obedient, i. e.
this problem only began after I updated Orgmode to 9.6, which is why I
considered it a bug in the first place. I am aware I have set the
keywords to be visible but would not expect them to appear at the end of
the parent heading for a folded subtree! (see the screenshots and the
explanatory .org file). Showing them there does not make any sense in my
opinion. I would expect Orgmode to hide these keywords when they are in
a collapsed subtree, regardless of the 'invisible text' setting, which
is how it was handled pre 9.6 unless I'm much mistaken.

Org may or may not do it, depending on the implementation details.
What you are seeing is because we changed the way Org is folding text to
use text properties instead of overlays (see
https://orgmode.org/Changes.html).

You may get the old behavior back by (1) Setting
`org-fold-core--optimise-for-huge-buffers' to '(grab-invisible); (2)
Setting `org-fold-core-style' to 'overlays before loading Org.

Thank you very much, that fixed it for me.

I do not consider what you report as a bug. There were ways to break Org
folding in the past (for example, by using overlays with high priority).
Actually, I am not sure why you even need to set 'invisible to nil in
your font-lock-keywords.
I chose this option to make it easier to remove emphasis markers but 
have now set up functions to do this for me automatically and have set 
'invisible to true.

Re: [Bug] 'org-font-lock-extra-keywords' appear next to the parent heading when its subtree is folded.

[Ihor Radchenko]

Philipp Kiefer  writes:

> * =problem 1: (repeatedly press C-c C-p starting at the bottom of the file 
> until you get up here -> point jumps up towards 'Export options' before it 
> reaches this line - or the line below. Seems to be caused by the emphasis 
> used on this line plus the fact that this heading has subitems)=

I was unable to reproduce on main doing the following:

1. Put necessary config into /tmp/bug.el

(setq org-emphasis-alist   
(quote (
("*" bold)
("/" org-verbatim)
; ("/" italic)
; ("_" underline)
("_" (:strike-through t))
("~" org-code)
; ("~" org-code verbatim)
("=" (:foreground "white" :background "dimgray"))
("+" (:foreground "white" :background "darkslateblue"))
; ("+" (:strike-through t))
)))

2. cd /path/to/Org/git/main/branch
3. make repro REPRO_ARGS="-l /tmp/bug.el"
4. Open bugs illustrated and explained.org
5. Follow the instructions

Using Emacs from master branch.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [Bug] 'org-font-lock-extra-keywords' appear next to the parent heading when its subtree is folded.

[Ihor Radchenko]

Philipp Kiefer  writes:

>> You set 'invisible text property to nil, which tells Emacs - make the
>> text visible. Emacs obeys.
>
> Well, with Orgmode version 9.5, Emacs was never *this* obedient, i. e. 
> this problem only began after I updated Orgmode to 9.6, which is why I 
> considered it a bug in the first place. I am aware I have set the 
> keywords to be visible but would not expect them to appear at the end of 
> the parent heading for a folded subtree! (see the screenshots and the 
> explanatory .org file). Showing them there does not make any sense in my 
> opinion. I would expect Orgmode to hide these keywords when they are in 
> a collapsed subtree, regardless of the 'invisible text' setting, which 
> is how it was handled pre 9.6 unless I'm much mistaken.

Org may or may not do it, depending on the implementation details.
What you are seeing is because we changed the way Org is folding text to
use text properties instead of overlays (see
https://orgmode.org/Changes.html).

You may get the old behavior back by (1) Setting
`org-fold-core--optimise-for-huge-buffers' to '(grab-invisible); (2)
Setting `org-fold-core-style' to 'overlays before loading Org.

I do not consider what you report as a bug. There were ways to break Org
folding in the past (for example, by using overlays with high priority).
Actually, I am not sure why you even need to set 'invisible to nil in
your font-lock-keywords.


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [Bug] 'org-font-lock-extra-keywords' appear next to the parent heading when its subtree is folded.

[Philipp Kiefer]

On 24.01.2023 12:34, Ihor Radchenko wrote:

Philipp Kiefer  writes:


Org mode version 9.6.1, GNU Emacs 27.2 (build 1, x86_64-w64-mingw32)

Please see the two screenshots here for illustration:

https://imgur.com/a/7EuUi0J  

(I'm assuming it's not a good idea - or not even possible - to send
e-mail attachments to the list?)

Quite the opposite - attachments are preferred when they do not have
large size. With attachments, the thread can be understood by the
readers years later, when whichever image hosting you use dwindles.


Great, attaching the screenshots in question. Also attaching an .org 
file illustrating the previously reported bug plus another related (and 
more serious as it impedes org functionality) one.




This is the relevant code from my init.el that seems to be causing this
issue:

(require 'org-habit nil t) ; relevant?
(defun org-add-my-extra-fonts ()
    "Add alert and overdue fonts."
    (add-to-list 'org-font-lock-extra-keywords
'("\\(³\\)\\([^\n\r\t]+\\)\\(³\\)" (1 '(face org-habit-alert-face
invisible nil)) (2 'org-habit-alert-face t) (3 '(face
org-habit-alert-face invisible nil))) t)

You set 'invisible text property to nil, which tells Emacs - make the
text visible. Emacs obeys.


Well, with Orgmode version 9.5, Emacs was never *this* obedient, i. e. 
this problem only began after I updated Orgmode to 9.6, which is why I 
considered it a bug in the first place. I am aware I have set the 
keywords to be visible but would not expect them to appear at the end of 
the parent heading for a folded subtree! (see the screenshots and the 
explanatory .org file). Showing them there does not make any sense in my 
opinion. I would expect Orgmode to hide these keywords when they are in 
a collapsed subtree, regardless of the 'invisible text' setting, which 
is how it was handled pre 9.6 unless I'm much mistaken.


The other bug illustrated in the .org file (also requiring specific 
settings, see the init.el section in the file) breaks 
'|org-previous-visible-heading' and probably some other upwards motion 
commands||in connection with parent items having  emphasis markers as 
well as subitems.

|
* 
* Export options: (ignore - but necessary to show problem 1)
#+title: En-GK 12
#+options: ':t *:t -:t ::t <:t H:0 \n:nil ^:nil arch:headline
#+options: author:nil broken-links:nil c:nil creator:nil
#+options: d:(not "LOGBOOK") date:nil e:t email:nil f:nil inline:t num:0
#+options: p:nil pri:nil prop:nil stat:t tags:t tasks:t tex:t
#+options: timestamp:nil title:t toc:nil todo:nil |:nil html-postamble:nil
#+HTML_HEAD_EXTRA: *{font-family: Garamond !important}
#+HTML_HEAD_EXTRA: *{font-size: medium;}
#+EXCLUDE_TAGS: NE
;; [[https://www.w3schools.com/cssref/pr_font_font-size.asp][CSS font-size 
property]]* 
* 
* =problem 1: (repeatedly press C-c C-p starting at the bottom of the file 
until you get up here -> point jumps up towards 'Export options' before it 
reaches this line - or the line below. Seems to be caused by the emphasis used 
on this line plus the fact that this heading has subitems)=
** top child 1
* +top+
** top child 1
* 
* 
* 
* problem 2: the keywords appear at the end of the parent heading when the 
subtree is folded under the parent.
** ³one child with font lock extra keyword³
** @another child with further font lock extra keyword@
* 
* 
* 
* init.el code causing *problem 1*:
(setq org-emphasis-alist   
(quote (
("*" bold)
("/" org-verbatim)
; ("/" italic)
; ("_" underline)
("_" (:strike-through t))
("~" org-code)
; ("~" org-code verbatim)
("=" (:foreground "white" :background "dimgray"))
("+" (:foreground "white" :background "darkslateblue"))
; ("+" (:strike-through t))
)))
* 
* init.el code causing *problem 2*:

(defun org-add-my-extra-fonts ()
  "Add alert and overdue fonts."
  (add-to-list 'org-font-lock-extra-keywords '("\\(³\\)\\([^\n\r\t]+\\)\\(³\\)" 
(1 '(face org-habit-alert-face invisible nil)) (2 'org-habit-alert-face t) (3 
'(face org-habit-alert-face invisible nil))) t)
  (add-to-list 'org-font-lock-extra-keywords '("\\(§\\)\\([^\n\r\t]+\\)\\(§\\)" 
(1 '(face org-habit-overdue-face invisible nil)) (2 'org-habit-overdue-face t) 
(3 '(face org-habit-overdue-face invisible nil))) t)
  (add-to-list 'org-font-lock-extra-keywords '("\\(@\\)\\([^\n\r\t]+\\)\\(@\\)" 
(1 '(face org-habit-clear-face invisible nil)) (2 'org-habit-clear-face t) (3 
'(face org-habit-clear-face invisible nil))) t))
(add-hook 'org-font-lock-set-keywords-hook #'org-add-my-extra-fonts)
* 
* 

Re: [Bug] 'org-font-lock-extra-keywords' appear next to the parent heading when its subtree is folded.

[Philipp Kiefer]

On 24.01.2023 04:52, Ruijie Yu wrote:

Philipp Kiefer  writes:


Org mode version 9.6.1, GNU Emacs 27.2 (build 1, x86_64-w64-mingw32)

Please see the two screenshots here for illustration:

https://imgur.com/a/7EuUi0J

(I'm assuming it's not a good idea - or not even possible - to send e-mail 
attachments to the list?)

This is the relevant code from my init.el that seems to be causing this issue:

(require 'org-habit nil t) ; relevant?
(defun org-add-my-extra-fonts ()
   "Add alert and overdue fonts."
   (add-to-list 'org-font-lock-extra-keywords 
'("\\(³\\)\\([^\n\r\t]+\\)\\(³\\)" (1 '(face org-habit-alert-face invisible 
nil)) (2 'org-habit-alert-face t) (3 '(face
org-habit-alert-face invisible nil))) t)
   (add-to-list 'org-font-lock-extra-keywords 
'("\\(§\\)\\([^\n\r\t]+\\)\\(§\\)" (1 '(face org-habit-overdue-face invisible 
nil)) (2 'org-habit-overdue-face t) (3
'(face org-habit-overdue-face invisible nil))) t)
   (add-to-list 'org-font-lock-extra-keywords 
'("\\(@\\)\\([^\n\r\t]+\\)\\(@\\)" (1 '(face org-habit-clear-face invisible 
nil)) (2 'org-habit-clear-face t) (3 '(face
org-habit-clear-face invisible nil))) t))
(add-hook 'org-font-lock-set-keywords-hook #'org-add-my-extra-fonts)

Can you reproduce it?

Thanks!

I am in a region where imgur is unavailable -- can you attach a minimal
org file where the issue manifests, and briefly describe what happens
and should happen?

Sorry, will attach such an org file in my reply to Ihor later.

Also, while looking into your init code, I noticed that
`org-font-lock-extra-keywords' seems to be public-facing (as there is no
double-dashes in the name), but has no docstring.  Either this variable
was intended to be private, or someone forgot to put a docstring to this
variable.


Thanks for making me realize that much of what is in my init.el is miles 
from 'best practice'. At the time I cobbled it together, I was just 
happy when things eventually worked the way I wanted them to and did not 
bother overmuch with proper procedure. No surprise then that it is 
coming back to bite me.


Will put in a 'defvar' statement with a docstring and rename all my 
private functions from 'org-xxx' to 'org--xxx'.

Re: [Bug] 'org-font-lock-extra-keywords' appear next to the parent heading when its subtree is folded.

[Ihor Radchenko]

Ruijie Yu via "General discussions about Org-mode."
 writes:

> Also, while looking into your init code, I noticed that
> `org-font-lock-extra-keywords' seems to be public-facing (as there is no
> double-dashes in the name), but has no docstring.  Either this variable
> was intended to be private, or someone forgot to put a docstring to this
> variable.

Rather forgot the docstring. But there is at least one in
`org-font-lock-set-keywords-hook'.

In any case, Org's fontification is going to be rewritten. So, I do not
think that we should fix this now and limit the possible changes
possible in the new fontification code.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [Bug] 'org-font-lock-extra-keywords' appear next to the parent heading when its subtree is folded.

[Ihor Radchenko]

Philipp Kiefer  writes:

> Org mode version 9.6.1, GNU Emacs 27.2 (build 1, x86_64-w64-mingw32)
>
> Please see the two screenshots here for illustration:
>
> https://imgur.com/a/7EuUi0J 
>
> (I'm assuming it's not a good idea - or not even possible - to send 
> e-mail attachments to the list?)

Quite the opposite - attachments are preferred when they do not have
large size. With attachments, the thread can be understood by the
readers years later, when whichever image hosting you use dwindles.

> This is the relevant code from my init.el that seems to be causing this 
> issue:
>
> (require 'org-habit nil t) ; relevant?
> (defun org-add-my-extra-fonts ()
>    "Add alert and overdue fonts."
>    (add-to-list 'org-font-lock-extra-keywords 
> '("\\(³\\)\\([^\n\r\t]+\\)\\(³\\)" (1 '(face org-habit-alert-face 
> invisible nil)) (2 'org-habit-alert-face t) (3 '(face 
> org-habit-alert-face invisible nil))) t)

You set 'invisible text property to nil, which tells Emacs - make the
text visible. Emacs obeys.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [Bug] 'org-font-lock-extra-keywords' appear next to the parent heading when its subtree is folded.

[General discussions about Org-mode.]

Philipp Kiefer  writes:

> Org mode version 9.6.1, GNU Emacs 27.2 (build 1, x86_64-w64-mingw32)
>
> Please see the two screenshots here for illustration:
>
> https://imgur.com/a/7EuUi0J
>
> (I'm assuming it's not a good idea - or not even possible - to send e-mail 
> attachments to the list?)
>
> This is the relevant code from my init.el that seems to be causing this issue:
>
> (require 'org-habit nil t) ; relevant?
> (defun org-add-my-extra-fonts ()
>   "Add alert and overdue fonts."
>   (add-to-list 'org-font-lock-extra-keywords 
> '("\\(³\\)\\([^\n\r\t]+\\)\\(³\\)" (1 '(face org-habit-alert-face invisible 
> nil)) (2 'org-habit-alert-face t) (3 '(face
> org-habit-alert-face invisible nil))) t)
>   (add-to-list 'org-font-lock-extra-keywords 
> '("\\(§\\)\\([^\n\r\t]+\\)\\(§\\)" (1 '(face org-habit-overdue-face invisible 
> nil)) (2 'org-habit-overdue-face t) (3
> '(face org-habit-overdue-face invisible nil))) t)
>   (add-to-list 'org-font-lock-extra-keywords 
> '("\\(@\\)\\([^\n\r\t]+\\)\\(@\\)" (1 '(face org-habit-clear-face invisible 
> nil)) (2 'org-habit-clear-face t) (3 '(face
> org-habit-clear-face invisible nil))) t))
> (add-hook 'org-font-lock-set-keywords-hook #'org-add-my-extra-fonts)
>
> Can you reproduce it?
>
> Thanks!

I am in a region where imgur is unavailable -- can you attach a minimal
org file where the issue manifests, and briefly describe what happens
and should happen?

Also, while looking into your init code, I noticed that
`org-font-lock-extra-keywords' seems to be public-facing (as there is no
double-dashes in the name), but has no docstring.  Either this variable
was intended to be private, or someone forgot to put a docstring to this
variable.

Thoughts?

Best,


RY

[Bug] 'org-font-lock-extra-keywords' appear next to the parent heading when its subtree is folded.

[Philipp Kiefer]

Org mode version 9.6.1, GNU Emacs 27.2 (build 1, x86_64-w64-mingw32)

Please see the two screenshots here for illustration:

https://imgur.com/a/7EuUi0J 

(I'm assuming it's not a good idea - or not even possible - to send 
e-mail attachments to the list?)


This is the relevant code from my init.el that seems to be causing this 
issue:


(require 'org-habit nil t) ; relevant?
(defun org-add-my-extra-fonts ()
  "Add alert and overdue fonts."
  (add-to-list 'org-font-lock-extra-keywords 
'("\\(³\\)\\([^\n\r\t]+\\)\\(³\\)" (1 '(face org-habit-alert-face 
invisible nil)) (2 'org-habit-alert-face t) (3 '(face 
org-habit-alert-face invisible nil))) t)
  (add-to-list 'org-font-lock-extra-keywords 
'("\\(§\\)\\([^\n\r\t]+\\)\\(§\\)" (1 '(face org-habit-overdue-face 
invisible nil)) (2 'org-habit-overdue-face t) (3 '(face 
org-habit-overdue-face invisible nil))) t)
  (add-to-list 'org-font-lock-extra-keywords 
'("\\(@\\)\\([^\n\r\t]+\\)\\(@\\)" (1 '(face org-habit-clear-face 
invisible nil)) (2 'org-habit-clear-face t) (3 '(face 
org-habit-clear-face invisible nil))) t))

(add-hook 'org-font-lock-set-keywords-hook #'org-add-my-extra-fonts)

Can you reproduce it?

Thanks!