Re: Enhancing the HTML exporter: Create a new backend or contribute to the upstream

[Max Nikulin]

On 08/09/2024 21:46, Max Nikulin wrote:

On 07/09/2024 18:53, Ihor Radchenko wrote:

Then, pagefind can be the default (it is MIT license - GPL compatible),


It might be more tricky:


Sorry for the noise. Of course, if you are not going to include any 
pagefind code into Org then it is not an issue.

Re: Enhancing the HTML exporter: Create a new backend or contribute to the upstream

[Max Nikulin]

On 07/09/2024 18:53, Ihor Radchenko wrote:

Then, pagefind can be the default (it is MIT license - GPL compatible),


It might be more tricky:


emacs-devel. Re: [Nicolas Graves] [PATCH v6 01/10] rde: emacs: Start 
emacs in --daemon mode, with shepherd and pid-file. Sun, 12 May 2024 
12:36:46 +0300

Nicolas Graves:

The code is given as MIT-0, hence also the two different licenses for
the two functions sd_notify and sd_is_socket. Not an expert on licenses
either, but with a proper flag about what this function's license is, I
guess it should be fine, since other projects also do that.


Eli Zaretskii:

The license is only half of the problem.  Every non-trivial
contribution to Emacs must have its copyright assigned to the FSF,
because the FSF is in charge of protecting the Emacs sources,
something that only the copyright holder can do, at least in some
countries.  You will need to assign the copyright as well (a
relatively simple procedure of filling a form and emailing it), but if
the code is not yours, you cannot assign its copyright.

Re: Adding text/org MIME type to jshttp/mime-db

[Max Nikulin]

On 05/08/2024 10:19, Joseph Turner wrote:

Max Nikulin writes:

- My impression is that multiple types may have the same suffix, so
   there should be "get all media types for given suffix" method.
- It should be possible to specify priorities within the same source.

[...]

However, on the topic of Org mode support in Agregore, I am not sure
that the ability to specify a different preference or even a "get all
media types for given suffix" function would help.


My idea is that this method pushes developers toward more flexible 
design in respect to specifying priorities of records. With its presence 
it is more probable that users of the DB will be prepared to cases when 
2 step mappings media type -> suffix -> media type and suffix -> media 
type -> suffix give values that are different from the original ones.



For now, perhaps we can override the MIME type map in the extension:

https://github.com/AgregoreWeb/extension-agregore-renderer/issues/8#issuecomment-2267958828


I have added a comment. The issue may be Content-Type header sent by 
server (or other side of p2p connection).

Re: Adding text/org MIME type to jshttp/mime-db

[Max Nikulin]

On 04/08/2024 00:37, Joseph Turner wrote:

jshttp/mime-types utility does this:

   // source preference (least -> most)
   var preference = ['nginx', 'apache', undefined, 'iana']

and then discards duplicate mappings which are of lower preference.


I can not say that I have consistent vision which way media type 
database should be designed. It is rather close to off-topic on this 
list (though `org-file-types' is a kind of override map, built-in 
mailcap.el is far from being perfect, and Emacs ignores shared-mime-info 
completely).


- Both fallback and override entries should be implemented.
- It seems there are differences which way custom entries should be 
handled during compiling database and when an application uses third 
party mappings. The reason is that version of external database may be 
older or newer than version of an application. Some conflicts may cause 
fatal error during compiling of database while applications have to be 
more intelligent in respect to conflict resolution.
- Priorities of media type to suffix and suffix to media type may be 
different.
- My impression is that multiple types may have the same suffix, so 
there should be "get all media types for given suffix" method.

- It should be possible to specify priorities within the same source.

I am a bit surprised that jshttp/mime-types developers use added entries 
as fallback, not as overrides for IANA, but it is their design decision. 
Since it is a DB, not DB user, they can just drop conflicting entries. 
Perhaps they still want to preserve ambiguous mapping of multiple media 
types to the same suffix (otherwise text/x-org should be dropped due to 
a conflict).



Max Nikulin writes:

I think, text/org may be added as well (without removing of
text/x-org).


Shall we consider this this after we find a way for these low-preference
mappings to be useful?


I still believe that the following behavior is preferable

- applicaion/vnd.lotus-organizer -> .org
- text/x-org -> .org
- text/org -> .org
- .org -> text/org (preferably without relying on x- heuristics)

Unless a user explicitly configured local mapping .org to 
applicaion/vnd.lotus-organizer

Re: Adding text/org MIME type to jshttp/mime-db

[Max Nikulin]

On 02/08/2024 02:55, Joseph Turner wrote:

Support for "text/org" in  would let
the Agregore browser () render Org files.
Then hyperdrive.el users could share Org files to be viewed in Agregore.

John Kitchin opened a PR back in 2017, which was rejected:

https://github.com/jshttp/mime-db/pull/82


I missed it at first, but it was *closed*, but not *rejected*.


2017-07-25T01:12:28Z

Add text/x-org

closes #82


So you can use text/x-org.


After I posted a comment on the thread, the maintainer responded:


I would suggest opening a new PR.


I think, text/org may be added as well (without removing of text/x-org).

Perhaps a worg page may be created to clarify current state of media 
type for org files. The question is what is the appropriate directory.

Re: Adding text/org MIME type to jshttp/mime-db

[Max Nikulin]

On 02/08/2024 02:55, Joseph Turner wrote:

Support for "text/org" in  would let
the Agregore browser () render Org files.
Then hyperdrive.el users could share Org files to be viewed in Agregore.

John Kitchin opened a PR back in 2017, which was rejected:

https://github.com/jshttp/mime-db/pull/82

[...]

I would suggest opening a new PR.

[...]

I'd like to open a new PR.  Would you elaborate on "de facto mimetype"?


Nothing more than my opinion:

I think, if you are going to convince somebody outside of Org community 
then you should mention that the format is supported by GitHub and 
GitLab. Org files are rendered using org-ruby gem. Only a part of 
features is available, but still useful.



shared-mime-info used by Linux desktop applications has .org text/org 
association:


(though text/x-org alias and -*- mode: org -*- magic are missed).
Thunderbird sends .org attachments as text/org without explicit 
configuration.


Till finalizing of https://orgmode.org/worg/org-syntax.html and writing 
spec (RFC) there is no point to send a request to IANA.


Emacs will continue to use text/x-org till somebody manages to convince 
developers that transition to text/org does not break backward compatibility


Actually Emacs will continue to use application/vnd.lotus-organizer in 
some cases since to /etc/mime.types has higher priority.


I am not familiar with Agregore, but my opinion that media type 
associations should be configurable in applications and either 
shared-mime-info or /etc/mime.types should has higher priority than any 
built-in mapping, perhaps besides some exceptions for media types 
directly related to the application.

Re: Capture, template expansion and keyword for link type

[Max Nikulin]

On 12/07/2024 23:34, Sébastien Gendre wrote:

Max Nikulin writes:

See specific store link functions, e.g. `org-gnus-store-link'.


Reading the `org-gnus-store-link' function, I see that is set: type,
file, node, link and description.


I see :from :to :subject :message-id :group, :date. Other keywords 
mentioned in the manual are added by `org-link-store-props'.



I wonder if I can write an Elisp function that retrieve all the
properties set for each org link types and tell we at which mode they
are used. After all, with Elisp code could be data too.


It may require non-trivial analysis of code.


I think I will need to make another subprotocol.
Maybe a "capture-plus" that will accept any keys and make all of them
accessible as a '%:keymord'.


I do not think there is any blocker that does not allow you to define 
your own subprotocol. If you find it useful then you will have 
opportunity to contribute it.



Not all books have authors in metadata. Some have editors since every
chapter has its own author. To avoid handling such logic in capture
templates I decided to resort to formatting inside an extension (or
formatting performed by a dedicated tool if site has API, e.g.
GitHub).


Can you tell me more about this ?


What particular you are interested in? Are you surprised that some books 
have several authors and some do not have it in citations? I believe 
that elisp code embedded into capture templates is inconvenient to 
debug, that is why I prefer to run formatter in some other way.



If you are using Emacs-30 I would suggest to keep away from
org-protocol hacks and to rely on `server-eval-args-left' handled by a
dedicated function

 emacsclient --eval '(my-capture)' 'my-proto://something?param=value'

For older versions you may either use backslash hell like in
emacsclient-mail.desktop or to define custom subprotocol for
org-protocol.


Is there any change planned for org-protocole in Emacs 30 ?


I do not expect it. Almost certainly only important fixes will be 
committed to that branch.



The big advantage for org-protocole, is that it's based on an URL that
any external software can recognize and pass to Emacs and it's easy to
write a custom sub-protocole.


`server-eval-args-left' allows to do the same without a set of ugly 
enough hacks that were unavoidable to make org-protocol working.

Re: [DISCUSSION] Possible inclusion of org-capture.el into Emacs core

[Max Nikulin]

I have dropped emacs-devel from Cc.

On 30/12/2023 19:15, Ihor Radchenko wrote:

With org-protocol, one can also make Emacs receive data from browser and
perform any action defined by custom handler function - all fully
configurable and not necessarily related to Org mode.


and

Ihor Radchenko [SUMMARY] #8 [[bbb:OrgMeetup]] on Wed, June 12, 19:00 
UTC+3. Sat, 22 Jun 2024 08:32:40 +.



  - Splitting Org into multiple small libraries is also a step forward
toward more modular Org and to converting some Org-specific
libraries to proper Emacs libraries, as requested by RMS:
https://list.orgmode.org/e1kikxv-0007iy...@fencepost.gnu.org/

[...]

  - org-protocol :: Processing external input into Emacs (from
command line, from clicked Urls as url handler). Unlike
running commands directly (which is unsafe), =org-protocol= can
be used for safe interaction with external processes


I have nothing against adding org-capture to Emacs core, but I think 
org-protocol should be just obsoleted. Since `server-eval-args-left' has 
been added to emacs-30, hacks from org-protocol have a little value. The 
new feature allows to implement handlers of URL schemes in a safe way 
and to avoid backslash hell similar to emacsclient-mail.desktop.


A tiny package may be created for convenience to have single .desktop 
file for all URI protocols (mailto:, org-protocol:). Leaving aside 
tricks with --display, handler should be like


 emacsclient --eval "(handle-url)" %u

or %U for multiple URLs. Packages just register supported protocols in 
e.g. an alist and the dispatcher calls a function for matched scheme 
with an argument from `server-eval-args-left'.

Re: Capture, template expansion and keyword for link type

[Max Nikulin]

On 12/07/2024 03:36, Sébastien Gendre wrote:

Max Nikulin writes:

See the "Link type, Available keywords" table in this section.
`org-capture' may be executed in various Emacs buffers and available
keywords may depend on major mode.


Is it necessary to call "org-store-link" before calling "org-capture" to
extracet the informations used by theses keywords ?

[...]

I found the documentation unclear about this.


Have a look into `org-capture' code. It calls `org-store-link'.


Is there a way to discover the keywords that a major mode provide to a
capture template ?


See specific store link functions, e.g. `org-gnus-store-link'.


%(let ((bla (plist-get (plist-get org-store-link-plist :query) :bla)))
(or bla "No bla"))"

emacsclient
'org-protocol:/capture?template=s&title=Hello&body=World&url=http:%2F%2Fexample.com&bla=foo'


A new parameter added to the org-protocol url is only accessible through
the ":query" key ? It will not create a new "%:keywoard" automatically ?


With "capture" subprotocol it does not mapped to keywords.


What I would love to do, is creating a Firefox extension that can
extract different metadata from known well known web site and pass them
to an Org-mode capture through org-protocol.

For example, for a book, extracting author, title, year, etc.


Not all books have authors in metadata. Some have editors since every 
chapter has its own author. To avoid handling such logic in capture 
templates I decided to resort to formatting inside an extension (or 
formatting performed by a dedicated tool if site has API, e.g. GitHub).



And then, I could simply create a capture template that would use the
keywords "%:author", "%:title", etc.


If you are using Emacs-30 I would suggest to keep away from org-protocol 
hacks and to rely on `server-eval-args-left' handled by a dedicated function


emacsclient --eval '(my-capture)' 'my-proto://something?param=value'

For older versions you may either use backslash hell like in 
emacsclient-mail.desktop or to define custom subprotocol for org-protocol.

Re: [PATCH] oc-csl: New custom option `org-cite-csl-sentence-case-bibtex-titles'

[Max Nikulin]

On 08/07/2024 19:56, András Simonyi wrote:

What I've been arguing for in this thread and what motivates the
current default behavior is that it is a general, style-independent
CSL assumption that the processor's input for title fields is in
sentence-case, and this requires converting the BibLaTeX title fields
in title-case to sentence-case before feeding into the CSL processor
independently of the style  -- please refer to my email dated 14 May
for further details and the references therein.


András, in 
https://list.orgmode.org/d8b1b73f-6b4e-4946-8873-748f78e19...@gmail.com 
I tried to ask if it is possible to determine if a specific CSL style 
uses title or sentence case. I believe that without "independently of 
the style" requirement behavior may be more convenient for users.


Consider a user who have .bib files following BibTeX recommendations: 
title case with hints where capital letters should be preserved during 
conversion to sentence case. Their is preparing a couple of papers. One 
journal requires StyleT with title case, another one StyleS with 
sentence case. If citeproc may determine what kind of capitalization is 
used by each style then it performs transformation to sentence case for 
StyleS, but passes strings as is for StyleT. Conversion to title case 
for StyleT is applied, but has no effect.


I can not figure out what user case is not covered and requires explicit 
setting.


Feel free to ignore this message.

Re: Capture from Firefox to Org-mode

[Max Nikulin]

On 09/07/2024 23:45, Morgan Willcock wrote:

Just to mention another option, I use this one:

https://addons.mozilla.org/en-US/firefox/addon/linkremark/


In its current state it does not extract metadata specific to books or 
events. Dublin Core or other similar metadata are on TODO list, but with 
low priority. The extension may give priority to doi: links and it may 
be starting point to fetch an entry from some database.


If "magazin to buy" page contains schema.org/Product metadata (LD-JSON 
or microdata) then it may work.


I wonder what is expected capture result for events and what metadata 
are available?


Currently the primary issue is compatibility with recently enforced 
Chrome requirements. More asynchronous code required to perform similar 
tasks (combination of async-await, callbacks and messages between 
extension components).


I have not tried it, but the following package has extractors for some 
sites:

Re: Capture, template expansion and keyword for link type

[Max Nikulin]

On 11/07/2024 13:59, Sébastien Gendre wrote:

I was reading the Org-mode manual, at section "10.1.3.2 Template
expansion":
https://orgmode.org/manual/Template-expansion.html

And I was a little bit confused by the "%:keyword". The manual say:
"Specific information for certain link types, see below."

Where did this link come from ?

When is it defined, and from/to where ?


See the "Link type, Available keywords" table in this section. 
`org-capture' may be executed in various Emacs buffers and available 
keywords may depend on major mode.



Is it possible to passe more keyword through an org-protocol call ?


Some time ago I tried %(exp) elisp snippets in templates (untested)

  "* [[%:link][%:description]]
#+BEGIN_QUOTE
%i
#+END_QUOTE
%(let ((bla (plist-get (plist-get org-store-link-plist :query) :bla)))
   (or bla "No bla"))"

emacsclient 
'org-protocol:/capture?template=s&title=Hello&body=World&url=http:%2F%2Fexample.com&bla=foo'

Re: [PATCH] oc-csl: New custom option `org-cite-csl-sentence-case-bibtex-titles'

[Max Nikulin]

On 06/07/2024 16:59, András Simonyi wrote:


Well, what they typically want is using the original title case
despite a style not containing any explicit instruction that the title
field should be in title-case, which, according to the latest. 1.02
CSL standard (https://docs.citationstyles.org/en/stable/specification.html),
implies that the field in question  should be formatted in
sentence-case:


If a style does not specify conversion or use "title" then result should 
be in title case. Title case is recommended for BibTeX databases. If you 
pass string in title case then conversion to title case should not 
change it. You do not need any conversion.


Original BibTeX title should be converted to sentence case only if 
specific style explicitly demand it.


I still have a hope that it should work without user settings in most 
cases. Maybe it is better to resort to particular examples with some 
style and a bib entry.

Re: org-babel-execute-src-block filters characters from :session *shell* output

[Max Nikulin]

On 01/07/2024 02:08, Ihor Radchenko wrote:

+++ b/lisp/ob-shell.el
@@ -273,7 +273,7 @@ (defvar org-babel-sh-eoe-indicator "echo 'org_babel_sh_eoe'"
   "String to indicate that evaluation has completed.")
 (defvar org-babel-sh-eoe-output "org_babel_sh_eoe"
   "String to indicate that evaluation has completed.")
-(defvar org-babel-sh-prompt "org_babel_sh_prompt> "
+(defvar org-babel-sh-prompt "𒆸 "
   "String to set prompt in session shell.")


I would use a longer prompt to reduce a chance that these pair of 
characters appear in output of some program.

Re: org-babel-execute-src-block filters characters from :session *shell* output

[Max Nikulin]

On 18/06/2024 00:57, Ihor Radchenko wrote:

Max Nikulin writes:


Some shells support "semantic shell" that allows terminal applications
e.g. to copy whole command output. It is based on escape sequences.
https://gitlab.freedesktop.org/Per_Bothner/specifications/blob/master/proposals/semantic-prompts.md


I am looking at this gitlab link, and they way it is implemented is
simply setting PROMPT in a way that adds these escape sequences to the
original PROMPT value.


It does a bit more. There are escape sequences to mark user input, 
program output, and to report program exit status. I had in mind support 
of this protocol in comint, so Org babel session make take advantage of 
it if the feature is available for a specific interpreter.

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 01/07/2024 23:01, Ihor Radchenko wrote:

Max Nikulin writes:

bug#71081 is about shell-command-on-region, not about mailcap.
(and pty may actually not matter there, because Windows' cmd.exe is not
even POSIX-compient...)


Of course, there are no pty's on Windows and cmd.exe is a special case. 
I was trying to say that it may be not so easy to start a process 
without tty allocation on Linux.



Either explicitly specify
=
[...]

May you submit a patch for the WORG docs that documents this caveat?


9733cc41 2024-07-03 17:57:15 +0700 ob-doc-shell.org: Warn concerning 
optional stdin


Feel free to rewrite added text or to move it from introduction to some 
other section. HTML page should be updated when somebody with payed 
account will push more changes.

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 01/07/2024 16:51, Ihor Radchenko wrote:

Max Nikulin writes:


My experience is that it is not easy to start a process without a tty in
Emacs. Eli will object.


I guess not on Windows (bug#71081). But otherwise I agree.


It was for starting mailcap viewer and Windows was not an issue.


Although, I am not sure what we need to document and what we just need
to rewrite.


I have posted it to this thread and to https://debbugs.gnu.org/67259

Some commands try to read standard input if it is available.  In 
interactive sessions it may be hidden due to time interval between typed 
commands.  Be careful when they are used in Org source blocks.

[[https://mywiki.wooledge.org/BashFAQ/089][BASH FAQ #89]]
warns concerning =ssh= and =ffmpeg=.  Either explicitly specify
=>file.log' <<"EOF"
Added by org-babel
EOF
#+end_example

(Quotes around "EOF" suppress variable expansion in the text.)  Tools 
may have dedicated options, for example =ssh -n= is a more concise way 
to avoid the pitfall.



In theory, we may work around all the pty-related problems
by simply running a script file (that possibly also echoes its commands
to keep them recorded for user examination).


Ideally some kind of bracketed paste should be added to `shell', 
otherwise it is insecure. However it is not an option for dash. I still 
have slight hope that implementation of =C-x C-e= might give another 
idea. Some subtle issues may arise with "source", but perhaps they are 
more rare than naive ssh usage. If you know how to add text to shell 
buffer without sending it to the process then you may try. I would leave 
an option to switch back to "direct" input.

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 30/06/2024 23:42, Ihor Radchenko wrote:

Max Nikulin writes:


On 30/06/2024 22:28, Ihor Radchenko wrote:

I do not see a great benefit of re-implementing session support via
actual terminal. (I think we agree here).


`shell' and so comint allocates "actual" pty.


Are you sure?


ps wf
PID TTY  STAT   TIME COMMAND
   2269 pts/0Ss 0:01 /bin/bash
 123454 pts/0Sl 0:16  \_ emacs -Q
 125104 pts/4Ss+0:00  \_ /bin/bash --noediting -i
 129807 pts/5Ss+0:00  \_ /bin/bash
 132172 pts/6Ss+0:00  \_ /usr/bin/bash --noediting -i

pts/0 is created by konsole, others are `shell' and `term' sessions 
inside Emacs.


My experience is that it is not easy to start a process without a tty in 
Emacs. Eli will object.

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 30/06/2024 22:28, Ihor Radchenko wrote:

I do not see a great benefit of re-implementing session support via
actual terminal. (I think we agree here).


`shell' and so comint allocates "actual" pty. I expect that bracketed 
paste is just some escape sequences delimiting pasted text (fcntl calls 
are less probable). The obstacle here is TERM=dumb.


I would be OK with just documenting that ob-shell blocks differ from 
interactive sessions and some care is required. It is you who want to 
provide experience close to interactive sessions.


There is a chance that some shell built-in command allows to get 
behavior close to edit-and-execute-command (C-x C-e) that, unlike 
"source", echoes input.


Currently I have no over ideas how to avoid interpreting some lines as 
input to earlier commands (and at the same time send some lines to stdin 
of commands as it would be in interactive sessions).

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 30/06/2024 18:06, Ihor Radchenko wrote:

Max Nikulin writes:


https://debbugs.gnu.org/67259

Notice that the report is for *interactive* sessions. Org babel sessions
are not so interactive. They are to maintain state, not to interact with
users directly...


You are not right. We have legitimate cases when people can switch to
session buffer, enter commands there, switch back to Org buffers,
send code blocks to the same session, and so on.


Timings are still important part of interactive sessions and ob-shell 
has no chance to simulate them having just block of commands.


By the way, dash does not support bracketed paste since it is a minimal 
shell mainly for interpreting scripts. So result of pasting several 
commands at once is different from bash. I do not think it is related to 
dash bugs.



Possible voluntary decision is to consider every session source block
similar to "C-x C-e" in bash prompt, so nothing should become implicit
stdin of commands. Content of a source block should be saved into a
temporary file then "source /path/to/block_content" command should be
send to the session buffer. Likely command depends on shell.


This may work, but we should at least echo the original commands into
session buffer, because users may want to examine them.


I have no idea which way bash or readline function 
edit-and-execute-command is implemented, but commands are echoed when 
editor is closed. Anyway it may greatly vary across shells.


I am unaware if it is possible to detect if bracketed paste is 
implemented and to send input as bracketed paste. Perhaps some escape 
sequences are involved. ob-shell might offer variants for sessions: dumb 
stdin, source a temporary file, or bracketed paste. Anything besides 
stdin (with its confusing differences from interactive sessions) is 
almost certainly requires shell-specific code. And `shell' is likely not 
suitable with its TERM=dumb.

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 18/11/2023 22:51, Matt wrote:

   On Fri, 17 Nov 2023 23:07:57 +0100  Matt  wrote ---

The second claim has nothing to do with Org Babel.  I was able to
confirm it and provide the steps to reproduce.  I think it would make
sense to report it upstream and let them decide if it's expected
behavior.  I'm still happy to do that, but I need to step away from the
keyboard :)



https://lists.gnu.org/archive/html/bug-gnu-emacs/2023-11/msg00976.html

https://debbugs.gnu.org/67259

Notice that the report is for *interactive* sessions. Org babel sessions 
are not so interactive. They are to maintain state, not to interact with 
users directly. Consider


fakessh() {
bash -c 'read -t 10 -r; printf "fakessh read: %s\n" "$REPLY"';
}

fakessh
echo next

If you type them in terminal then result depends on delay between 
"fakessh" and "echo next" commands. SSH is an example of command that 
*may* read stdin while its command is running. To avoid ambiguity you 
have to do either


fakessh "Reproducing" the issue in xterm-like applications or various emacs 
shells makes things even more complicated. Result depends on TERM value 
and if bracketed paste is enabled. With active bracketed paste shell 
knows if you paste single line or a several lines into *shell prompt*, 
so it does not feed second and following lines to first command stdin. 
In the case of


#+begin_src bash :session *ob-shell*
  bash -c 'read -t 10 -r; printf "read: %s\n" "$REPLY"'
  echo next
#+end_src

comint can not guess if all lines are commands or some of them should be 
considered as input to the previous command. Timing or bracketed paste 
are irrelevant for ob-shell sessions.


Possible voluntary decision is to consider every session source block 
similar to "C-x C-e" in bash prompt, so nothing should become implicit 
stdin of commands. Content of a source block should be saved into a 
temporary file then "source /path/to/block_content" command should be 
send to the session buffer. Likely command depends on shell.

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 18/11/2023 15:04, Max Nikulin wrote:

On 17/11/2023 17:17, Ihor Radchenko wrote:

I see bash vs. dash difference with public key authorization, so no need 
for password prompts. I have not figured out how to construct an example 
without ssh since this command *may* read stdin, but does not do it in a 
same way as e.g. cat(1).


I expect the following couple of lines simulates ssh behavior fairly 
well (besides maybe macOS with its old GPLv2 bash).


bash -c 'read -t 0.5 -r line; printf "read: %s\n" "$line"'
printf 'printf\n'


cat ssh-script.sh
ssh -p  127.0.0.1 'echo foo>/tmp/foo'
echo done

[...]

dash 
[...]
I am unsure if POSIX specifies exact behavior of shell when commands are 
read from stdin.


It is a bug in dash that should be fixed in next version. Behavior of 
bash is correct despite some users may expect "done" printed.


<https://lore.kernel.org/dash/20221213221732.6mvv22u7ktdoz...@tarta.nabijaczleweli.xyz/t/>
наб  to d...@vger.kernel.org
[PATCH] input: preadfd: read standard input byte-wise
Tue, 13 Dec 2022 23:17:32 +0100

<https://bugs.debian.org/862907>
"dash: Incorrectly slurps script from stdin (POSIX compliance issue)"

And curiously ssh command was involved as well
<https://michael-prokop.at/blog/2017/05/18/debugging-a-mystery-ssh-causing-strange-exit-codes/>
"Debugging a mystery: ssh causing strange exit codes?"

A citation from POSIX (XCU):

<https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sh.html#tag_20_117_06>
STDIN in /sh - shell, the standard command language interpreter/

When the shell is using standard input and it invokes a command that
also uses standard input, the shell shall ensure that the standard
input file pointer points directly after the command it has read when
the command begins execution. It shall not read ahead in such a manner
that any characters intended to be read by the invoked command are
consumed by the shell (whether interpreted by the shell or not) or
that characters that are not read by the invoked command are not seen
by the shell. When the command expecting to read standard input is
started asynchronously by an interactive shell, it is unspecified
whether characters are read by the command or interpreted by the
shell.


I do not think any script should rely on this behavior, it is better to 
explicitly use "here documents"


cat <Just a reminder: reading from stdin in shell scripts may interfere with 
various commands


<https://mywiki.wooledge.org/BashFAQ/089>
"I'm reading a file line by line and running ssh or ffmpeg, only the 
first line gets processed!"


So either "ssh -n" or "ssh I agree with Ihor that ob-shell should not feed scripts to shell stdin 
(maybe besides the case when it is explicitly requested by the user 
through some header arguments).


It seems, shell sessions
<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=67259>
is a different case unrelated to script files.

Re: does org mode require a separate LaTeX installation to export?

[Max Nikulin]

On 19/06/2024 19:55, Christopher W. Ryan wrote:

In other words, if a complete novice, knowing nothing about LaTeX and
not having it on their machine ever, installed emacs and created an org
mode file, could they export via latex to PDF?  Could they export to html?


For HTML export, LaTeX may be required if you need math included as 
images, some MathML converters may need LaTeX as well. On the other hand 
MathJax relies on client JavaScript in browsers.


It is possible to convert HTML to PDF e.g. by printing it from a 
browser. Chrome/chromium have command line options to do it without 
starting GUI. Default CSS is likely not optimal for this workflow. 
Unless other factors are involved, I expect better quality of documents 
generated using LaTeX.

Re: [PATCH] oc-csl: New custom option `org-cite-csl-sentence-case-bibtex-titles'

[Max Nikulin]

On 19/06/2024 16:27, András Simonyi wrote:

On Wed, 15 May 2024 at 13:48, Max Nikulin wrote:


I had a hope that it might alleviate the issue and to make things
working out of the box for more users.

[...]

Of course it could be checked whether
a CSL style contains explicit instructions about how to format titles
and skip  sentence-case conversion of input for those but this
probably would not
solve the problems of users who do not want the conversion at all.


Do you mean using title (original) case despite some style requites 
sentence case for English entries? I believed that fields should be 
surrounded with double braces {{...}} this case and the only issue is 
difference of original capitalization and result of conversion to 
sentence case and then back to title case.



It is not clear for me why `org-cite-csl-sentence-case-bibtex-titles' is
a part of Org, not of citeproc-el. The only thing that Org can do is to
pass it to citeproc-el. It is not configurable per .org file and likely
it should not be. From my point of view it might be more suitable per
.bib file. Anyway it is almost unrelated to Org.


citeproc-el doesn't have customizable user options by design because
it was conceived as a relatively low level
rendering library. Also, it's used by several user-facing packages by
now in addition to Org (org-ref, citar etc.) so
I think it would be both confusing and difficult to try to keep this
type of customizations in the library.


My expectation is that to get consistent results across applications, 
they should share the same preference. However since your choice is 
conscious, I do not insist.

Re: [BUG] Trailing dash is not included in link [9.7.3 (9.7.3-2f1844 @ /home/mwillcock/.emacs.d/elpa/org-9.7.3/)]

[Max Nikulin]

On 16/06/2024 22:59, Ihor Radchenko wrote:

Max Nikulin writes:


I suspect, it worked prior to v9.5. Without a unit test it may be
accidentally broken again.


No, it did not work.
If you can, please do not make such assertions without testing.


I am sorry, I had no intention to offend you. I missed that the removed 
line with explicit list of punctuation characters was commented out. I 
have tried the regexp used before (a part of v6.34)


facedba05 2009-12-09 15:13:50 +0100 Carsten Dominik: Use John 
Gruber's regular expression for URL's


and it seems trailing dash was allowed.


+: https://domain/test-


example.org, example.net, example.com are domains reserved for usage in
examples:
<https://www.iana.org/assignments/special-use-domain-names/special-use-domain-names.xhtml>


And so?


http://example.org/dash- may be a bit better for docs. (For IPv6 
addresses the difference should be more noticeable, but I do not 
remember what range is reserved for usage in examples there.)



I have realized that some Org regexps use [:punct:] *regexp class* and
others *syntax class*, see latex math regexp. I am in doubts if the
discrepancy is intentional.


It is not intentional, but using syntax classes can sometimes be
fragile.


Do you mean that result depends on current buffer? I do not have strong 
opinion what variant should be used. What I do not like is that in the 
case of $n$-th the character after second "$" is tested against syntax 
class, while regexp class is used for links. This subtle difference is 
almost certainly ignored in alternative implementations of the parser. 
However I am not sure what characters besides dash and apostrophe are 
affected and whether it depends on locale.



09ced6d2c 2024-02-03 15:15:46 +0100 Ihor Radchenko: org-link-plain-re:
Improve regexp heuristics

[...]

  (link http://example.org/a
[...]

It is heuristics. We cannot be 100% right. So, it is what it is.


From my point of view it is at least close to a regression. I do not 
have any argument against http://example.org/a, but the regexp should 
not match whole "http://example.org/a

[...]

Nowadays it is likely better to inspect
autolinking code for GitHub/GitLab or widely used python packages.


If you have concrete proposals, please share them.


Not yet. I consider inspecting mozilla's code as a kind of negative 
result from the point of view of usefulness for Org. Expanding test 
suite by gathering examples of failed heuristics from bug reports 
require enough reports. https://wpt.live/url/resources/urltestdata.json 
(https://github.com/web-platform-tests/wpt) is too specific for browsers 
and HTML/JS.



I would consider [:space:] or \s-.


Do you mean "[^[:punct:][:space:]\t\n]"?


I believe it might be an improvement ([:space:] includes \t).

Re: [BUG] Trailing dash is not included in link [9.7.3 (9.7.3-2f1844 @ /home/mwillcock/.emacs.d/elpa/org-9.7.3/)]

[Max Nikulin]

On 14/06/2024 21:04, Ihor Radchenko wrote:

Morgan Willcock writes:


i.e. Inserting "https://domain/test-"; into the buffer will create a
clickable link for "https://domain/test";.


I improved the heuristics we use to detect plain links.
Fixed, on main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=73da6beb5



+++ b/etc/ORG-NEWS

[...]

+*** Trailing =-= is now allowed in plain links


After a look into

7dcb1afb6 2021-03-24 21:27:24 +0800 Ihor Radchenko: Improve 
org-link-plain-re


I suspect, it worked prior to v9.5. Without a unit test it may be 
accidentally broken again.



+: https://domain/test-


example.org, example.net, example.com are domains reserved for usage in 
examples: 




(or (regexp "[^[:punct:] \t\n]")


I have realized that some Org regexps use [:punct:] *regexp class* and 
others *syntax class*, see latex math regexp. I am in doubts if the 
discrepancy is intentional.


I have noticed that the following change

09ced6d2c 2024-02-03 15:15:46 +0100 Ihor Radchenko: org-link-plain-re: 
Improve regexp heuristics


that causes

(link http://example.org/a
(link href="http://example.org/a%3Cb)">http://example.org/a%3Cb)


I expect that ")" should not be parsed as a part of the link. Balanced 
brackets are tricky with regexps (and it is not possible to match 
arbitrary nested ones).


Perhaps "[^[:punct:] \t\n]" is too strict in respect to spaces. It does 
not allow the recommended workaround with zero width space:


(org-export-string-as
 "http://example.org\N{ZERO WIDTH SPACE}[fn::footnote]" 'html 'body)
"
href=\"http://example.org​[fn::footnote]\";>http://example.org​[fn::footnote]

"

Actually some kind of non-breakable space should be better in such cases:

(org-export-string-as
 "http://example.org\N{NO-BREAK SPACE}[fn::footnote]" 'html 'body)
"
href=\"http://example.org [fn::footnote]\";>http://example.org [fn::footnote]

"

I would consider [:space:] or \s-.

As to the original bug report, while reading it, I noticed that 
thunderbird includes dash into the recognized link for


  "https://domain/test-";

I decided to look into its implementation and to my surprise I found: 
``punctation chars and "-" at the end are stipped off.'' I realized that 
double quotes along with angle brackets are treated as a recommended way 
to mark URLs in plain text. Thunderbird does not consider dash as a part 
of links for e.g. http://example.org/t- It might be an attempt to 
reserve possibility to assemble URLs wrapped into several lines with 
added hyphenation marks, but it has not been implemented (RFC2396 
appendix E warns about accidentally added hyphens).


https://www.bucksch.org/1/projects/mozilla/16507/
https://searchfox.org/mozilla-central/source/netwerk/streamconv/converters/mozTXTToHTMLConv.cpp#line-243
mozTXTToHTMLConv::FindURLEnd

Implementation is tricky, I have not noticed anything that may be reused 
to improve heuristics for Org. Nowadays it is likely better to inspect 
autolinking code for GitHub/GitLab or widely used python packages.

Re: org-babel-execute-src-block filters characters from :session *shell* output

[Max Nikulin]

On 15/06/2024 20:19, Ihor Radchenko wrote:

The underlying cause is limitation of Emacs API for interactive shells -
we cannot easily distinguish command output from prompt and other extra
staff your shell/other interactive command spits into the buffer.
So, we have to either filter output the prompts ourselves to get the
command output reliably or redirect output to files, where nothing
litters the actual output with prompts.


Some shells support "semantic shell" that allows terminal applications 
e.g. to copy whole command output. It is based on escape sequences.


- 
https://docs.kde.org/stable5/en/konsole/konsole/semantic-shell-integration.html
- 
https://gitlab.freedesktop.org/Per_Bothner/specifications/blob/master/proposals/semantic-prompts.md

- https://github.com/tmux/tmux/issues/3064

Re: [PATCH] orgweb: Update org-ruby link (tools)

[Max Nikulin]

On 09/06/2024 02:18, Ihor Radchenko wrote:

Max Nikulin writes:

... I have tried to blindly update zh-CN file. Likely it should be
done more accurately.


Yes. Let's not fiddle with Chinese version. It is already out of sync,
but we cannot do much unless we get someone Chinese-speaking volunteer.


Feel free to skip chunks you do not like.




Would it be acceptable to add an additional link to the dedicated page
on Worg?
[[https://orgmode.org/worg/org-tutorials/org-ruby.html][Worg]]


Yes, but I'd prefer if such link would be not just a text link, but
something more compact. Maybe something like "wiki" button on top-left
corner of the logo.


I do not like and idea of a button over an active link.

Since org-ruby has no its own logo I would consider 3 smaller links instead:
- https://github.com/wallyqs/org-ruby with the github logo
- https://rubygems.org/gems/org-ruby with the current gem icon
- https://orgmode.org/worg/org-tutorials/org-ruby.html with styled 
"Worg" text.



Note that the extended tools page on
https://orgmode.org/worg/org-tools/index.html already has the link you proposed.


Formally it is more appropriate for 
https://orgmode.org/worg/org-translators.html "Org-mode Import/Export 
Tools". (I have no idea why this page is not in the org-tools folder.)


From my point of view, when a user sees the ruby icon nothing suggests 
that, besides clicking on the link, they may find additional info on a 
page linked from another page accessible through a link at the bottom of 
the current one.

[PATCH] orgweb: Update org-ruby link (tools)

[Max Nikulin]

Hi,

It seems the org-ruby project link on the https://orgmode.org/tools.html 
page is outdated and should be updated to the repository of its current 
maintainer.


I suggest to move the link to the Import&Export section since the 
primary purpose of the project is conversion to HTML, Markdown, or 
textile. I have tried to blindly update zh-CN file. Likely it should be 
done more accurately.


Would it be acceptable to add an additional link to the dedicated page 
on Worg?

[[https://orgmode.org/worg/org-tutorials/org-ruby.html][Worg]]From 65f797a418d73077171b5f74b5930179939e7a3f Mon Sep 17 00:00:00 2001
From: Max Nikulin 
Date: Sat, 8 Jun 2024 23:01:56 +0700
Subject: [PATCH 1/2] tools.org: Update link to org-ruby

* tools.org:
* zh-CN/tools.org: Update org-ruby project page.

Use link from <https://rubygems.org/gems/org-ruby>
---
 tools.org   | 2 +-
 zh-CN/tools.org | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools.org b/tools.org
index 447ec57..525953a 100644
--- a/tools.org
+++ b/tools.org
@@ -232,7 +232,7 @@ ** Perl
 /Org​::parser/
 {{{end-link}}}
 ** Ruby
-{{{gh(bdewey/org-ruby)}}}
+{{{gh(wallyqs/org-ruby)}}}
 [[https://upload.wikimedia.org/wikipedia/commons/7/73/Ruby_logo.svg]]
 
 /org-ruby/ gem --- parse Org files and convert to HTML or textile. Used
diff --git a/zh-CN/tools.org b/zh-CN/tools.org
index aeca181..77f838f 100644
--- a/zh-CN/tools.org
+++ b/zh-CN/tools.org
@@ -131,7 +131,7 @@ ** Perl
 /Org​::parser/
 {{{end-link}}}
 ** Ruby
-{{{gh(bdewey/org-ruby)}}}
+{{{gh(wallyqs/org-ruby)}}}
 [[https://upload.wikimedia.org/wikipedia/commons/7/73/Ruby_logo.svg]]
 
 /org-ruby/ 是一个解析 Org 文件以及将其转换为文本格式以及 HTML 格式的 Ruby 库。此外，GitHub 使用这个库来显示网页上的 Org 文件。
-- 
2.39.2

From c38db1f3dfaf17f441347e4b1f12c6bd957413ec Mon Sep 17 00:00:00 2001
From: Max Nikulin 
Date: Sat, 8 Jun 2024 23:13:07 +0700
Subject: [PATCH 2/2] tools.org: Move org-ruby to export section

* tools.org:
* zh-CN/tools.org: Put org-ruby to exporters
---
 tools.org   | 14 +++---
 zh-CN/tools.org | 12 ++--
 2 files changed, 13 insertions(+), 13 deletions(-)

diff --git a/tools.org b/tools.org
index 525953a..89ce863 100644
--- a/tools.org
+++ b/tools.org
@@ -88,6 +88,13 @@ ** Pandoc
 The Swiss-army knife of markup format converters.
 It can convert documents between many source formats, including Org
 {{{end-link}}}
+** Org-ruby
+{{{gh(wallyqs/org-ruby)}}}
+[[https://upload.wikimedia.org/wikipedia/commons/7/73/Ruby_logo.svg]]
+
+Gem and CLI tool to convert Org files to HTML, Markdown, or textile. Used
+by GitHub
+{{{end-link}}}
 * Publishing
 :PROPERTIES:
 :HTML_CONTAINER_CLASS: columns
@@ -231,13 +238,6 @@ ** Perl
 
 /Org​::parser/
 {{{end-link}}}
-** Ruby
-{{{gh(wallyqs/org-ruby)}}}
-[[https://upload.wikimedia.org/wikipedia/commons/7/73/Ruby_logo.svg]]
-
-/org-ruby/ gem --- parse Org files and convert to HTML or textile. Used
-by GitHub
-{{{end-link}}}
 ** Rust
 {{{gh(poiscript/orgize)}}}
 [[https://upload.wikimedia.org/wikipedia/commons/d/d5/Rust_programming_language_black_logo.svg]]
diff --git a/zh-CN/tools.org b/zh-CN/tools.org
index 77f838f..95519f3 100644
--- a/zh-CN/tools.org
+++ b/zh-CN/tools.org
@@ -37,6 +37,12 @@ ** Pandoc
 
 瑞士军刀级别的 markup 文档转换器：它能将文档在不同格式之间自由转换，包括 Org。
 {{{end-link}}}
+** Org-ruby 是
+{{{gh(wallyqs/org-ruby)}}}
+[[https://upload.wikimedia.org/wikipedia/commons/7/73/Ruby_logo.svg]]
+
+个解析 Org 文件以及将其转换为文本格式以及 HTML 格式的 Ruby 库。此外，GitHub 使用这个库来显示网页上的 Org 文件。
+{{{end-link}}}
 * 发布
 :PROPERTIES:
 :HTML_CONTAINER_CLASS: columns
@@ -130,12 +136,6 @@ ** Perl
 
 /Org​::parser/
 {{{end-link}}}
-** Ruby
-{{{gh(wallyqs/org-ruby)}}}
-[[https://upload.wikimedia.org/wikipedia/commons/7/73/Ruby_logo.svg]]
-
-/org-ruby/ 是一个解析 Org 文件以及将其转换为文本格式以及 HTML 格式的 Ruby 库。此外，GitHub 使用这个库来显示网页上的 Org 文件。
-{{{end-link}}}
 ** Rust
 {{{gh(poiscript/orgize)}}}
 [[https://upload.wikimedia.org/wikipedia/commons/d/d5/Rust_programming_language_black_logo.svg]]
-- 
2.39.2

Re: Incremental search in hidden text does not work in Org 9.7 Emacs 28

[Max Nikulin]

On 06/06/2024 01:06, Ihor Radchenko wrote:

Remember that older Org mode also did not have link paths searchable.


You are right, I forgot about it.

Since the feature was advertised in ORG-NEWS


 Hidden parts of the links can now be searched and revealed during isearch
another entry should be added that it does not work any more. I would 
mark this heading as an obsolete one as well.

[BUG] M-RET may cause hang in the case of nested list

[Max Nikulin]

Hi,

I experience hangs after hitting M-RET. Org main HEAD or 9.5, Emacs-28

 8< 
- First level item
  - nested item
  after nested item


 >8 

M-RET causes 100% CPU load. If I hit C-g then "after nested item" 
disappears and new item is created on that line.

Re: [PATCH] Add support for shortdoc link type

[Max Nikulin]

On 13/05/2024 20:04, Bruno Cardoso wrote:

+++ b/etc/ORG-NEWS


Bruno, I am sorry, but Org-9.7 has been released and your patch can not 
be cleanly applied any more. Please, rebase it and check that the new 
feature is described in the proper section for changes that have not 
released yet.


A couple of notes that unlikely affect real life usage.


+  (defun org-link--store-shortdoc (&optional _interactive?)
+"Store \"shortdoc\" type link."
+(when (eq major-mode 'shortdoc-mode)


`derived-mode-p' is more reliable in general.


+  (let* ((buffer (buffer-name))
+ (group (when (string-match "*Shortdoc \\(.*\\)\\*" buffer)
+  (match-string 1 buffer
+(if (and group (assoc (intern-soft group) shortdoc--groups))
+(org-link-store-props :type "shortdoc"
+  :link (format "shortdoc:%s" group)
+  :description nil)
+  (user-error "Unknown shortdoc group: %s" group)


I believed that store link functions should fail silently if no suitable 
link may be offered.

Re: Incremental search in hidden text does not work in Org 9.7 Emacs 28

[Max Nikulin]

On 05/06/2024 22:06, Ihor Radchenko wrote:

`org-fold-core-style' text-properties is going to be obsoleted.
I tried to support isearch, but it is ultimately an uphill battle
against bugs popping out here and there.


I have seen these words in another thread, but I did not expect a 
regression in comparison to 9.6.


At least search in folded headers and drawers works, but without search 
in link targets 9.7 is hardly usable for daily usage for me. I have not 
tested yet to which degree overlays are painful for my notes file.

Incremental search in hidden text does not work in Org 9.7 Emacs 28

[Max Nikulin]

Hi,

Consider a file with a link
 8< 
Text

[[file:text.org][Something]]
 >8 

Incremental search
   C-s file
does not work with current main HEAD (v9.7) and Emacs-28.

`org-fold-core-style' text-properties

There is no issue for Org 9.6.28. Link part is temporary revealed when 
search is active.

Re: question about links, macros + org-publish and "parametrization"

[Max Nikulin]

On 30/05/2024 23:33, Ihor Radchenko wrote:

Max Nikulin writes:


I can not figure out if it is possible with some trick to expand
{{{year}}} before file:... is parsed.


Not possible. Macros are not recognized in verbatim contexts like link
path.


I had in mind some technique like \expandafter in TeX or intermediate 
macro in the case of C preprocessor.


 8< 
#+macro: mend }}}
#+macro: year 24
#+macro: source [[file:./$1/t.org][for the year $1]]
#+macro: target {{{source({{{year}}}){{{mend}}}

{{{source(24)}}}
{{{target}}}
{{{source({{{year}}}){{{mend}}}
 >8 

It signals

(error "Undefined Org macro: source; aborting")

during processing source code blocks. Export buffer content at this step:
 8< 
#+macro: mend }}}
#+macro: year 24
#+macro: source [[file:./$1/t.org][for the year $1]]
#+macro: target {{{source({{{year}}}){{{mend}}}

[[file:./24/t.org][for the year 24]]
{{{source(24)}}}
{{{source(24)}}}
 >8 

So I have almost achieved the goal despite the approach is rather 
fragile. I might work with additional pass of macro processing added as 
an extra export filter.


However I believe it is better to use either a custom link type or an 
eval macro instead of set of kludges with postponing expansion.


It is not intuitive that macro parameters are not eagerly expanded like 
function arguments in most common procedural programming languages. 
Macro languages have their own shortcomings.

Re: question about links, macros + org-publish and "parametrization"

[Max Nikulin]

On 30/05/2024 12:03, Martin Steffen wrote:

#+macro: year 24
#+macro: source [[file:./$1/t.org][for the year $1]]
#+macro: target {{{source({{{year}}})}}}


Custom link type might be more convenient

(info "(org) Adding Hyperlink Types")
https://orgmode.org/manual/Adding-Hyperlink-Types.html

I can not figure out if it is possible with some trick to expand 
{{{year}}} before file:... is parsed.

Re: Adding custom providers for thingatpt.el

[Max Nikulin]

On 21/05/2024 23:24, Jim Porter wrote:

On 5/21/2024 3:32 AM, Max Nikulin wrote:

On 20/05/2024 09:33, Jim Porter wrote:

+++ b/lisp/org.el

[...]

+(require 'thingatpt)


So it becomes hard dependency. However it seems thingatpt is anyway 
loaded through some indirect dependency.


Probably through ol-eww.


I do not use ol-eww, so I do not have it in org-modules. The reason is

#+begin_src sql

intended just to have a code example with syntax highlighting. It causes 
loading of sql.el that may call `thing-at-point-looking-at' from 
`sql-read-table-name'.


I have no idea concerning opinion of developers, but I would prefer to 
have loading of libraries like thingatpt on demand.

Re: Saving some kitten, plus some questions along the way

[Max Nikulin]

On 20/05/2024 04:24, Stefan Monnier wrote:

Subject: [PATCH 3/4] testing/org-test.el (): Remove dead code

(featurep 'org) is always non-nil here since we have a (require 'org)
further up.  I suspect other `require`s nearby could be removed or
moved to toplevel.


Just a guess. Perhaps earlier there was intentionally no (require 'org) 
to ensure that org-test.el loads Org from the sibling directory, not 
from the version bundled with Emacs. Stefan, it was you who added 
(require 'org) during switching to lexical binding.

Re: Using a custom environment for a src language

[Max Nikulin]

On 20/05/2024 08:57, Rustom Mody wrote:

I have my own Python environment called PYT.
#+BEGIN_PYT
contents
#+END_PYT


See `org-latex-custom-lang-environments' for #+begin_src python.

Re: Adding custom providers for thingatpt.el

[Max Nikulin]

On 20/05/2024 09:33, Jim Porter wrote:

+++ b/lisp/org.el
@@ -81,6 +81,7 @@
 (require 'calendar)
 (require 'find-func)
 (require 'format-spec)
+(require 'thingatpt)


So it becomes hard dependency. However it seems thingatpt is anyway 
loaded through some indirect dependency.

Re: org-babel shell in windows not finishing command

[Max Nikulin]

On 21/05/2024 01:37, Ihor Radchenko wrote:

echo 'echo foo' | cmdproxy.exe yields

: Microsoft Windows [Version 10.0.19045.2251]
: (c) Microsoft Corporation. All rights reserved.
:
: c:\Users\johndoe\Org>echo foo
: foo


Out of curiosity, does the /q option help to suppress the banner?

Re: [DISCUSSION] The meaning of :cmdline header argument across babel backends

[Max Nikulin]

On 21/05/2024 01:01, Matt wrote:

I'm okay with these.  I can start on a patch for :script-args and
:program-args within ob-shell.


Frankly speaking your plan is not clear for me. My special concern is 
DWIM behavior

https://list.orgmode.org/874jbkcmyg.fsf@localhost
(Ihor Radchenko Mon, 29 Apr 2024 13:33:59 +)
and

 #+begin_src sh :script-args 1 ; touch /tmp/not-an-arg

if you are going to pass it literally to "sh -c" then it is 
:script-cmdline rather than :script-args.


I expect a way to explicitly specify if it is a single argument or 
multiple ones


#+begin_src sh :script-args '("a b c")

vs.

#+begin_src sh :script-args '("a" "b" "c")

preferably passed as separate arguments of process-files or at least 
properly escaped.


As to literal command line, taking into account stripped outer quotes 
issue, I do not like requirement to quote characters for shells. Even 
splitting string into arguments using `read' might be better, but there 
are still enough issues.


Besides interpreters, there is may be a stack of "launchers" like 
toolbox in the case of applications installed as isolated flatpak/snap 
packages:


Florin Boariu to emacs-orgmode. org-ditaa woes. Thu, 19 Oct 2023 
12:59:59 +0200.

https://list.orgmode.org/ZTEML8zWrB6kQflk@toolbox

Re: [PATCH] Add support for shortdoc link type

[Max Nikulin]

On 13/05/2024 20:04, Bruno Cardoso wrote:

On 2024-05-13, 18:14 +0700, Max Nikulin wrote:


Sorry, but did you manually edit the patch?


I didn't. I generated it again and applied without errors.


Thank you. Now your patch can be applied on my side as well.

Please, proceed with the copyright agreement form.

Re: [POLL] Dealing with +1m/y repeaters when jumping to impossible date (should 05-31 +1m be 07-01 or 06-30?)

[Max Nikulin]

On 14/05/2024 19:56, Ihor Radchenko wrote:

Max Nikulin writes:


+(defun org-time-inc (unit value time)

Is there a chance that support of intervals like 1h20m will be required
later?


Not sure again. I simply used ts-inc function signature from Adam's
ts.el as reference.


ts.el has `ts-adjust' that may change several fields. I had in mind 
specification of time interval from RFC5545 iCalendar P4h20m.



+(if (memq unit '(day month year))
+nil ; Avoid auto-adjustments of time when jumping across DST.


Sorry, but you have to use -1, otherwise

(format-time-string
   "%F %T %Z %z"
   (encode-time '(0 30 12 15 1 2000 'ignored nil "Africa/Juba"))
   "Africa/Juba")
(error "Specified time is not representable")


Hmm. I am not sure if it is a real problem in practice.


You are right. When you are using TZ as integer offset from UTC, 
`encode-time' does not signal any error. It ignores DST. However it is 
the case of conversion to local time a timestamp with arbitrary explicit 
offset, e.g. taken from a Date email header. So my example is wrong for 
current patch revision, but the code needs a fix.


You need to get local timestamp for given local time. You should use -1 
for DST and nil for TZ.


In some cases it may cause change like 1:30 to 2:30, but it is not worse 
than date drift from 31 to 30 or even 28.



The reason why I forced DST nil is hysteresis of glibc when dealing with
DST transitions:


As I wrote yesterday, you example is unrelated to glibc hysteresis. The 
reliable way to deal with ambiguous time close to time transition is to 
have get next transition function for given timezone and a way to 
resolve ambiguous time (DST can not help in some cases). Heuristics with 
trying a day before and a day after should work, but it may impact 
performance.


Your patch allows users to choose if they prefer May, 31 to July, 1 or 
to July, 30 drift. It may be an improvement some of them.


I think, they actually expect choice to get June, 30 and July, 31 or to 
ignore June since it has 30 days only.

Re: [PATCH] oc-csl: New custom option `org-cite-csl-sentence-case-bibtex-titles'

[Max Nikulin]

On 11/05/2024 22:33, András Simonyi wrote:

since bibtex and biblatex requires title fields to be in title case
but CSL assumes that they are in sentence-case, citeproc-el converts
title fields in bib(la)tex bibliography databases into sentence-case
before processing them except for entries with an explicit non-English
langid value.


I am not a user of citeproc-el, so feel free to disregard my comments.

In the past I had to adjust BibTeX styles, but yesterday I was surprised 
that there are options for upper case, lower case, and sentence-style 
capitalization, but not for title-style capitalization. It seems that 
both approaches with title case and with sentence case have some 
shortcomings. Likely title case like in BibTeX requires more explicit 
hints and perhaps there are cases when available hints are not enough to 
get specific formatting. I still expect that CSL needs hints as well to 
avoid improper formatting.


Is it possible to keep title formatting from .bib files till it becomes 
known that specific style requires sentence case for particular entry 
type? I had a hope that it might alleviate the issue and to make things 
working out of the box for more users.



I'm a bit unsure about naming the option:
Perhaps `org-cite-csl-sentence-case-bibtex-titles-without-langid'


A variant: org-cite-csl-bibtex-title-to-sentence-case


@@ -584,7 +599,8 @@ property in INFO."
  (processor
   (citeproc-create
(org-cite-csl--style-file info)
-   (citeproc-hash-itemgetter-from-any bibliography)
+   (citeproc-hash-itemgetter-from-any
+bibliography (not org-cite-csl-sentence-case-bibtex-titles))
(org-cite-csl--locale-getter)
locale)))
 (plist-put info :cite-citeproc-processor processor)


I am not in the context, so I may be completely wrong.

Does it means that you added one more argument to `citeproc-create' and 
that consistent Org and citeproc-el versions must be used? If so, 
wouldn't it better to pass a property list to allow newer Org to work 
with older citeproc-el or vice versa? It may be tricky to preserve 
backward-forward compatibility on this step, but it should make further 
changes easier. It may be reasonable to explicitly add version of 
"protocol" to the property list, so that citeproc-el may decide if error 
should be signaled in the case of serious version difference.


It is not clear for me why `org-cite-csl-sentence-case-bibtex-titles' is 
a part of Org, not of citeproc-el. The only thing that Org can do is to 
pass it to citeproc-el. It is not configurable per .org file and likely 
it should not be. From my point of view it might be more suitable per 
.bib file. Anyway it is almost unrelated to Org.

Re: [POLL] Dealing with +1m/y repeaters when jumping to impossible date (should 05-31 +1m be 07-01 or 06-30?)

[Max Nikulin]

On 14/05/2024 19:56, Ihor Radchenko wrote:

Max Nikulin writes:

On 13/05/2024 17:07, Ihor Radchenko wrote:


<2025-01-31 Fri +1m>
<2025-02-28 Fri +1m>
<2025-03-28 Fri +1m>


Instead of using timestamp obtained on previous step, use original
timestamp and multiple of the interval.


This is not possible because of how `org-auto-repeat-maybe' is designed.


Then the only way to get reasonable results is to store in :PROPERTIES: 
either original date (and iteration count) or current date before 
clamping (2025-02-31)



No idea. I am not aware about any existing built-in API that provides
date increments.


The context was that day start time may give some surprise as well. I am 
unsure concerning namely increments. I just expected a set of more 
accurate functions for working with dates.



+  (org-encode-time

[...]

+(+ (if (eq unit 'day) value 0)(decoded-time-day time))


Have you considered using `min' with result of `date-days-in-month' here
(or its sibling from timezone.el)?


Not sure if it is going to be simpler.


My idea was to avoid the backward `while' loop in the code below this line.


(format-time-string
   "%F %T %Z %z"
   (encode-time '(0 30 12 15 1 2000 'ignored nil "Africa/Juba"))
   "Africa/Juba")
(error "Specified time is not representable")


Hmm. I am not sure if it is a real problem in practice.
String values of time zone are only possible for hand-crafted time
values.


This example avoids changing system time or starting Emacs with TZ 
environment. The following example does not include explicit timezone:


TZ=Africa/Juba emacs -Q --batch \
--eval "(encode-time '(0 30 12 15 1 2000 'ignored nil nil))"


The reason why I forced DST nil is hysteresis of glibc when dealing with
DST transitions:


At first I was trying to figure out what time zone had 24:00<->23:00 
transition in 2012.



(cl-loop for m in '(1 2 3 4 5 6 7 8 9 10 11 12)
   collect (decode-time (encode-time `(0 0 0 4 ,m 2012 nil -1 10800

(0 0 23 3 1 2012 2 nil 7200)


This result depends on your timezone. It is consistent with e.g. 
Asia/Istanbul. It had +02:00 offset in January, but you requested 
+03:00. The result of conversion is -1 hour difference in requested 
timestamp and conversion result. It is unrelated to internal DST state 
and hysteresis caused by this state.


When DST is nil
2012-01-04 00:00:00 +03:00 === 2012-01:03 23:00:00 +02:00

Avoid passing timezone offset when you do not know it.

I agree that there are issues with handling tm_gmtoff and tm_isdst
https://data.iana.org/time-zones/theory.html#POSIX-extensions
Results may vary across libraries.


Although, forcing DST nil will not always help here


It is fragile, I would not rely on ignoring offset when DST is 
specified. Actually you pass mutually inconsistent values, so it is 
either undefined behavior or close to it.

Re: [POLL] Dealing with +1m/y repeaters when jumping to impossible date (should 05-31 +1m be 07-01 or 06-30?)

[Max Nikulin]

On 13/05/2024 17:07, Ihor Radchenko wrote:


However, there are still some issues remaining.
When updating timestamps repeating monthly across months with 30, 31,
and 28 days we get

<2025-01-31 Fri +1m>
<2025-02-28 Fri +1m>
<2025-03-28 Fri +1m>
...
<2026-01-28 Wed +1m>


Instead of using timestamp obtained on previous step, use original 
timestamp and multiple of the interval.


I have the following in my notes. However I have not find suitable 
functions in elisp:


https://debbugs.gnu.org/54764#10
Paul Eggert. Sat, 9 Apr 2022 00:52:57 -0700
Generally speaking, 
when Org mode is doing calendrical calculations it should use 
calendrical functions rather than encode-time+decode-time, which are 
best used for time calculations not calendar calculations.


Do you have any idea what Paul was writing about?

On 13/05/2024 17:07, Ihor Radchenko wrote:

+   (type-unit (pcase type
+("h" 'hour) ("d" 'day) ("w" 'week)
+("m" 'month) ("y" 'year
I do not mind against `pcase' here, I just expected something like 
`assoc' and `string-to-char'.



+ ((or "m" "y")

Another comment that may be ignored: `type-unit' values might be used here.


+  (org-encode-time
+   (list
+(+ (if (eq unit 'second) value 0) (decoded-time-second time))
+(+ (if (eq unit 'minute) value 0) (decoded-time-minute time))
+(+ (if (eq unit 'hour) value 0)   (decoded-time-hour time))
+(+ (if (eq unit 'day) value 0)(decoded-time-day time))


Have you considered using `min' with result of `date-days-in-month' here 
(or its sibling from timezone.el)?



+(defun org-time-inc (unit value time)
Is there a chance that support of intervals like 1h20m will be required 
later?



+(+ (if (eq unit 'month) value 0)  (decoded-time-month time))
+(+ (if (eq unit 'year) value 0)   (decoded-time-year time))
+(decoded-time-weekday time)
+(if (memq unit '(day month year))
+nil ; Avoid auto-adjustments of time when jumping across DST.


Sorry, but you have to use -1, otherwise

(format-time-string
 "%F %T %Z %z"
 (encode-time '(0 30 12 15 1 2000 'ignored nil "Africa/Juba"))
 "Africa/Juba")
(error "Specified time is not representable")


+  (decoded-time-dst time))
+(decoded-time-zone time))
+(if (not org-repeat-round-time) new-time
I am in doubts if `org-time-inc' should access `org-repeat-round-time' 
directly or its value should be passed as an explicit argument. Perhaps 
the additional argument may be optional with fallback to 
`org-repeat-round-time' when it is omitted.

Re: [PATCH] Add support for shortdoc link type

[Max Nikulin]

On 11/05/2024 23:58, Bruno Cardoso wrote:

On 2024-05-10, 18:09 +0700, Max Nikulin wrote:

<https://orgmode.org/worg/org-contribute.html#copyright>


I haven't yet. Will be glad to.


I hope, next patch revision will be the final one, so it can be 
notification that the procedure is completed. (Rebase to main HEAD might 
be necessary.)



The attached patch is now in this format.



+++ b/lisp/ol.el
@@ -1582,7 +1582,47 @@ PATH is a symbol name, as a string."
  :follow #'org-link--open-help
  :store #'org-link--store-help)
 


Sorry, but did you manually edit the patch?

git apply /tmp/org/0001-ol-support-for-shortdoc-link-type.patch
error: patch failed: lisp/ol.el:1582
error: lisp/ol.el: patch does not apply

and I see only 6 context lines, not 7.


+  (condition-case nil
+  (progn
+(shortdoc-display-group group fn)
+(and str (not fn) (search-forward str nil t)))
+(error (message "Unknown shortdoc group or malformed link: `%s'"
+path)


If you wish to use `user-error' as in earlier revisions then you may use

(error (user-error "Unknown shortdoc group or malformed link: `%s'"
path)

An alternative might be something like (feel free to ignore)

  (condition-case err
;; ...
(error (signal 'user-error (cdr err ; more )

to preserve message generated by `shortdoc-display-group'.

Re: [PATCH] Add support for shortdoc link type

[Max Nikulin]

On 09/05/2024 04:26, Bruno Cardoso wrote:

On 2024-05-08, 18:20 +0700, Max Nikulin wrote:


I have not managed to get all known groups with `shortdoc-display-group'
in Emacs-28. Is it a feature of `shortdoc' in Emacs-29?


In Emacs-29 `shortdoc' is an alias for `shortdoc-display-group'.
Interactvely, it prompts the user for a group from `shortdoc--groups'
(that's what I meant by "all known groups").


Thank you for the explanation. Reading "lists all" I expected something 
like list of all info manuals in the case of M-x info RET when there are 
no info buffers yet.



"\\`\\([^:]+\\)\\(?:::\\(.+\\)\\'\\)?"


`shortdoc-display-group' already handles empty group or function.
Replacing "*" by "+" will throw an "Args out of range" error in the
corner case you mentioned ("shortdoc:::file"). So I think it's fine to
keep "*" there.


Out of range error happens because of `string-match' is called despite 
`match-string' failure and match groups from some earlier regexp are 
applied. Generally `match-string' should be inside `if', `when', `and', 
etc., however I agree that the regexp with stars always succeeds, 
perhaps really matching nothing. That is why I do not insist.


The following garbage in - garbage out case is a bit confusing:

=> "No such documentation group "


ol.el: Add support for `shortdoc' link type


Bruno, has you signed the copyright form? This patch is above the 
TINYCHANGE limit.

<https://orgmode.org/worg/org-contribute.html#copyright>

It will be a bit less work for the maintainer if you attach result of 
"git format-patch" command. It adds email-like headers allowing to 
properly set commit author.



* lisp/ol.el: Add support for storing and inserting links to `shortdoc'
documentation groups for Emacs Lisp functions.


List of new functions is missed:

* lisp/ol.el (org-link--open-shortdoc org-link--store-shortdoc)
(org-link--complete-shortdoc): Add support...


* doc/org-manual.org (External Links): Add shortdoc link type
documentation.

* etc/ORG-NEWS (=ol.el=: Support for =shortdoc= link type): Document
the new feature.


Empty line is separator before free-form comments, so list of formal 
changes should be dense.



+  (condition-case nil
+  (progn
+(shortdoc-display-group group fn)
+(and str (not fn) (search-forward str nil t)))
+(user-error "Unknown shortdoc group: %s" group


Sorry, it does not work so. It means catch `user-error' however 
`shortdoc-display-group' signals `error'. Try to enable debug-on-error 
and to open a broken link. Either `condition-case' should be dropped or 
proper signal should be captured.

Re: [PATCH] Fix regex for determining image width from attribute

[Max Nikulin]

On 08/05/2024 17:54, Ihor Radchenko wrote:

Max Nikulin writes:


#+attr_html: :alt Image width test
#+attr_beamer: :width \linewidth
#+attr_latex: :width +.5\textwidth
#+attr_md: :width 75%
[[file:babelfish.png]]

- I do not mind that just "\linewidth" is ignored.
- The case of "+.5" should either be supported or at least documented
since it is a valid floating number.


Was it supported before?


No. It is my expectation based on earlier discussions that fractional 
numbers were added namely for LaTeX. I used leading dot in 
\includegraphics[width=...]{}, so I was surprised that Org ignores it. 
Explicit "+" is just generalization, this form is supported by LaTeX and 
elisp.



- It is really confusing that #+attr_html casts shadow on #+attr_md.


No, it is not. I see no reason to prioritize markdown attributes.


Above #+attr_html does not specify :width while #+attr_md has a valid 
value. It is the reason why I do like current behavior.



+++ b/etc/ORG-NEWS


I think, it is better to avoid "is ignored" here.


May you convert your suggestion into a patch?


See the attachment.From 7bc9d909867bf2f99a77d5d1554cd41e4fc664ae Mon Sep 17 00:00:00 2001
From: Max Nikulin 
Date: Thu, 9 May 2024 17:32:54 +0700
Subject: [PATCH] ORG-NEWS: Reword inline image width note

* etc/ORG-NEWS: Avoid possible confusion related to
"#+attr_org: :width" example.
---
 etc/ORG-NEWS | 12 ++--
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 3c597db40..a34307295 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -15,16 +15,16 @@ Please send Org bug reports to mailto:emacs-orgmode@gnu.org.
 ** Important announcements and breaking changes
 *** Inline image width value in =#+attr_org= is preferred over other =#+attr_...= keywords
 
-Previously, when ~org-image-actual-width~ is a list, Org used the
+Previously, when ~org-image-actual-width~ is a list or nil, Org used the
 first =#+attr_...= keyword containing =:width ...= to compute the inline
-image width:
+image width.  Now, =#+attr_org=, if present, takes precedence.
+In the following example the image preview has width of 75%
+while earlier versions pick 33%.
 
-: #+attr_html: :width 30%
-: #+attr_org:  :width is ignored
+: #+attr_html: :width 33%
+: #+attr_org:  :width 0.75
 : [[image.png]]
 
-Now, =#+attr_org=, if present, takes precedence.
-
 *** =ox-html=: When exporting footnotes with custom non-number names, the names are used as link anchors
 
 Previously, link anchors for footnote references and footnote
-- 
2.39.2

Re: [PATCH] Add support for shortdoc link type

[Max Nikulin]

On 08/05/2024 08:11, Bruno Cardoso wrote:


The following now work as expected:

- [[shortdoc:string]]
- [[shortdoc:string::#try-completion]]
- [[shortdoc:string::identical contents]]


Thanks.

Bruno, the patch should have proper commit message, see


Some minor notes:


+++ b/doc/org-manual.org
@@ -3382,6 +3382,15 @@ Here is the full set of built-in link types:
 
   Execute a shell command upon activation.
 
+- =shortdoc= ::

+
+  Link to short documentation summary for an Emacs Lisp function group.
+  Since Emacs 28, user command ~shortdoc-display-group~ lists all known
+  documentation groups.
I have not managed to get all known groups with `shortdoc-display-group' 
in Emacs-28. Is it a feature of `shortdoc' in Emacs-29?



+(string-match "\\`\\([^:]*\\)\\(?:::\\(.*\\).*\\'\\)?" path)


My bad. In
"\\`\\([^:]*\\)\\(?:::#\\(.*\\).*\\'\\)?"
second .* was unnecessary, it survived from an earlier variant with \\| 
and \\' outside of the regex group.


Likely it is better to not allow empty group or function, so "+" instead 
of "*"


"\\`\\([^:]+\\)\\(?:::\\(.+\\)\\'\\)?"

since  is hardly a valid link. However it might be 
search text link in the list of all groups (Emacs-29+ only?).


It is related to corner cases, so I do not insist.


+(let* ((group (match-string 1 path))
+   (str (match-string 2 path))
+   (fn (when (and str (string-match "^#" str))
+ (intern-soft (string-remove-prefix "#" str)


Nitpick, feel free to ignore:
- `when' is redundant here due to `and' inside its condition.
- `string-patch-p' would not affect match groups, so it is a bit cheaper.
- Since it is already known that first character is "#", `substring' 
instead of `string-remove-prefix' is an extra step in microoptimization


Untested: (fn (and str (eq ?# (string-to-char str)) (substring str 1)))

Re: [PATCH] Fix regex for determining image width from attribute

[Max Nikulin]

On 02/05/2024 23:28, Ihor Radchenko wrote:


1. #+attr_org is prioritised


I ma afraid, the code is a bit fragile. Consider

#+attr_html: :alt Image width test
#+attr_beamer: :width \linewidth
#+attr_latex: :width +.5\textwidth
#+attr_md: :width 75%
[[file:babelfish.png]]

- I do not mind that just "\linewidth" is ignored.
- The case of "+.5" should either be supported or at least documented 
since it is a valid floating number.

- It is really confusing that #+attr_html casts shadow on #+attr_md.


+++ b/etc/ORG-NEWS
@@ -13,6 +13,18 @@ Please send Org bug reports to mailto:emacs-orgmode@gnu.org.
 
 * Version 9.7 (not released yet)

 ** Important announcements and breaking changes
+*** Inline image width value in =#+attr_org= is preferred over other 
=#+attr_...= keywords
+
+Previously, when ~org-image-actual-width~ is a list, Org used the
+first =#+attr_...= keyword containing =:width ...= to compute the inline
+image width:
+
+: #+attr_html: :width 30%
+: #+attr_org:  :width is ignored
+: [[image.png]]
+
+Now, =#+attr_org=, if present, takes precedence.


I think, it is better to avoid "is ignored" here.

Previously, when ~org-image-actual-width~ is a list or nil,
Org used the first =#+attr_...= keyword containing =:width ...=
to compute the inline image width.
Now, =#+attr_org=, if present, takes precedence.
In the following example the image preview has width of 70%
while earlier versions take 33%.

: #+attr_html: :width 33%
: #+attr_org:  :width 0.7
: [[image.png]]

Re: [DISCUSSION] Sorting strings in Org mode vs. system locale

[Max Nikulin]

On 07/05/2024 20:09, Ihor Radchenko wrote:

Max Nikulin writes:


I consider the following as a kind of graceful degradation

(defun org-sort-function-fallback-downcase
 (a b &optional LOCALE IGNORE-CASE)
   (if ignore-case
  (string-collate-lessp (downcase a) (downcase b) locale ignore-case)
(string-collate-lessp a b locale ignore-case)))


It is indeed better than `org-sort-function-downcase'.


`compare-strings' with upcase conversion under the hood may be an 
alternative.



I would consider a setter function for `org-sort-function' to avoid
branches based of `func-arity' in `org-string<'.


Setter is not reliable when setq is used, so I prefer arity check.


I bothers me as well. Another idea is to require 2 optional argument and 
thus wrappers for 2 argument functions. My expectation that extra 
function call still may be cheaper.

Re: Bug: Final zero after decimal point is stripped when inline code evaluated

[Max Nikulin]

On 07/05/2024 21:25, Charles Millar wrote:

On 5/7/24 7:49 AM, Max Nikulin wrote:

On 07/05/2024 03:05, Charles Millar wrote:


(setq toconvert 5.000)
(number-to-string toconvert)

evaluates to "5".


I get "5.0" (Linux), so I suspect some mistake.


I am using Debian Testing, full upgrade as of last Saturday.


Debian bookworm stable and Emacs-28.2.

LANG, LC_NUMERIC, LC_ALL environment variable should not affect 
formatting in numbers in Emacs since the "C" locale is forced for numbers.


The only guess I have is non-default value of `float-output-format'.

Re: Bug: Final zero after decimal point is stripped when inline code evaluated

[Max Nikulin]

On 07/05/2024 03:05, Charles Millar wrote:
I appreciate that mathematically a trailing zero or zeros may be 
non-significant; however, in my use case, i.e. correct format in a text, 
they are necessary. Another example, in addition to my Dollars and cents 
scenario, may be a table that that has been created by using append, and 
the table appears as follows because trailing zeros were disregarded.


This 1.222
that 3.444
it   5.6
last 7.691

Question arises - is the correct number reported on line "it" 5.600 or 
has some editing omitted the last two decimal places?


I am unsure what do you mean by "using append".

From my point of view there are 2 cases:
- When output format is fixed
  | 2 | 1.41 |
  | 3 | 1.73 |
  #+TBLFM: $2=(sqrt $1);%.2f
- When calculations should be performed with fixed point,
  usual floating point representation gives unexpected results
  src_elisp{(- 0.3 (+ 0.1 0.1 0.1))}
  {{{results(=-5.551115123125783e-17=)}}}

Some programming languages have decimal type for numbers:
https://docs.python.org/3/library/decimal.html

As to finances, I was surprised that ledger, hledger, and beancount have 
different internal representations for amounts and there are various 
issues with rounding.


Org just should avoid unnecessary conversions between strings an 
numbers, but Ihor is right and implementation would require enough efforts.



(setq toconvert 5.000)
(number-to-string toconvert)

evaluates to "5".


I get "5.0" (Linux), so I suspect some mistake.

Re: [DISCUSSION] Sorting strings in Org mode vs. system locale

[Max Nikulin]

On 05/05/2024 18:59, Ihor Radchenko wrote:

Ihor Radchenko writes:


If a user access same files from Linux and macOS then it
may be really annoying to get different order of entries in agenda. For
most of Linux users it is better to use more smart
`string-collate-lessp'. Some care is required to sort entries obtained
from multiple buffers in predictable environment (locale, case
conversion table).


I agree. We can introduce a new customization -
`org-string-sort-function' that will be used across Org mode to sort
user text.


See the attached tentative patch.
I added a customization, made everything in Org obey it, and provided
some default options for MacOS users.


Contrary to Eli, I still think that there are enough locales where 
completely disregarding IGNORE-CASE is worse than fallback to `downcase' 
when IGNORE-CASE is t. Perhaps some kind of normalization (NFD?) may 
improve results further.


I consider the following as a kind of graceful degradation

(defun org-sort-function-fallback-downcase
   (a b &optional LOCALE IGNORE-CASE)
 (if ignore-case
(string-collate-lessp (downcase a) (downcase b) locale ignore-case)
  (string-collate-lessp a b locale ignore-case)))

(defcustom org-sort-function
 (if (string-collate-lessp "a" "B" "C" t)
#'string-collate-lessp
  #'org-sort-function-fallback-downcase))

I would consider a setter function for `org-sort-function' to avoid 
branches based of `func-arity' in `org-string<'.


I see a little point in purely downcase comparator 
`org-sort-function-downcase'.

Re: Bug: Final zero after decimal point is stripped when inline code evaluated

[Max Nikulin]

On 06/05/2024 19:44, Ihor Radchenko wrote:

Charles Millar writes:

#+NAME: TABLE2
|22578.60|

src_latex[:var printthis=TABLE2[-1,-1] :eval yes :results replace
:exports results]{\num{printthis}} {{{results(@@latex:\num{22578.6}@@)}}}


This is expected. When you assign :var, Org mode reads the table data,
converting it to number.


After discussion of ob-shell related issues I am in doubts if org-babel 
should eagerly convert strings to numbers. Perhaps numbers should be 
recognized for specific backends only (or vice versa conversion should 
be suppressed for specific backends).



#+name: TABLE2
|"22578.60"|


It may require boilerplate code for post-processing causing inconvenience.

[PATCH] test-org-macro.el: Add test for CVE-2024-30202 (was: Re: [ANN] Emergency bugfix release: Org mode 9.6.23)

[Max Nikulin]

On 25/03/2024 00:16, Ihor Radchenko wrote:


I just released Org mode 9.6.23 that fixes several critical
vulnerabilities.


Since a variant of exploit has been published, it is time to add a test 
that might prevent code change re-introducing the most severe vulnerability.
From af8cddb44f5ee01fb1c7c9cf664ddbc83c63ac56 Mon Sep 17 00:00:00 2001
From: Max Nikulin 
Date: Mon, 6 May 2024 19:04:17 +0700
Subject: [PATCH] test-org-macro.el: Add test for CVE-2024-30202

* testing/lisp/test-org-macro.el (test-org-macro/initialize-templates):
A new test that no code is evaluated when an Org file is opened
(CVE-2024-30202).

Ihor Radchenko [ANN] Emergency bugfix release: Org mode 9.6.23.
Sun, 24 Mar 2024 17:16:50 +.
<https://list.orgmode.org/871q7zbldp.fsf@localhost>
---
 testing/lisp/test-org-macro.el | 10 ++
 1 file changed, 10 insertions(+)

diff --git a/testing/lisp/test-org-macro.el b/testing/lisp/test-org-macro.el
index 3339945fa..88a51693c 100644
--- a/testing/lisp/test-org-macro.el
+++ b/testing/lisp/test-org-macro.el
@@ -22,6 +22,16 @@ ;;; Code:
 
 ;;; Macros
 
+(ert-deftest test-org-macro/initialize-templates ()
+  "Test `org-macro-initialize-templates'."
+  ;; No code is executed during loading of Org mode files."
+  (should
+   (org-test-with-temp-text
+   "#+MACRO: title (eval (eval-and-compile (error \"CVE-2024-30202\")))"
+ (progn
+   (org-macro-initialize-templates)
+   t
+
 (ert-deftest test-org/macro-replace-all ()
   "Test `org-macro-replace-all' specifications."
   ;; Standard test.
-- 
2.39.2

Re: [PATCH] Add support for shortdoc link type

[Max Nikulin]

On 06/05/2024 04:35, Bruno Cardoso wrote:

Fixed.


I have tried the patch. It mostly works, so I do not insist on further 
polishing. However it should be resent with proper description.



I had dropped the search code because I wasn't considering the search
option properly. If I understand correctly, both "::" and "::#" should
match a function OR a search string? (e.g.:
`shortdoc:text-properties::#pos-property' will also search for
"pos-property" in the shortdoc buffer). This is the current behavior in
the attached patch.


It seems my comments have led mostly to confusion rather than 
improvements. I am sorry for that. I have in mind 2 kind of links:
-  that may be handled solely by 
`shortdoc-display-group' with second argument.

-  that should cause
  (shortdoc-display-group group)
  (search-forward fn nil t) ; or (word-search-forward fn nil t)

For `re-search-forward' used in the latest patch revision, from my point 
of view, link should be  
(similar to file: links).


Support of search variant may be dropped, but that case I would not 
allow "::" as the separator. Mandatory "::#" allows to add text and 
regexp search later.



+++ b/doc/org-manual.org
@@ -3382,6 +3382,13 @@ Here is the full set of built-in link types:
 
   Execute a shell command upon activation.
 
+- =shortdoc= ::

+
+  Link to short documentation summary for an Emacs Lisp function group.
+  Since Emacs 28, user command ~shortdoc-display-group~ lists all known
+  documentation groups. For more information, see [[info:emacs#Name Help]]
+  and [[info:elisp#Documentation Groups]].


I am unsure that links without description is a bright idea. Try
https://orgmode.org/manual/Activation.html
and M-: (info "(org) Activation") RET
to get an impression of export result.


+++ b/lisp/ol.el
@@ -1582,6 +1582,44 @@ PATH is a symbol name, as a string."
  :follow #'org-link--open-help
  :store #'org-link--store-help)
 
+ "shortdoc" link type

+(when (version<= "28.0.90" emacs-version)
+  (defun org-link--open-shortdoc (path _)
+"Open a \"shortdoc\" type link.
+PATH is a group name, \"group::#function\" or \"group::search_string\"."
+(string-match "\\`\\([^:]*\\)\\(?:::#?\\(.*\\)\\)?\\'" path)


I just have realized that it is better to ignore anything unexpected 
after ":", e.g. if only ::# is allowed (no search text option)


(string-match "\\`\\([^:]*\\)\\(?:::#\\(.*\\).*\\'\\)?" path)

Try [[shortdoc:string:invalid]].

Re: [PATCH] Add support for shortdoc link type

[Max Nikulin]

On 05/05/2024 00:33, Bruno Cardoso wrote:


I updated the patch to use the optional function argument instead.


Thanks


+++ b/doc/org-manual.org
@@ -3372,6 +3372,10 @@ Here is the full set of built-in link types:
 
   Execute a shell command upon activation.
 
+- =shortdoc= ::

+
+Link to short documentation summary for a function group.  For more 
information, see [[info:emacs#Name Help][Help by Command or Variable Name]].


I would consider explicit mention that it is related to Emacs Lisp and 
perhaps that the origin of its name is the ~shortdoc-display-group~ user 
command.  might be a better link.


This line is rather long though I do not have a link where formatting 
conventions are documented.



+++ b/etc/ORG-NEWS
@@ -1369,6 +1369,10 @@ place the entry in the ~Misc~ category if 
~TEXINFO_DIR_CATEGORY~ is missing.
 =TEXINFO_DIR_TITLE= is renamed to =TEXINFO_DIR_NAME=.
 The old name is obsolete.
 
+*** =ol.el=: Support for =shortdoc= link type

+
+Add support for storing and inserting links to =shortdoc= groups.


Again I am unsure that it is clear to all readers.


+++ b/lisp/ol.el
@@ -1582,6 +1582,41 @@ PATH is a symbol name, as a string."
  :follow #'org-link--open-help
  :store #'org-link--store-help)
 
+ "shortdoc" link type

+(when (version< "27" emacs-version)


Is it correct?

2a7488d42d8 2020-10-11 05:51:16 +0200 Lars Ingebrigtsen: Add support for 
displaying short documentation for function groups


git tag --contains 2a7488d42d8
emacs-28.0.90


+  (defun org-link--open-shortdoc (path _)
+"Open a \"shortdoc\" type link.
+PATH is a group name or \"group::#function\"."
+(string-match "\\`\\([^#:]*\\)\\(::#?\\(.*\\)\\)?\\'" path)


I think, it is enough to use [^:] since next group is started from "::". 
You may use use non-capturing group \\(?:::# ...\\) (requires update of 
next index below) unless you are going add search code back. I am unsure 
concerning your intentions. You dropped search code, but "#" after "::" 
is still optional.



+(let ((group (match-string 1 path))
+  (fn (match-string 3 path)))
+  (condition-case nil
+  (shortdoc-display-group group (intern-soft fn))
+(error (message "Unknown shortdoc group: %s" group)


I do not have strong opinion what is better here: `user-error' (used in 
earlier revisions) or `error'. However I do not see the point of 
capturing original error and signalling another one if it is not 
`user-error'.

Re: Importing "quoted" strings in `org-babel-import-elisp-from-file'

[Max Nikulin]

Max Nikulin writes:

Quotes are not stripped:


On 04/05/2024 18:51, Ihor Radchenko wrote:

Fixed, on main.


Another case:

(org-babel-read "\"Newlines\"\n" t)
"\"Newlines\"
"

Re: Importing "quoted" strings in `org-babel-import-elisp-from-file'

[Max Nikulin]

On 04/05/2024 18:51, Ihor Radchenko wrote:

Max Nikulin writes:

Quotes are not stripped:

(org-babel-read "\"abc\nsdf\"" t)
"\"abc
sdf\""


Fixed, on main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=edb5eaaac


Thanks


(org-babel-read "(\n\"abc\"\n" t)

progn: End of file during parsing


I cannot reproduce. Still, I added a prophylactic measure to recover
from parsing errors.


(should (equal "\"foo\"(\"bar\"" (org-babel-read "\"foo\"(\"bar\"" 
inhibit)))


you added to tests, does not cause the error since the unbalanced 
parenthesis is after first string. With the new regexp the error might 
happen for


(org-babel-read "\"Quoted closing quote:\\\"" t)

but it is caught by `ignore-errors'.

Perhaps `save-match-data' may be dropped if `string-match-p' is used 
instead of `string-match'. New code does not rely on match groups.

Re: Importing "quoted" strings in `org-babel-import-elisp-from-file'

[Max Nikulin]

On 04/05/2024 15:03, Ihor Radchenko wrote:

Max Nikulin writes:

I have no idea if "other\n\"string\"\nlines" may be passed
`org-babel-read', but it is not discarded by the current regexp:

"^[[:space:]]*\"\\(.*\\)\"[[:space:]]*$"


I do not see why we should limit things to single-line strings.


Quotes are not stripped:

(org-babel-read "\"abc\nsdf\"" t)
"\"abc
sdf\""

(org-babel-read "(\n\"abc\"\n" t)

progn: End of file during parsing

My conclusion that the current regexp may give both false positives and 
false negatives. `read` errors are not handled, but likely it is a 
result of wrong guarding regexp.



1. read is faster


I would not argue since I have no benchmark. My expectation was that 
replace is mostly no-op, so original string is returned while using 
`read' requires some overhead for a temporary buffer.



2. read is less maintenance - we can rely upon robust implementation
provided by Emacs itself instead of doing something custom, with
potential bugs.

Re: Importing "quoted" strings in `org-babel-import-elisp-from-file'

[Max Nikulin]

On 03/05/2024 19:06, Ihor Radchenko wrote:

Max Nikulin writes:


What I do not like in `org-babel-read' is false positive for escaped
quote when actually backslash is escaped:

(org-babel-read "\"1\" 2 \"3\"" t)
"1\\"


Fixed, on main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=2028bb15c


I have no idea if "other\n\"string\"\nlines" may be passed 
`org-babel-read', but it is not discarded by the current regexp:


"^[[:space:]]*\"\\(.*\\)\"[[:space:]]*$"

Is there a reason why it is necessary to call `read' twice on the same 
content? From my point of view, result of first call may be returned.


Does `read' have other role than unescaping backslash-protected 
characters? Likely it can be done by `replace-regexp-in-string', see the 
attachment. I have tried regexp on the following string:


   (let ((cases '(("" . nil)
   ("\"" . nil)
   ("\"\"" . t)
   ("\"\\\"" . nil)
   ("\"\"" . t)
   ("a" . nil)
   ("\"a\"" . t)
   ("\\\"a\\\"" . nil)
   ("\\\"a\"" . nil)
   ("\"a\\\"\"" . t)
   ("\"aa\\\"bb\"" . t)
   (" \"aa( bb\"" . tdiff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 470db9fe6..a44d27cba 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -3346,6 +3346,26 @@ (defun org-babel-script-escape (str &optional force)
 	  (t str
 (condition-case nil (org-babel-read escaped) (error escaped
 
+(defconst org-babel--quoted-string-regexp
+  (rx string-start
+  (zero-or-more (or space ?\n))
+  ?\"
+  (group
+   (zero-or-more
+;; Anything besides double quotes ended not with backslash.
+(zero-or-one (zero-or-more (not ?\"))
+		 (not (or ?\" ?\\)))
+;; Skip backslashes escaping themselves.
+	(zero-or-more "")
+;; Escaped quotes are allowed.
+	(zero-or-one "\\\"")))
+  ?\"
+  (zero-or-more (or space ?\n))
+  string-end)
+  "Regexp matching single string in double quotes.
+Group 1 is text inside quotes. String may be optionally padded with
+spaces. Backslashes quote other characters.")
+
 (defun org-babel-read (cell &optional inhibit-lisp-eval)
   "Convert the string value of CELL to a number if appropriate.
 Otherwise if CELL looks like Lisp (meaning it starts with a
@@ -3361,15 +3381,11 @@ (defun org-babel-read (cell &optional inhibit-lisp-eval)
  ;; FIXME: Arbitrary code evaluation.
 	 (eval (read cell) t))
 	((save-match-data
-   (and (string-match "^[[:space:]]*\"\\(.*\\)\"[[:space:]]*$" cell)
-;; CELL is a single string
-(with-temp-buffer
-  (insert cell)
-  (goto-char 1)
-  (read (current-buffer))
-  (skip-chars-forward "[:space:]")
-  (eobp
- (read cell))
+   (and (string-match org-babel--quoted-string-regexp cell)
+;; Unquote characters escaped by backslashes similar to `read'.
+(replace-regexp-in-string
+ "\\(?:\\(.\\)\\|\n\\)" "\\1"
+		 (match-string 1 cell) 'fixedcase nil
 	(t (org-no-properties cell
 
 (defun org-babel--string-to-number (string)

Re: [PATCH] Add support for shortdoc link type

[Max Nikulin]

On 04/05/2024 04:41, Bruno Cardoso wrote:

On 2024-05-02, 17:41 +0700, Max Nikulin wrote:




[...]

I implemented the search option by borrowing the regexp from `ol-info':


It is necessary to maintain backward compatibility for `ol-info'. I do 
not know what kind of separator was added earlier: "::" or "#". I 
suppose, for new link type, rules more close to file: links may be used, see

(info "(org) Search Options")
https://orgmode.org/manual/Search-Options.html

That is why I suggested shortdoc:GROUP::#FUNCTION. This way 
shrotdoc:GROUP::SEARCH_STRING is possible as well.



   (shortdoc-display-group group)
   (when fn
 (re-search-forward (concat "^(" (string-remove-prefix "#" fn))
nil t)


Notice that `shortdoc-display-group' has optional FUNCTION argument.


 (beginning-of-line)))
   (error (message "Unknown shortdoc group: %s" group)

It works, but I'm not quite sure if this is the best approach.


I do not have strong opinion as well. Some points to consider:
- Unambiguous parsing of link target.
- Flexibility it respect to later improvements.

Re: [BUG] HTML export does not preserve footnote label [9.6.15 (release_9.6.15 @ /usr/local/share/emacs/30.0.50/lisp/org/)]

[Max Nikulin]

On 03/05/2024 18:14, Ihor Radchenko wrote:

Max Nikulin writes:


Yes, sure. It is fine to reuse an existing user option. Though reading
through its docstring and the code, I cannot tell what this is doing
exactly. Is it applying to all HTML elements, or just headings?

On my end, I have that option set to nil, but exported headings do use
their CUSTOM_ID.


It seems it was broken by
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=5e9953fa0e

`user-label' is set to CUSTOM_ID when it is set, so checking the
variable is not effective any more.


Nothing is broken. CUSTOM_IDs for _headings_ were used unconditionally.

org-html-prefer-user-labels
 ...
 Independently of this variable, however, CUSTOM_ID are always
 used as a reference.


My bad. I believed that this option is (or was) necessary to get anchor 
names as CUSTOM_ID, but I have tried Org-9.1.9 and I see elements having 
id attribute with CUSTOM_ID values.

Re: [BUG] HTML export does not preserve footnote label [9.6.15 (release_9.6.15 @ /usr/local/share/emacs/30.0.50/lisp/org/)]

[Max Nikulin]

On 03/05/2024 13:59, Protesilaos Stavrou wrote:

From: Max Nikulin Fri, 26 Apr 2024 16:53:23 +0700

Another option may be to rely on the existing one:
`org-html-prefer-user-labels'


Yes, sure. It is fine to reuse an existing user option. Though reading
through its docstring and the code, I cannot tell what this is doing
exactly. Is it applying to all HTML elements, or just headings?

On my end, I have that option set to nil, but exported headings do use
their CUSTOM_ID.


It seems it was broken by
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=5e9953fa0e

`user-label' is set to CUSTOM_ID when it is set, so checking the 
variable is not effective any more.

Re: link can not be created in a line with another link.

[Max Nikulin]

On 02/05/2024 18:15, Ihor Radchenko wrote:

May you try the attached patch (on top of the latest main)?
Does it feel better?


I see regressions only (Emacs 28).

[[#foo]]
[[#bar]]

SPC causes permanent appearance of square brackets for [[#foo]].

I do not like flashes especially since they do not help to avoid issues 
with prepending a new link


- [[#bar]]

Re: [BUG] #+CALL - permission error -bash

[Max Nikulin]

On 02/05/2024 21:55, Ihor Radchenko wrote:

vitalij writes:


in org-babel-sh-evaluate
  file:~/.emacs.d/elpa/org-9.6.28/ob-shell.el::300

this do apply: (process-file "/tmp/babel-NfRG9P/sh-script-jmKNA4"
  "/tmp/babel-NfRG9P/sh-stdin-o3CEm5" # nil nil)

I don't allow executables in /tmp folder!


Why do you think that it is a bug in Org mode?
AFAIK, it is generally expected that anything can go into tmp.


There are various guides recommending noexec, however they warn that 
some issues should be expected.


Securing Debian Manual
- https://www.debian.org/doc/manuals/securing-debian-manual/ch04s10.en.html
  4.10. Mounting partitions the right way
- 
https://www.debian.org/doc/manuals/securing-debian-manual/checklist.en.html

  B.2. Configuration checklist

Security in Arch Linux wiki:
https://wiki.archlinux.org/title/Security#Mount_options
Mount options


How to make it this way:

(process-file "/use/bin/bash /tmp/babel-NfRG9P/sh-script-jmKNA4"
 "/tmp/babel-NfRG9P/sh-stdin-o3CEm5" # nil nil)

???

It is like when you do:
./a.sh

instead of:
bash a.sh


Likely you mean

(process-file "/usr/bin/bash"
 "/tmp/babel-NfRG9P/sh-stdin-o3CEm5" # nil 
"/tmp/babel-NfRG9P/sh-script-jmKNA4")


but it would ignore shebang. Try

#!/bin/sh -e
false
printf "Should not be executed\n"

So it is necessary to either drop :shebang or set temporary directory to 
a suitable path.

Re: Importing "quoted" strings in `org-babel-import-elisp-from-file'

[Max Nikulin]

On 01/05/2024 19:19, Ihor Radchenko wrote:

And replacing the call to org-babel-string-read with org-babel-read does not
break any tests...


Since

005e68294 2009-06-11 17:04:42 -0700 Eric Schulte: making progress 
bringing org-babel-R.el into the new evaluation schema


`org-babel-string-read' has been stripping quotes, but `org-string-read' 
got this feature later in


60a8ba556 2011-03-02 07:55:39 -0700 Eric Schulte: ob: read string 
variable values wrapped in double quotes, removing the quotes


So now outer quotes are stripped twice.

(org-babel-string-read "\"\"1\"\"")
"1"

(org-babel-read "\"\"1\"\"" t)
"\"\"1\"\""

There are still more subtle differences between these functions:

(org-babel-string-read "\"1\" 2 \"3\"")
"1\" 2 \"3"

(org-babel-read "\"1\" 2 \"3\"" t)
"\"1\" 2 \"3\""

What I do not like in `org-babel-read' is false positive for escaped 
quote when actually backslash is escaped:


(org-babel-read "\"1\" 2 \"3\"" t)
"1\\"

(org-babel-read "\"1|\" 2 |\"3\"" t)
"\"1|\" 2 |\"3\""

Test suite is far from being exhaustive.

Re: [PATCH] Add support for shortdoc link type

[Max Nikulin]

On 02/05/2024 02:24, Bruno Cardoso wrote:

+++ b/lisp/ol.el
@@ -1597,6 +1597,37 @@ PATH is a symbol name, as a string."
  (org-link-set-parameters "help"
   :follow #'org-link--open-help
   :store #'org-link--store-help)
+ "shortdoc" link type
+(when (version< "27" emacs-version)
+  (defun org-link--open-shortdoc (group _)
+"Open a \"shortdoc\" type link.
+GROUP is a symbol name."
+(if (assoc (intern group) shortdoc--groups)
+(shortdoc-display-group group)
+  (user-error "Unknown shortdoc group: %s" group)))


`condition-case' may help to avoid the internal `shortdoc--groups' 
variable here. As to completion, it is better to ask for public API. 
However emacs developers likely will decline such request.


Consider the following just as ideas.

- Support of search options. Buttons in help pages move point to 
specific functions.




- At first I considered adding shortdoc to help links that may call 
either `describe-function' or `describe-variable', but it is unlikely 
possible.

[PATCH] test-ob-shell.el: Skip based on feature detection

[Max Nikulin]

On 28/04/2024 20:11, Ihor Radchenko wrote:

https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=e4ab416fc



Max Nikulin  writes:

Feature detection should be more reliable.

[...]


Feel free to submit a patch.


See the attachment.
From 46db53cebef5644be4abd8dcb18ca777ec0c3be3 Mon Sep 17 00:00:00 2001
From: Max Nikulin 
Date: Thu, 2 May 2024 17:09:10 +0700
Subject: [PATCH] test-ob-shell.el: Skip based on feature detection

* testing/lisp/test-ob-shell.el (test-ob-shell/bash-uses-assoc-arrays)
(test-ob-shell/bash-uses-assoc-arrays-with-lists): Use feature detection
shell command instead of version comparison to skip tests for
associative arrays.

It affects macOS since Apple ships BASH 3.2 licensed as GPLv2.
---
 testing/lisp/test-ob-shell.el | 20 +++-
 1 file changed, 7 insertions(+), 13 deletions(-)

diff --git a/testing/lisp/test-ob-shell.el b/testing/lisp/test-ob-shell.el
index f58e85815..b760031b4 100644
--- a/testing/lisp/test-ob-shell.el
+++ b/testing/lisp/test-ob-shell.el
@@ -230,6 +230,10 @@ (ert-deftest test-ob-shell/bash-uses-assoc-arrays ()
 Bash will see a table that contains the first column as the
 'index' of the associative array, and the second column as the
 value. "
+  (skip-unless
+   ;; Old GPLv2 BASH in macOSX does not support associative arrays.
+   (if-let ((bash (executable-find "bash")))
+   (eq 0 (process-file bash nil nil nil "-c" "declare -A assoc_array"
   (org-test-with-temp-text
   "#+NAME: sample_mapping_table
 | first  | one   |
@@ -240,13 +244,6 @@ (ert-deftest test-ob-shell/bash-uses-assoc-arrays ()
 echo ${table[second]}
 
 #+end_src "
-(skip-unless (executable-find "bash"))
-(skip-unless
- (let* ((version-string (shell-command-to-string "bash -c 'echo $BASH_VERSION'"))
-(major-version (and (string-match "^\\([0-9]+\\)\\." version-string)
-			(string-to-number (match-string 1 version-string)
-   ;; Bash 4.0 introduced associative arrays support.
-   (>= major-version 4)))
 (should
  (equal "two"
 (org-trim (org-babel-execute-src-block))
@@ -256,13 +253,10 @@ (ert-deftest test-ob-shell/bash-uses-assoc-arrays-with-lists ()
 
 Bash will see an associative array that contains each row as a single
 string. Bash cannot handle lists in associative arrays."
-  (skip-unless (executable-find "bash"))
   (skip-unless
-   (let* ((version-string (shell-command-to-string "bash -c 'echo $BASH_VERSION'"))
-  (major-version (and (string-match "^\\([0-9]+\\)\\." version-string)
-			  (string-to-number (match-string 1 version-string)
- ;; Bash 4.0 introduced associative arrays support.
- (>= major-version 4)))
+   ;; Old GPLv2 BASH in macOSX does not support associative arrays.
+   (if-let ((bash (executable-find "bash")))
+   (eq 0 (process-file bash nil nil nil "-c" "declare -A assoc_array"
   (org-test-with-temp-text
   "#+NAME: sample_big_table
 | bread |  2 | kg |
-- 
2.39.2

Re: [BUG] ob-shell: internal representation of cmdline arguments the same

[Max Nikulin]

On 29/04/2024 19:22, Ihor Radchenko wrote:

Matt writes:


#+begin_src bash :cmdline "1 2 3"

[...]

To force quotes in the :cmdline one can do

#+begin_src bash :cmdline "\"1 2 3\""
echo "$1"
#+end_src


I consider it as a kind of pitfall inconsistent with DWIM concept. An 
idea of a kludge is below.


#+begin_src sh :cmdline "1 2 3" :results verbatim
  printf '%s\n' "$@"
#+end_src

#+RESULTS:
: 1 2 3

#+begin_src sh :cmdline 1 2 3 :results verbatim
  printf '%s\n' "$@"
#+end_src

#+RESULTS:
: 1
: 2
: 3

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 73fb70c26..efba97f2c 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -3363,7 +3363,7 @@ (defun org-babel-read (cell &optional 
inhibit-lisp-eval)

((save-match-data
(and (string-match "^[[:space:]]*\"\\(.*\\)\"[[:space:]]*$" 
cell)

 (not (string-match "[^\\]\"" (match-string 1 cell)
- (read cell))
+ (propertize (read cell) 'org-babel-value 'quoted-string))
(t (org-no-properties cell

 (defun org-babel--string-to-number (string)
diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
index 35d9e9376..77c3fe261 100644
--- a/lisp/ob-shell.el
+++ b/lisp/ob-shell.el
@@ -300,6 +300,9 @@ (defun org-babel-sh-evaluate (session body &optional 
params stdin cmdline)

 of the statements in BODY, if RESULT-TYPE equals `value' then
 return the value of the last statement in BODY."
   (let* ((shebang (cdr (assq :shebang params)))
+ (cmdline-quote
+  (and cmdline
+   (get-text-property 0 'org-babel-value cmdline) "\""))
  (async (org-babel-comint-use-async params))
 (results-params (cdr (assq :result-params params)))
 (value-is-exit-status
@@ -329,7 +332,9 @@ (defun org-babel-sh-evaluate (session body &optional 
params stdin cmdline)

 nil
 (if shebang (when cmdline (list cmdline))
   (list shell-command-switch
-(concat (file-local-name script-file) 
" " cmdline)

+(concat (file-local-name script-file)
+" " cmdline-quote cmdline
+cmdline-quote)
(buffer-string
   (session ; session evaluation
 (if async

Re: [PATCH] org.el: Call EXT-enable for `org-modules' (was Re: [PATCH] org-ctags.el: Do not activate on load)

[Max Nikulin]

On 01/05/2024 17:21, Ihor Radchenko wrote:

Max Nikulin writes:


However now I am in doubts why org-ctags is not a minor mode.


What would be the benefit?


Documented conventions how to enable or disable minor mode.

Re: [PATCH] ob-lua: Support all types and multiple values in results

[Max Nikulin]

On 30/04/2024 03:26, Rudolf Adamkovič wrote:

+local value = result[1]
+if string.find(value, '[%%(%%[{]') == 1 then
+  return quotes .. value .. quotes


Am I wrong that quotes in value may cause issues? I expect some way of 
escaping " characters.

Re: link can not be created in a line with another link.

[Max Nikulin]

On 29/04/2024 19:45, Alexandros Prekates wrote:


The same behavior happens with emacs -Q.

bla [[foo]]

If i add :
[[boo]] bla [[foo]]
org will create the two links.

But if i add:
[[boo][  bla [[foo]]
with the intention to type: [[boo][description]]  bla [[foo]]
then what i will get is: a link to boo with description : bla [[foo


This case you can keep calm and continue typing "description]]".

I have a more weird example. Consider you are going to get

- [[#foo]] and [[#bar]]

starting from

- [[#bar]]

There is no problem till

- [[#foo[[#bar]]

However in response to "]" point is moved after "["

- [[#foo][[#bar]]

that is displayed as

- [#bar

I do not see other ways than undo, SPC, and moving back. It would be 
great to keep point between brackets "][" this case. I have no 
idea if it is feasible to postpone hiding link target when such pattern 
is typed.


I have descriptive links enabled despite most of links in my notes are 
like [[#CUSTOM_ID]]. There are still some links with descriptions.

Re: [BUG] ob-shell: results missing leading quotes

[Max Nikulin]

On 29/04/2024 18:58, Ihor Radchenko wrote:


The nested "..." are stripped on purpose via `org-babel-string-read'.
Although this function dates back to R output processing and I do not
fully understand why stripping nested quotes is useful for all possible
babel backends. But that's a completely different story.


I would say heuristics is quite fragile

#+begin_src elisp :results verbatim
  (org-babel-string-read "\"1 2\" 3 \"4 5\"")
#+end_src

#+RESULTS:
: "1 2\" 3 \"4 5"

Leading and trailing quote characters are not stripped in the case of

#+begin_src elisp :results verbatim
  (org-babel-read "\"1 2\" 3 \"4 5\"" t)
#+end_src

#+RESULTS:
: "\"1 2\" 3 \"4 5\""

Re: [PATCH] org-ctags.el: Do not activate on load

[Max Nikulin]

On 30/04/2024 19:59, Ihor Radchenko wrote:

/bin/sh: 1: Syntax error: Bad function name
(ert-test-failed
  ((should
(test-org-ctags/list-elements-equal-p
(list ...)
(test-org-ctags/with-fake-ctags temp-dir "regexp" ...)
'(2)
"Regexp should be escaped."))
   :form
   (test-org-ctags/list-elements-equal-p
("--regex-orgmode=/<<([^<>]+)>>/\\1/d,definition/")
"Sould be overwritten by org-ctags mock script"
(2)
"Regexp should be escaped.")
   :value nil :explanation "Shell command failed"))
FAILED   630/1311  test-org-ctags/create-tags-escape (0.003107 sec) at 
../lisp/test-org-ctags.el:132


At least error message is not bad. Dash does not like
ctags-mock() { ... }

diff --git a/testing/lisp/test-org-ctags.el b/testing/lisp/test-org-ctags.el
index 7f5fca948..b8e3e4d22 100644
--- a/testing/lisp/test-org-ctags.el
+++ b/testing/lisp/test-org-ctags.el
@@ -76,17 +76,17 @@ (defmacro test-org-ctags/with-fake-ctags
 (,dir (concat ,base "/" ,subdir))
 (,temp-file (concat ,dir "/ctags.txt"))
 (org-ctags-path-to-ctags
- (test-org-ctags/mock-command ,temp-file "ctags-mock"))
+ (test-org-ctags/mock-command ,temp-file "ctags_mock"))
 ,buffer)
(make-directory ,dir)
(unwind-protect
;; `org-ctags' commands call `buffer-file-name'.
(with-current-buffer
(setq ,buffer (find-file-noselect ,temp-file))
- (insert "Sould be overwritten by org-ctags mock script")
+ (insert "Should be overwritten by org-ctags mock script")
  (save-buffer)
  ,@body
- (test-org-ctags/get-args ,temp-file ,base "ctags-mock\n"))
+ (test-org-ctags/get-args ,temp-file ,base "ctags_mock\n"))
  (kill-buffer ,buffer)
  (delete-file ,temp-file)
  (delete-directory ,dir)

[PATCH] org.el: Call EXT-enable for `org-modules' (was Re: [PATCH] org-ctags.el: Do not activate on load)

[Max Nikulin]

On 30/04/2024 17:02, Ihor Radchenko wrote:

Max Nikulin writes:


Isn't it better to modify buggy org-ctags than to add various kludges to
tests?


Applied, onto main.


I was expecting that you would be against it since it is a breaking change.

See a follow-up to
Ihor Radchenko. Re: Autoloading side effects
(was: Re: [BUG] org-mouse is activated without explicit require)
Mon, 12 Dec 2022 10:25:16 +
<https://list.orgmode.org/871qp4j4vn.fsf@localhost>

However now I am in doubts why org-ctags is not a minor mode.From 9cf1ebd0d8a0b1aa059f049aeb0c297b114aa4e2 Mon Sep 17 00:00:00 2001
From: Max Nikulin 
Date: Tue, 30 Apr 2024 18:05:11 +0700
Subject: [PATCH] org.el: Call EXT-enable for `org-modules'

* lisp/org.el (org-load-modules-maybe): Activate extensions that
implement EXT-enable functions.
* etc/ORG-NEWS:
* lisp/org-ctags.el: Update docs.

It should simplify configuration for `org-ctags' users after badb09d67.
The idea was discussed in

Ihor Radchenko. Re: Autoloading side effects
(was: Re: [BUG] org-mouse is activated without explicit require)
Mon, 12 Dec 2022 10:25:16 +
<https://list.orgmode.org/871qp4j4vn.fsf@localhost>

An alternative from the same thread is converting org modules
to minor modes.
---
 etc/ORG-NEWS  | 23 +++
 lisp/org-ctags.el |  5 -
 lisp/org.el   | 11 +--
 3 files changed, 32 insertions(+), 7 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 8dbc3292d..708d11f4f 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -651,10 +651,14 @@ just =id:= links.
 
 *** ~org-ctags~ is not activated by default any more
 
-To follow Emacs [[info:elisp#Coding Conventions][coding conventions]] and to avoid confusion of users
+To follow Emacs [[info:elisp#Coding Conventions][coding conventions]]
+and to avoid confusion of users
 who accidentally get ~org-ctags~ autoloaded due to help completion,
-the library does not modify ~org-open-link-functions~ during loading
-any more.  Run ~org-ctags-enable~ to setup hooks and advices:
+the library does not modify ~org-open-link-functions~ any more
+when it is loaded using ~(require 'org-ctags)~ or a similar construct.
+To setup hooks and advices either
+[[*~feature-enable~ functions are called for extensions from ~org-modules~][customize ~org-modules~]]
+and add ~org-ctags~ or run ~org-ctags-enable~ explicitly:
 
 #+begin_src emacs-lisp
 (with-eval-after-load "org-ctags"
@@ -1473,10 +1477,21 @@ buffer it will be added.  If not specified, new headings are created
 at level 1 at the end of the accessible part of the buffer, as before.
 
 ** Miscellaneous
+*** ~feature-enable~ functions are called for extensions from ~org-modules~
+
+Accordingly to Emacs [[info:elisp#Coding Conventions][coding conventions]]
+loading a library must not modify behavior
+since it may be done for the sake of help or command completion.
+To make user configuration more convenient, when an extension ~EXT~ listed
+in ~org-modules~ implements ~EXT-enable~ function, it is executed
+during loading of Org mode or in response to customization of ~org-modules~.
+So instead of ~(require 'EXT)~ in your init file add ~EXT~ to ~org-modules~.
+See also
+[[*~org-ctags~ is not activated by default any more][~org-ctags~ is not activated by default any more]].
+
 *** =org-crypt.el= now applies initial visibility settings to decrypted entries
 
 Previously, all the text was unfolded unconditionally, including property drawers.
-
 *** Blank lines after removed objects are now retained during export
 
 When certain objects in Org document are to be excluded from export,
diff --git a/lisp/org-ctags.el b/lisp/org-ctags.el
index d3af103dd..9313c43cd 100644
--- a/lisp/org-ctags.el
+++ b/lisp/org-ctags.el
@@ -57,10 +57,13 @@ ;;; Commentary:
 ;;(add-hook 'org-mode-hook
 ;;  (lambda ()
 ;;(define-key org-mode-map "\C-co" 'org-ctags-find-tag-interactive)))
+;;
+;; To activate the library, you either need to add it to `org-modules'
+;; or to call `org-ctags-enable' explicitly:
+;;
 ;;(with-eval-after-load "org-ctags"
 ;;  (org-ctags-enable))
 ;;
-;; To activate the library, you need to call `org-ctags-enable' explicitly.
 ;; It used to be invoked during library loading, but it was against Emacs
 ;; policy and caused inconvenience of Org users who do not use `org-ctags'.
 ;;
diff --git a/lisp/org.el b/lisp/org.el
index 2d1a2055f..703ef 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -724,10 +724,17 @@ (defvar org-modules-loaded nil
 
 ;;;###autoload
 (defun org-load-modules-maybe (&optional force)
-  "Load all extensions listed in `org-modules'."
+  "Load all extensions listed in `org-modules'.
+If an extension defines EXT-enable function then invoke it
+to activate the module."
   (when (or force (not org-modules-loaded))
 (dolist (ext org-modules)
-  (condition-case-unless-debug nil (requir

[PATCH] org-ctags.el: Do not activate on load

[Max Nikulin]

On 29/04/2024 20:12, Ihor Radchenko wrote:

Max Nikulin writes:


Notice that new tests for org-ctags do not require user interactions.

[...]

Of course, the cause is the known side effect of loading org-ctags.

Maybe we can disable the tests until that bug is fixed.
Or, ideally, load org-ctags only when the relevant tests are running and
unload it after they finish.


Isn't it better to modify buggy org-ctags than to add various kludges to 
tests?
From 0d260a0f260afb0f407d00b6a53ed2121a240203 Mon Sep 17 00:00:00 2001
From: Max Nikulin 
Date: Mon, 29 Apr 2024 21:34:13 +0700
Subject: [PATCH 1/2] org-ctags.el: Define unload function
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* lisp/org-ctags.el (org-ctags-unload-function): New function to cleanup
during `unload-feature' call.
(org-ctags--open-link-functions-list org-ctags-open-link-functions):
Define and use list of options available for `org-open-link-functions'.
(org-ctags--visit-tags-table): Give a name to remove the function from
`org-mode-hook' on library unload.

Prevent the following error after library unloading

Symbol’s function definition is void: org-ctags-find-tag
---
 lisp/org-ctags.el | 60 +++
 1 file changed, 40 insertions(+), 20 deletions(-)

diff --git a/lisp/org-ctags.el b/lisp/org-ctags.el
index 6431a2765..c6f7fc708 100644
--- a/lisp/org-ctags.el
+++ b/lisp/org-ctags.el
@@ -161,6 +161,20 @@ (defcustom org-ctags-path-to-ctags
   :version "24.1"
   :type 'file)
 
+(defconst org-ctags--open-link-functions-list
+  (list
+   #'org-ctags-find-tag
+   #'org-ctags-ask-rebuild-tags-file-then-find-tag
+   #'org-ctags-rebuild-tags-file-then-find-tag
+   #'org-ctags-ask-append-topic
+   #'org-ctags-append-topic
+   #'org-ctags-ask-visit-buffer-or-file
+   #'org-ctags-visit-buffer-or-file
+   #'org-ctags-fail-silently)
+  "Options for `org-open-link-functions'.
+Ensure that the user option and `unload-feature'
+use the same set of functions.")
+
 (defcustom org-ctags-open-link-functions
   '(org-ctags-find-tag
 org-ctags-ask-rebuild-tags-file-then-find-tag
@@ -168,14 +182,7 @@ (defcustom org-ctags-open-link-functions
   "List of functions to be prepended to ORG-OPEN-LINK-FUNCTIONS by ORG-CTAGS."
   :version "24.1"
   :type 'hook
-  :options '(org-ctags-find-tag
- org-ctags-ask-rebuild-tags-file-then-find-tag
- org-ctags-rebuild-tags-file-then-find-tag
- org-ctags-ask-append-topic
- org-ctags-append-topic
- org-ctags-ask-visit-buffer-or-file
- org-ctags-visit-buffer-or-file
- org-ctags-fail-silently))
+  :options org-ctags--open-link-functions-list)
 
 
 (defvar org-ctags-tag-list nil
@@ -191,18 +198,20 @@ (defcustom org-ctags-new-topic-template
   :type 'string)
 
 
-(add-hook 'org-mode-hook
-  (lambda ()
-(when (and org-ctags-enabled-p
-   (buffer-file-name))
-  ;; Make sure this file's directory is added to default
-  ;; directories in which to search for tags.
-  (let ((tags-filename
- (expand-file-name
-  (concat (file-name-directory (buffer-file-name))
-  "/TAGS"
-(when (file-exists-p tags-filename)
-  (visit-tags-table tags-filename))
+(defun org-ctags--visit-tags-table ()
+  "Load tags for current file.
+A function for `org-mode-hook."
+  (when (and org-ctags-enabled-p
+ (buffer-file-name))
+;; Make sure this file's directory is added to default
+;; directories in which to search for tags.
+(let ((tags-filename
+   (expand-file-name
+(concat (file-name-directory (buffer-file-name))
+"/TAGS"
+  (when (file-exists-p tags-filename)
+(visit-tags-table tags-filename)
+(add-hook 'org-mode-hook #'org-ctags--visit-tags-table)
 
 
 (advice-add 'visit-tags-table :after #'org--ctags-load-tag-list)
@@ -219,6 +228,17 @@ (defun org-ctags-enable ()
 (add-hook 'org-open-link-functions fn t)))
 
 
+(defun org-ctags-unload-function ()
+  "Disable `org-ctags' library.
+Called by `unload-feature'."
+  (put 'org-mode 'find-tag-default-function nil)
+  (advice-remove 'visit-tags-table #'org--ctags-load-tag-list)
+  (advice-remove 'xref-find-definitions
+ #'org--ctags-set-org-mark-before-finding-tag)
+  (dolist (fn org-ctags--open-link-functions-list)
+(remove-hook 'org-open-link-functions fn nil)))
+
+
 ;;; General utility functions.  ===
 ;; These work outside org-ctags mode.
 
-- 
2.39.2

From 5

Re: [PATCH] org-ctags.el: Protect shell specials in directory name

[Max Nikulin]

On 29/04/2024 00:02, Ihor Radchenko wrote:

Max Nikulin writes:


Tests must be fully automated. We use make test in CI and things like
project-compile are non-interactive.


I do not mind. Requiring unexpected user interactions is a feature of
org-ctags.


So, may you update the patch to make tests automated?


Notice that new tests for org-ctags do not require user interactions. Try

make test BTEST_RE=test-org-ctags/

Re: [PATCH] org-ctags.el: Protect shell specials in directory name

[Max Nikulin]

On 28/04/2024 23:55, Ihor Radchenko wrote:

Max Nikulin  writes:


I tried to run make test with your patch applied, but I am getting
interactive prompt there:

  Visit tags table (default TAGS):


You may press RET...

It is due to invasive org-ctags behavior.


Tests must be fully automated. We use make test in CI and things like
project-compile are non-interactive.


I do not mind. Requiring unexpected user interactions is a feature of 
org-ctags.

Re: [PATCH] org-ctags.el: Protect shell specials in directory name

[Max Nikulin]

On 28/04/2024 19:53, Ihor Radchenko wrote:

May you submit a patch?

Max Nikulin writes:

See the attachments.


I tried to run make test with your patch applied, but I am getting
interactive prompt there:

 Visit tags table (default TAGS):


You may press RET...

It is due to invasive org-ctags behavior.

Nick Dokos. org-ctags land grab. Mon, 20 Mar 2023 23:36:09 -0400.
https://list.orgmode.org/87o7omg4ie@alphaville.usersys.redhat.com

[PATCH] org-ctags.el: Protect shell specials in directory name

[Max Nikulin]

On 20/03/2024 19:08, Ihor Radchenko wrote:

Max Nikulin writes:

The committed change is anyway incomplete.


May you submit a patch?


See the attachments.From 067dc590bb1c26c881f14d218da2cd502413ec5d Mon Sep 17 00:00:00 2001
From: Max Nikulin 
Date: Wed, 27 Mar 2024 23:04:07 +0700
Subject: [PATCH 1/2] org-ctags.el: Protect shell specials in directory name

* lisp/org-ctags.el (org-ctags-create-tags): Escape shell specials.

Directory name (the argument or `default-directory') may contain various
characters interpreted by shell.  Effects may vary from just incorrect
actual path to execution of a command embedded into path.  Neither
double nor single quotes is a safe way to use directory name in shell
commands since the name may contain these characters.

A follow-up to
Martin Marshall. [PATCH] `org-ctags-create-tags` creates empty TAGS file.
Fri, 09 Feb 2024 18:57:48 -0500.
<https://list.orgmode.org/87h6ihgphf@martinmarshall.com>
---
 lisp/org-ctags.el | 15 +--
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/lisp/org-ctags.el b/lisp/org-ctags.el
index 6431a2765..52b21dbd1 100644
--- a/lisp/org-ctags.el
+++ b/lisp/org-ctags.el
@@ -477,18 +477,21 @@ (defun org-ctags-create-tags (&optional directory-name)
 its subdirectories contain large numbers of taggable files."
   (interactive)
   (cl-assert (buffer-file-name))
-  (let ((dir-name (or directory-name
-  (file-name-directory (buffer-file-name
+  (let ((dir-name (shell-quote-argument
+   (expand-file-name
+(if directory-name
+(file-name-as-directory directory-name)
+  (file-name-directory (buffer-file-name))
 (exitcode nil))
 (save-excursion
   (setq exitcode
 (shell-command
  (format (concat "%s --langdef=orgmode --langmap=orgmode:.org "
- "--regex-orgmode=\"%s\" -f \"%s\" -e -R %s")
+ "--regex-orgmode=%s -f %sTAGS -e -R %s*")
  org-ctags-path-to-ctags
- org-ctags-tag-regexp
- (expand-file-name (concat dir-name "/TAGS"))
- (expand-file-name (concat (shell-quote-argument dir-name) "/*")
+ (shell-quote-argument org-ctags-tag-regexp)
+ dir-name
+ dir-name)))
   (cond
((eql 0 exitcode)
 (setq-local org-ctags-tag-list
-- 
2.39.2

From 52611183ff73d0701d41102e0fa97134178cae10 Mon Sep 17 00:00:00 2001
From: Max Nikulin 
Date: Sun, 28 Apr 2024 14:13:04 +0700
Subject: [PATCH 2/2] test-org-ctags.el: Test escaping of shell arguments

* testing/lisp/test-org-ctags.el (test-org-ctags/create-tags-escape):
A new test that tag regexp and directory names are properly quoted
while "*" wildcard is active.
(test-org-ctags/list-elements test-org-ctags/list-elements-equal-p)
(test-org-ctags/list-elements-equal-explain): Helpers to provide
informative failure messages.
(test-org-ctags/with-fake-ctags): A helper to create temporary
directories and a file and to temporary arrange a mock shell command
instead of ctags executable.
(test-org-ctags/mock-command test-org-ctags/get-args): Helpers to define
a mock shell command and to obtain its actual arguments.
---
 testing/lisp/test-org-ctags.el | 192 +
 1 file changed, 192 insertions(+)
 create mode 100644 testing/lisp/test-org-ctags.el

diff --git a/testing/lisp/test-org-ctags.el b/testing/lisp/test-org-ctags.el
new file mode 100644
index 0..7f5fca948
--- /dev/null
+++ b/testing/lisp/test-org-ctags.el
@@ -0,0 +1,192 @@
+;;; test-org-ctags.el --- tests for org-ctags.el  -*- lexical-binding: t -*-
+
+;; Copyright (C) 2024 Max Nikulin
+;; Authors: Max Nikulin
+
+;; This file is not part of GNU Emacs.
+
+;; This program is free software; you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; This program is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Code:
+
+;; Alternative implementation for `test-org-ctags/mock-command'
+;; is required for cmd.exe.
+(unless (string-equal "-c" shell-command-switch)
+  (signal 'missing-test-dependency "POSIX shell"))
+
+(require 'org-ctags)
+
+ Helpers:
+
+(defun test-org-ctags/mock-command (temp-file command-name)
+

Re: [PATCH] Re: [BUG] ob-shell: :shebang changes interpretation of :cmdline

[Max Nikulin]

On 27/04/2024 17:31, Max Nikulin wrote:

On 26/04/2024 18:49, Ihor Radchenko wrote:



+    shell-file-name

...

+    (list shell-command-switch
+  (concat (file-local-name 
script-file)  " " cmdline))))



Max Nikulin writes:

Using `shell-command-switch' unconditionally may lead to executing
/bin/sh instead of shell specified by `shell-file-name' for script 
files

having no shebang, see

https://superuser.com/questions/502984/writing-shell-scripts-that-will-run-on-any-shell-using-multiple-shebang-lines


I conclude that your concern, while being valid, is a _different_ bug.
Thus, I do not see it as a blocker for my patch - my patch will fix the
*original bug reported on top of this thread*.


My concern is that your patch trying to fix one bug (I am not convinced 
it is an improvement despite it is a step toward consistency) introduces 
another one that is not currently present in the code.


I read current variant of code once more. -c is unconditionally added, 
so your variant just makes the bug apparent. I still do not like behavior of


:cmdline 1 ; touch /tmp/not-an-arg

as I stated in

Re: [DISCUSSION] The meaning of :cmdline header argument across babel 
backends. Sat, 27 Apr 2024 17:53:25 +0700.

https://list.orgmode.org/v0ilf6$34l$1...@ciao.gmane.io

Re: [PATCH] ob-lua: Support all types and multiple values in results

[Max Nikulin]

On 26/04/2024 20:40, Ihor Radchenko wrote:

+++ b/lisp/ob-lua.el
@@ -282,6 +282,8 @@ (defvar org-babel-lua-wrapper-method
   end
end
return result
+   elseif type(it) == 'string' then
+  return '\"' .. it .. '\"'


If you are adding quotes around then quote characters in the value 
should be escaped.

Re: [DISCUSSION] The meaning of :cmdline header argument across babel backends

[Max Nikulin]

On 26/04/2024 20:09, Ihor Radchenko wrote:

Max Nikulin writes:


However looking wider, I do not like that :cmdline for ob-shell has
different meaning than for other languages, see e.g. ob-sql. Only for
shell this parameter is treated as arguments of a *script*. In other
cases :cmdline is used to specify arguments of *interpreter* and I think
ob-shell should follow this convention.


Alas, we already have the current state of affairs documented in
https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-shell.html#orge70bc7b

So, no breaking changes.


It is documented as
" :cmdline  ... [arg_n]

Use the :cmdline header arg to pass arguments to a shell command."

However current implementation allows code injection through args, 
including a trivial one


#+header-arg: :results verbatim
#+begin_src sh :cmdline 1 ; touch /tmp/not-an-arg
  printf '%s\n' "$@"
#+end_src

#+RESULTS:
: 1

"touch ..." *are not arguments of the script*. So users should be 
careful to get documented behavior.



And shell scripts are not like SQL queries - they often do need to check
arguments. So, the current behaviour is justified, IMHO.


stackoverflow is full of suggestion how to pass arguments to a SQL 
script executed by mysql. Unfortunately it is unsafe and allows 
injection of code. psql (PostgreSQL) allows to pass parameters, however 
it is more like :var than script arguments. So it is true that CLI 
clients for SQL databases do not implement positional parameters.


ARGV is treated in a quite specific way by awk. It may contain file 
names, variable assignments, and might be overwritten in BEGIN block. 
However a close ob-awk header argument is :cmd-line, not :cmdline, so 
inconsistency is even greater.



What might be done is introducing _two_ different header arguments - one
for interpreter switches, and another for script/program switches.

Say, :interpreter-cmdline and :script-cmdline.
Then, we can call the current :cmdline behaviour "dwim" and allow users
to be more explicit if necessary.


It is too easy to confuse org-babel, so "dwim" works in simple cases 
only. Independent header arguments make things more clear, I would 
prefer :script-args. The question is whether they should be interpreted 
by shell (flexibility and shooting feet) or more strict syntax `("hello 
world" 1 a) should be used.

Re: [PATCH] Re: [BUG] ob-shell: :shebang changes interpretation of :cmdline

[Max Nikulin]

On 26/04/2024 18:49, Ihor Radchenko wrote:



+shell-file-name

...

+(list shell-command-switch
+  (concat (file-local-name script-file)  " " 
cmdline))))



Max Nikulin writes:

Using `shell-command-switch' unconditionally may lead to executing
/bin/sh instead of shell specified by `shell-file-name' for script files
having no shebang, see

https://superuser.com/questions/502984/writing-shell-scripts-that-will-run-on-any-shell-using-multiple-shebang-lines


I conclude that your concern, while being valid, is a _different_ bug.
Thus, I do not see it as a blocker for my patch - my patch will fix the
*original bug reported on top of this thread*.


My concern is that your patch trying to fix one bug (I am not convinced 
it is an improvement despite it is a step toward consistency) introduces 
another one that is not currently present in the code.



(with-temp-file script-file
(if shebang (insert shebang "\n")
   (insert "#!" shell-file-name "\n"))
(when padline (insert "\n"))
(insert body))


This code has an issue. Interpretation of relative file names in 
shebangs varies across shells.


If you insist on parsing :cmdline by shell (I do not like it) then you 
may try


  ' 
 '

Re: FAILED test-ob-shell/bash-uses-assoc-arrays

[Max Nikulin]

On 26/04/2024 18:08, Ihor Radchenko wrote:

Max Nikulin writes:


My guess is that GPLv2 BASH on macOS does not support associative
arrays. Perhaps these tests should be skipped if BASH_VERSION is not
fresh enough (not supplied by Apple).


https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=e4ab416fc


Feature detection should be more reliable.

;; Old GPLv2 BASH in macOSX does not support associative arrays.
(if-let ((bash (executable-find "bash")))
(eq 0 (process-file
   bash nil nil nil
   "-c" "declare -A assoc_array")))

Even version check may be performed by shell

(if-let ((bash (executable-find "bash")))
(eq 0 (process-file
   bash nil nil nil
   "-c" "[ ${BASH_VERSINFO[0]} -ge 4 ]")))

Re: [BUG] HTML export does not preserve footnote label [9.6.15 (release_9.6.15 @ /usr/local/share/emacs/30.0.50/lisp/org/)]

[Max Nikulin]

On 26/04/2024 15:17, Protesilaos Stavrou wrote:

Since we are now using labels for the HTML export, I think it makes
sense to optionally use those for the anchor tags as well.

[...]


+(defcustom org-html-footnote-use-label-for-anchor-text nil


Another option may be to rely on the existing one: 
`org-html-prefer-user-labels'

Re: Trailing whitespace after export snippets without a transcoder

[Max Nikulin]

On 23/04/2024 02:01, Ihor Radchenko wrote:

For example, consider an HTML exporter that aligns tags nicely and
keeps blank lines between markup blocks for readability.  If we
remove such blank lines unconditionally, it will be problematic.


I consider that just newlines are enough to make HTML markup human 
readable. I believe blank lines appear in HTML due to conditional 
constructs interpreted by various template engines and almost nobody 
cares concerning actual formatting in such cases.


However I proposed to make this feature an option that is turned on by 
default.



I guess that I can change the condition to not include trailing space
from (rx whitespace eol) to (rx (any " \t|) eol).


One more time I forgot that neither \n nor non-breakable space are 
included into post-blank.


I think, more permissive regexp may be used. At least it should accept 
newlines and any space after it


(rx (any " \t" eol) (zero-or-more whitespace) eos)

Moreover, post-blank of the pruned object may be ignored when the 
following element starts with spaces other than purely zero width ones.


In my opinion, keeping extra spaces (e.g. post-blank ones from pruned 
objects) makes less harm than aggressively stripping them. Anyway some 
backends must normalize spaces (while for others they do not matter).


While newline characters are not affected, this part of change does not 
affect accidental split of paragraphs.


My feeling is that extensive test suite is required. It would be easier 
to review what cases are not handled yet.

Re: [PATCH] Re: [BUG] ob-shell: :shebang changes interpretation of :cmdline

[Max Nikulin]

On 24/04/2024 19:52, Ihor Radchenko wrote:

Max Nikulin writes:

I believe, multiple arguments should be specified as '(1 a "b c").


Yes, but we do not, in general, know how to split them.


Something should be changed anyway since current behavior is 
inconsistent and so is buggy.


The only difference of script arguments from :var is that just a string 
should be converted to a list having single value. It should be possible 
to specify list of script argument as a reference to a named element 
similar to


#+name: shvar
#+header: :var a='(1 abc "def ghi") :results verbatim
#+begin_src bash
  printf '%s\n' "${a[@]}"
#+end_src

#+name: varval
- 1
- bcd
- list items

#+call: shvar(a=varval)


With shebang (as header arg or as part of the body) command should be
  /path/to/script [ARGUMENT]...
when there is no shebang
  /shell/executable /path/to/script [ARGUMENT]...


Maybe instead of `process-file' we can simply use `shell-command'?


Doesn't `shell-command` call `process-file` with `shell-file-name` and 
`shell-command-switch' under the hood like in your patch?

Re: [BUG] No space after footnote with org-export-with-footnotes set to nil [9.6.1 ( @ /Users/test/.emacs.d/elpa/28.0/develop/org-9.6.1/)]

[Max Nikulin]

On 27/03/2023 21:16, Ihor Radchenko wrote:

+++ b/etc/ORG-NEWS
@@ -145,6 +145,27 @@ execution completes.  The new ~:async~ header allows users 
to continue
  editing with Emacs while a ~:session~ block executes.
  
  ** Miscellaneous

+*** Blank lines after removed objects are not retained during export


I have realized that this change affects *space characters*, not *blank 
lines*.

Re: FAILED test-ob-shell/bash-uses-assoc-arrays

[Max Nikulin]

On 24/04/2024 19:54, Ihor Radchenko wrote:

Max Nikulin  writes:


On 15/04/2024 23:46, Alexander Adolf wrote:

 FAILED  test-ob-shell/bash-uses-assoc-arrays  ((should (equal "two"
(org-trim (org-babel-execute-src-block :form (equal "two" "three")
:value nil :explanation (arrays-of-different-length 3 5 "two" "three"
first-mismatch-at 1))
 FAILED  test-ob-shell/bash-uses-assoc-arrays-with-lists  ((should
(equal "20 cm" (org-trim (org-babel-execute-src-block :form (equal
"20 cm" "50 dl") :value nil :explanation (array-elt 0 (different-atoms
(50 "#x32" "?2") (53 "#x35" "?5"


My guess is that GPLv2 BASH on macOS does not support associative
arrays. Perhaps these tests should be skipped if BASH_VERSION is not
fresh enough (not supplied by Apple).


I guess so. Which bash versions should we cut off?


/usr/share/doc/bash/NEWS.gz

This is a terse description of the new features added to bash-4.0 since
the release of bash-3.2.  As always, the manual page (doc/bash.1) is
the place to look for complete descriptions.

[...]

ii. The shell provides associative array variables, with the appropriate
support to create, delete, assign values to, and expand them.


it is consistent with license change discussions
https://news.ycombinator.com/item?id=20102597

FAILED test-ob-shell/bash-uses-assoc-arrays

[Max Nikulin]

On 15/04/2024 23:46, Alexander Adolf wrote:

FAILED  test-ob-shell/bash-uses-assoc-arrays  ((should (equal "two"
(org-trim (org-babel-execute-src-block :form (equal "two" "three")
:value nil :explanation (arrays-of-different-length 3 5 "two" "three"
first-mismatch-at 1))
FAILED  test-ob-shell/bash-uses-assoc-arrays-with-lists  ((should
(equal "20 cm" (org-trim (org-babel-execute-src-block :form (equal
"20 cm" "50 dl") :value nil :explanation (array-elt 0 (different-atoms
(50 "#x32" "?2") (53 "#x35" "?5"


My guess is that GPLv2 BASH on macOS does not support associative 
arrays. Perhaps these tests should be skipped if BASH_VERSION is not 
fresh enough (not supplied by Apple).

Re: [PATCH] Re: [BUG] ob-shell: :shebang changes interpretation of :cmdline

[Max Nikulin]

On 23/04/2024 17:28, Ihor Radchenko wrote:


I propose the attached patch.



+++ b/lisp/ob-shell.el
@@ -322,14 +322,12 @@ (defun org-babel-sh-evaluate (session body &optional 
params stdin cmdline)
  (with-temp-buffer
 (with-connection-local-variables
  (apply #'process-file
-(if shebang (file-local-name script-file)
-  shell-file-name)
+shell-file-name
stdin-file
 (current-buffer)
 nil
-(if shebang (when cmdline (list cmdline))
-  (list shell-command-switch
-(concat (file-local-name script-file)  " " 
cmdline)
+(list shell-command-switch
+  (concat (file-local-name script-file)  " " 
cmdline


Using `shell-command-switch' unconditionally may lead to executing 
/bin/sh instead of shell specified by `shell-file-name' for script files 
having no shebang, see


https://superuser.com/questions/502984/writing-shell-scripts-that-will-run-on-any-shell-using-multiple-shebang-lines

The kernel refuses to execute such scripts and returns ENOEXEC, so the
exact behavior depends on the program you run such a script /from/.

- bash 4.2.39 -- uses itself
- busybox-ash 1.20.2 -- uses itself
- dash 0.5.7 -- runs /bin/sh
- fish 1.23.1 -- complains about ENOEXEC, then blames the wrong file
- AT&T ksh 93u+2012.08.01 -- uses itself
- mksh R40f -- runs /bin/sh
- pdksh 5.2.14 -- runs /bin/sh
- sh-heirloom 050706 -- uses itself
- tcsh 6.18.01 -- runs /bin/sh
- zsh 5.0.0 -- runs /bin/sh
- cmd.exe 5.1.2600 -- looks at you funny

I am not going to spend time testing current versions.

I believe, multiple arguments should be specified as '(1 a "b c").

With shebang (as header arg or as part of the body) command should be
/path/to/script [ARGUMENT]...
when there is no shebang
/shell/executable /path/to/script [ARGUMENT]...

Re: [PATCH] Re: [BUG] ob-shell: :shebang changes interpretation of :cmdline

[Max Nikulin]

On 23/04/2024 17:51, Max Nikulin wrote:
I am in favor of dropping `shell-command-switch' in the latter case to 
pass arguments literally in both cases.


Dropping "-c" may have side effects. Instead of :shebang, a source block 
may have shebang in the body


#+begin_src bash
#!/bin/bash -e
echo first; false; echo second
#+end_src

This shebang is ignored if the script is executed as
bash /tmp/script
and respected in the case of
bash -c /tmp/script
Shebang in the script body may be detected to run it as
/tmp/script
or
/bin/bash -e /tmp/script

To avoid interpretation of shell specials in script arguments when "-c" 
is used, it is possible to use a trick

bash -c /tmp/script ob-shell arg1 arg2 arg3
The -c option adds extra execve() call in comparison to
/tmp/script arg1 arg2 arg3
Perhaps it may be neglected.

It may be more tricky on Windows where shebangs are likely ignored even 
by bash. However I do not thing ob-shell is working on windows since 
`shell-command-switch' should be /c there instead of -c since default 
shell is cmd.exe.

Re: [POLL] Should we enable or disable automatic tag alignment by default everywhere

[Max Nikulin]

On 23/04/2024 18:47, Ihor Radchenko wrote:

Max Nikulin writes:


There should be a way to align tags for a specific heading even when
`org-auto-align-tags' is nil.


I can convert org-align-tags into a command.


I would prefer to have C-u C-c C-q (`org-set-tags-command') applied just 
to the current heading (or to active region). Unfortunately it would be 
a breaking change. I have no idea how to make (org-align-tags) call 
easier than (org-align-tags 'all) without breaking changes. Otherwise it 
would be inconsistent and M-: (org-align-tags) can be used in the last 
resort.

Re: [PATCH] Re: [BUG] ob-shell: :shebang changes interpretation of :cmdline

[Max Nikulin]

On 21/04/2024 22:09, Matt wrote:

The proposed solution assumes we intend to parse the characters
following :cmdline as space delimited and grouped by quotes.  However,
AFAICT, the parsing issue makes this solution ambiguous.


Matt, I am sorry, but I do not agree with your proposal. I do not think 
that `split-string-and-unquote' will solve all issues.


Certainly issues with formatting of output should be treated separately.

I figured out there is at least one more issue. Consider

#+property: header-args:bash :results verbatim
#+begin_src bash :cmdline $LANG :shebang #!/bin/bash
  printf '"%s" ' "$0" "$@"
  printf '\n'
  tr '\000' '\n' First line is argv as it is represented for a script, next lines are 
exec arguments at lower level (actual executable may be obtained from

"readlink /proc/$$/exe")

Notice that in the former case "$LANG" is passed literally, but in the 
latter it is expanded. I am in favor of dropping `shell-command-switch' 
in the latter case to pass arguments literally in both cases.


I think, it would be more consistent with :var to specify multiple 
arguments using elisp lists

#+header: :cmdline '("first 1" "second 2")

However looking wider, I do not like that :cmdline for ob-shell has 
different meaning than for other languages, see e.g. ob-sql. Only for 
shell this parameter is treated as arguments of a *script*. In other 
cases :cmdline is used to specify arguments of *interpreter* and I think 
ob-shell should follow this convention.


Actually script arguments (and :stdin) might be applied to python and at 
least some other languages, so support of this feature should be moved 
from ob-shell to common org-babel code.


My point:
- header arguments should have as close as possible meaning across 
various languages.
- withing ob-shell variation of ways to execute a script should be 
minimized either some parameters (:cmdline, :shebang, :stdin) are 
specified or not.


Finally a note on tests

+(ert-deftest test-cmdline-alone-and-with-shebang-have-same-result ()
+  "Pass arguments to a block.  Don't use shebang.  Then use
+shebang set to the same language as the block.  The result should
+be the same."
+  (should (equal
+   (org-test-with-temp-text
+   "#+begin_src bash :cmdline 1 2 3
+echo \"$1\"
+
+#+end_src"
+ (org-babel-execute-src-block))
+   (org-test-with-temp-text
+   "#+begin_src bash :cmdline 1 2 3 :shebang #!/usr/bin/env bash
+echo \"$1\"
+
+#+end_src"
+ (org-babel-execute-src-block)
I believe, that starting point of the discussion is that the results 
should be same and decision what is more correct is the result of the 
discussion. Unit tests should check both cases independently and should 
fix particular treatment of arguments.

Re: MathML and ODT export: inline possible?

[Max Nikulin]

On 22/04/2024 16:05, Fraga, Eric wrote:

On Friday, 19 Apr 2024 at 23:19, Max Nikulin wrote:

MathJax may be your friend. LaTeXML and katex do not add  as well.


MathJax works well but for websites; my context is that I need to
prepare a Word document to share with others.


I have not tried it, but I suggested MathJax expecting that the 
following features may be combined in a small custom CLI tool:

- MathML,
- Server-side rendering.

Concerning pandoc, perhaps it is possible to create a Lua filter to 
handle some specific cases. Again, I have seen some buzzwords in docs 
and discussions, but I have not tried it myself.


Certainly my expectations in respect to LaTeXML failed. I believed that 
a tool used for https://dlmf.nist.gov/ should be close to perfect. The 
reason might be that it is tested with browsers, but not with office 
software. LaTeX3 code appeared to be a severe performance test for engines.